XDJ-RX, Beats drifting despite matching BPM

So I recently bought myself an XDJ-RX, and I have fairly good experience with DJing. One thing I have found is when I beatmatch by ear, it sounds great, and when I put the second track in, after about 20 seconds the tracks drift apart and the beatmatch is lost, making it sound terrible. I don't want to have to adjust my tracks every 20 seconds, once the beats have been matched, they should stay matched, right? I have analysed the tracks in "Normal" mode, opposed to "Dynamic" and I have adjusted the grids on my tracks so that they are perfect, yet the problem still persists. I'm guessing this is a firmware issue?

I have seen a lot of people also making comments on the same topic and how it is a problem for them, so I know it is not just me!