Try the following steps:
1. Change the setting of "Backspin length" to "short" in the Utility mode
a) Disconnect the USB cable connecting the unit and computer.
b) Turn on the power of the unit while pressing the [SHIFT] and [PLAY/PAUSE] buttons on the left deck.
c) Press pad 6 on the right deck (that's the bottom row, second on the left. Or see picture on page 28 in the user manual)
d) Turn off the power of the unit.
2. Factory reset
a) Disconnect the USB from the RX
b) Turn on the power of RX while pressing [SHIFT] button on the left deck and [DECK 1] button at the same time.
c) Wait for about 8 seconds.
d) Press the [SYNC] button on both decks for about 3 seconds.
When the [SYNC] buttons' LED is turned off, reconnect the USB and launch rekordbox to see if the latency has improved.
3. Update firmware to version 1.04
Make sure to update your DDJ-RX firmware to version 1.04 (or later if available).