Official comment
First problem ... the XDJ-RX shipped with a license key for rekordbox 3, which required a key to use the basic EXPORT mode. With rekordbox 4, we removed that requirement as rekordbox dj (PERFORMANCE mode) was released. rekordbox 5 continues to operate the same way.
TL;RD - that key is not valid for unlocking the rekordbox dj plus-pack, and would require that you purchase one or obtain one bundled with a compatible hardware product. Sorry!
Second issue, you'll need to bypass the High Sierra driver security by doing the following:
1) Install the XDJ-RX drivers again.
2) Disable the driver security.
3) Install the drivers again immediately.