If you choose to continue doing external mixing with software, lower the channel trims in Rekordbox for each PFL channel and turn off autogain. If Rekordbox has outputs attenuation that affects the individual channels when using external mixing, then turn that on instead. Your adjustment of the knob in the upper right I suspect only affects the master out when using internal mixer mode.
Alternatively, if the DJM-900 has the ability to do all proper MIDI stuff and receive a master & a headphone signal separately and directly to those mixer sections while all the mixing occurs in Rekordbox, then that will allow the on-screen information to be adjusted by the DJM-900. That would also mean the on screen master knob will be valid, though probably mapped to the mixer's. In such a situation, also turn off autogain and be conservative with levels. DJ software tends to be clunky with headroom.
FWIW, in most situations outside of extreme use stuff like crazy pitch % with keylock on where Elastique is advantageous, the players in standalone mode using either CDs, USB drives, or Link track offloading from old Rekordbox will give you better sound than Traktor.
