You may already have the headphones selected by Windows as the output device, thus it's not available for rekordbox to obtain exclusive access. Check that.
Headphones not selectable as audio output
Since a couple of months I haven't been able to select my bluetooth headphones as audio output in rekordbox. No controller or external device is connected to rekordbox. As shown below rekordbox detects the headphones (Momentum 3) but when selecting them the audio output just switches back to "no device selected" (see second picture).
Any ideas on how to fix this? I have been able to use these headphones before with rekordbox but at some point a couple months ago after an update it stopped working...