Problem 1) djay for Windows doesn't (yet) support ASIO drivers, meaning it can only use a single stereo pair for output. The DDJ-RB is an ASIO-based device with four (two stereo) outputs; one pair for the headphone cueing, one pair for the master out. Because djay is forced to use WASAPI (or WDM line drivers), it doesn't have the ability to send audio to anything beyond the first stereo pair, meaning there isn't a proper output for the DDJ-RB unless you want to go to mono, in which case the channel headphone cueing still doesn't work properly. This is partly due to their software design, partly due to how the hardware is configured to route audio to the headphones; the pressing the MASTER is simply grabbing the master audio out and putting it to the headphone output; the software does this on the L channel (in split mode) and can't send the R signal because it's not going to the channel cue audio routing (the 3/4 outputs for the headphones).
Problem 2) The cue buttons will illuminate, they just won't work properly per above. You may also notice the LED meters for the signal level don't work at all - this is because the output MIDI mapping required to get that to work hasn't been configured in the software.
Unfortunately this software is relatively new and even their support forums had very little in the way of answers or workarounds; they did have one, but I tried it, and it didn't work very well, so I won't even bother linking or telling you about it.
Solution 1) Use rekordbox; it's what this controller was designed for.