Couldn't figure out how to do this, but worked around it by running both Rekordbox and Traktor at the same time, both with the DDJ-RX set as their input/output, then having Rekordbox for playing digital and in the background, Traktor with the two decks as live inputs. This means I can use the Traktor EQ and filters (which are much better than the DDJ-RX internal ones) on the vinyl after mapping the channel 3 and 4 controls on the DDJ-RX to Traktor.
Not a robust solution that you'd use in a club of course, but seems to work fine for home use. Only real downside I guess is I can't add Rekordbox effects to the vinyl input (I'm not that much of an effects user anyway, and could use the Traktor ones with another MIDI controller), and I can't record the whole set internally in Rekordbox (but can use an external recorder or something like Blackhole).
Live external input would be a nice feature for future though :)
