Normally Rekordbox is capable of making a good quality recording via the internal recording option. You can set the auto normalize function, and recordings can be saved in loss less format. In this case your recorded mix will sound as good as the quality of your tracks.
When all your recordings in Ableton, Rekordbox, Audacity, etc. can't make a satisfactory recording, maybe the orignal mixed tracks are not in high quality. After all you can't make perfect mix recordings with lower quality tracks.
Personally I use loss less AIFF tracks, each about 50-80MB in size. When I record a mix in Rekordbox with those tracks, and the mix in finished, I use Platinum Notes to improve the sound quality. The result is a smooth normalised track, with just a little extra warmth in it, and absolutely not over edited.