Your first issue is something I've noticed in the past with other FX units, the EFX-1000 included, but with the RMX-1000 I'm pleased with the results. This makes me think that your issue could be down to a few factors...
- The Send/Return quality of the DJM-800. Newer models such as the 850, 900 or 2000 might have a better quality Send/Return loop - of this I'm unsure, but it's a possibility.
- The quality and length of the cables you're using for Send/Return. Cheaper cables, like basic, 5+ metre long guitar leads will sound shoddy compared to shorter, higher quality patch leads. I'm not saying to spend hundreds on cables, but £20 (approx $35) in cables is well worth the investment. Beyond that, since the audio signal is passing through a processor and several interconnects, you have to expect tiny alterations in sound quality. None that should be truly noticable though. Sometimes just hitting that FX button on the mixer is enough (especially as you can hear a tiny transition "blip" - just like when you engage a Color FX or Inst FX) to trigger your brain into thinking there's a difference.
The second issue is simple, the audio signal has to travel from the mixer to the RMX, it has to travel around the RMX through various signal processors, and it has to travel back from the RMX to the mixer. Quite literally the processed audio has to travel a much greater distance, add to that the few milliseconds that will be lost due to the complex operations the signal processors have to carry out on the effected audio, and the result is a very very slight, but totally unavoidable delay. To minimise, get shorter cables, but that slight delay will always be there. The only way to avoid it is by putting your RMX onto the Master output (thus losing the ability to send individual channels to the RMX), or by breaking the laws of physics.