Official comment
Okay, so the reason this happens is the audio from the mic is routed into the software, but there is a bypass switch on the back of the RR to directly route the mic through the hardware instead:
PC Mix (the rear slide switch) is specified as follows:
PIC Mix ON: Send MIC and PHONO/LINE audio to the computer
PIC Mix OFF: MIC and PHONO/LINE audio is not sent to the computer and directly output from the hardware audio output.
Therefore, when PC Mix is off, how high or low the buffer size is, the output audio will not be affected.