There are a few threads between this (VDJ) forum, the CDJ-350 and the CDJ-850 forum that discuss users having dropouts of audio and control -- we may have found the source of the problem!
If you are running VDJ Pro, this will be the only way you have to correct the issue - LE users don't have the appropriate settings (although I may be wrong on that, I haven't fully tested it with LE).
Go to your Config, select the Performances tab and where the Sound Card Latency probably reads "AUTO", change that to another value; this value will differ from computer to computer -- use the lowest setting is that doesn't result in hiccupping or glitches in the audio.
It would seem that in AUTO mode, the software tries to dynamically adjust the latency of the soundcard, but in doing so, can cause a disconnect of the driver and thus a total freeze of the controls and the audio playback will stop.
Please try setting this from AUTO to a fixed value and let us know if this resolves any of your issues with the 350/850 as soundcard/controllers for Virtual DJ.