The reverse buffer is much shorter than the forward buffer, and that amount is limited / determined by the model of player and the speed of your USB device in filling the buffer. For example, a CDJ900 has a much smaller and slower buffer than the CDJ2000nexus.
If you are already using a high-performance USB drive (the most expensive aren't always the best), you should be okay on that end, but it may simply be the amount / speed you're spinning the jog. Try doing it a bit slower and you'll find better results.