Several factors...
1) The speed of the USB drive.
2) The filetype of the song you're loading.
3) The number of memory / hot cue points.
4) Whether another track is already loaded / being loaded.
Start with the first one -- you say you have several USB drives, what is the make / model of them? Are any of them what one might consider a "performance" USB drive?