You need to match the BPM to the BPM the track was playing at while synced with the pitch slider after you disable SYNC mode.
E.g. If a track is playing at 129BPM with SYNC mode on, but is originally 126BPM, when you disable SYNC mode the track is still locked at 129BPM while the pitch slider is at 0%, you would need to raise the pitch to 129BPM to fully disable SYNC and then drop the pitch back to 0% to get back to 126BPM.
I'm assuming it's this way incase you accidentally disable sync to stop the track jumping in BPM mid mix.