What is your turntable setup for ripping your vinyl collection?
Many years ago when I first switched from vinyl to digital music, I had the exact same problem when I loaded vinyl rips into Ableton Live. The tempo always fluctuated up and down slightly. It was a bitch to warp the tracks as a result.
One very important tip I have learned from experience: When recording vinyl to digital format, it is best to use the rubber mat that comes with your turntable instead of a DJ slipmat. The rubber mat prevents the vinyl record from slipping as the platter spins, which helps maintain a consistent tempo on the final recording.
I know that some people are going to cry out that the slip[mat makes no difference.... and in theory I would agree with them. But in experience, I have personally witnessed exactly the tempo fluctuation you mentioned. The rubber mat made a huge difference for me.
Now.... as for how to best fix your tracks that are already recorded and have the fluctuating tempo. One suggestion would be to use Ableton Live to warp those tracks to a constant tempo and then export to a new audio file. When you import the new audio file into Rekordbox, it will be much easier to match the grid.