I need help on downbeat

AT the start of a track I set the downbeat and all is good (150BPM) however my track slows down or speeds up 1/4 of the way into the song (throwing the BPM off to something like 150.6 BPM) At the halfway point the song kicks back in with a downbeat that is off however the beat is back to 150BPM so it's not like the beat slowed down or picked up and stayed that way.

Now I know how to set the red bar (Down beat) at this spot however when I do this my original down beat at the beging of the song moves... WTH! How can I set the down beat at the halfway point to get the grid back on track without screwing up the first down beat the the begining of the song?

On a side note, when RB or MIK shows me a BPM 150.6 before I start messing with my song I usually know the song is at 150.00 BPM and just speeds up somewhere but goes bact to 150.00 BPM. I know I just have to adjust... and to my point of throwing the down beat off at the begining of the song