Here's my suggestion. Your concern seems to be that you both started and finished at too high of a charge level and you are concerned about the safety?
I wouldn't pull down the loads right away; I'd load more with even lighter charges and start with those first. It may very well be that the Speer data is high for your rifle, but it can also turn out to be just fine. It would be a shame to pull down loads, just to turn around and load them again if IMR comes in on the whimpy side. Speer's data was right for some rifle, and your's may follow suit.
Worst case scenario you end up pulling the loads down a little later.
