Try 40-41 grains 4895/165 Sierra GK. Very mild (some would call wimpy) but crazy accurate in my last three 06's- and a sound killer on our eastern whitetails. Why beat yourself up with each trigger pull? Shoot loads like this and save the hot rod potential of full-snot '06 loads for situations that demand it.
The only thing that beats this load for me in an '06 are cast loads, but that's fodder for a different campfire, maybe.
What velocity are you getting with these loads?
