I don't think you're missing much, but this is what I know to be true. Berger's can sometimes be a difficult bullet to get to shoot well. RL22 is almost always a great move in a .300WM. I haven't loaded for a .300WM yet that didn't yield the best results after being pushed hard. If you don't have your heart set on the Berger, then try the 208 A-max. While it doesn't have the BC that the Berger does, it is an easier bullet to get to shoot well. Check your twist rate as well.
Hope this helps.
R.