How many different bullets have you tried with your rifle?
I have a Ruger #1V in .25-06 (not near the pedigree of your rifle) that I worked up a load in using the 100gr SGK. This rifle would regularly shoot this load about 0.75" @ 100yds.
A season later, I decided to swap bullets (110gr Sierra TGK) and I also upgraded scopes. At the range, I struggled with a similar issue as you describe...the first two shots were touching, or nearly so, and the third was 3-3.5" left. More shots carried on with the same theme.
I was assuming I had a loose screw somewhere and went over the entire rifle...everything was snug. I started suspecting the new scope. By chance, I had some of the old ammo onhand. With a hot barrel, it put my previous 100gr SGK load into a triangular group just under an inch @ 100yds.
By design, this rifle has some significant contact between the forearm and barrel.
I don't think I would start removing wood on that rifle. Maybe you just haven't found "THE" bullet yet?
