Wondering if brass that is fire formed and neck sized will tighten things up even more
Not to piss on your campfire, but I'd be more concerned with what happens if the stock moves on you a little and kills your sweet spot.
I have a GAY M77 MkII that shot as well as yours with cheap Rem Greenbox 130s, and in the "3s" with hand loads (130BTs and H4831). It was a crazy quick rifle to work up a load for, and I loved shooting it. A couple years later I got married, moved from a rental on a 20K acre ranch to a nice house in town, and bought a big gun safe. A couple years after that I take my rifle to the range to confirm my zero is where I left it, and all of a sudden I'm shooting 2" groups! Long story short, the walnut moved a LOT............he wood crackled as I backed out the action screws.
I bedded the action, floated the barrel, the old load was still crap, but no longer stringing. I worked up a new load that got things down to a consistent inch, but I've never approached those first groups with the factory settings. The barrel is capable, so I'm sure I could get much of that accuracy back if I stuck with it.