Aahh....the culprit emerges!
Some of the tightest groups I've fire with M70's have been with pre 64 FW's chambered 30/06. But.....they can be very persnickety and unless well bedded and tuned can be difficult.
I think the reason is simple. A 180 gr load is a lot of mass to push down the barrel of such a light rifle, and the barrel contour (IMO) is on the ragged edge of being heavy enough for a 30/06.
Put it in a light,stiff synthetic (which does not absorb vibration but rather transmits it) and the rifle will quickly show what it likes and dislikes. Small details become important with a light rifle and powerful load. It's harmonics.
Surprisingly (maybe) 150-165 gr bullets will sometimes shoot better.Matter of fact most of my pre 64 30/06's have been FW's and in general 165's will show better grouping ability than 180's but not always.
What I would try: Check your ammo for run out and segregate ammo using the stuff that measures .005 and under for grouping. Use the other stuff for off hand practice etc. See if that works. With proper bedding I have seen pre 64 06 FW's go from indifferent grouping to small groups by watching run out.I would bet $5 bucks that those "flyers" have excessive run out.
Get a Redding Competition or other straight line seating die and seat bullets straight as possible.Straight seating can be a bigger issue than seating depth.
Also, IMR4350 may not be "the" powder. I would try H4350 and RL 19.
Third, if the 180's still won't do it, switch to the 165 gr Partition. For most uses the animals will never know the difference.
BTW, stick those 180 gr loads in a standard grade and they will probably shoot knot holes. Just more rifle mass to soak up the vibes.
