I went ahead and removed the spacer on in the mag box. Instead of grinding the spacer out I tapped the tig welded side with a hammer a few times and used a large flat screwdriver by inserting it in the spacer itself and twisting the screwdriver. The spacer came out perfectly with no damage to the box itself.
Spacer in mag.
[Linked Image]

Removed spacer.
[Linked Image]

The internal mag space was increased from 3.38" to 3.67" which now allows me to seat 250gr Berger hybrids at a length which shoots really well in the rifle.

The rounds also feeds and ejects perfectly without any modification to the follower or anything else.