I went ahead and removed the spacer on in the mag box. Instead of grinding the spacer out I tapped the tig welded side with a hammer a few times and used a large flat screwdriver by inserting it in the spacer itself and twisting the screwdriver. The spacer came out perfectly with no damage to the box itself.
Spacer in mag.
Removed spacer.
The internal mag space was increased from 3.38" to 3.67" which now allows me to seat 250gr Berger hybrids at a length which shoots really well in the rifle.
The rounds also feeds and ejects perfectly without any modification to the follower or anything else.