I have shot thousands of rounds through aluminum frame guns. Never a gouge track. It is not a problem. Bullets slide right up. Even with high slide velocity and extra power magazine springs. Nonexistent concern.
The gouges are caused by unskirted followers, not the bullet. I've seen scratches on steel frame guns. The last round drags the point of the follower into the frame ramp - there are plenty of lightweights with beat up frame ramps out there. Chip McCormick specifically changed their PM+ follower design to address the problem.
