What caliber and loading? How are you gripping the gun? How hard are you gripping the gun? Without knowing these things it is hard to prescribe advice. I would try the following:

1. Grip the gun with your pinky curled under the grip.
2. Grip the gun firmly.
3. Try a lighter bullet.

The signs point to the gun riding in recoil more than normal before the bullet leaves the barrel. 1 and 2 help control the gun so it recoils less. 3 gives the bullet less barrel time so it exits sooner.

I wouldn’t be filing the sights or clocking the barrel of an adjustable sighted gun. That is typically only done to the fixed sighted guns like the SAA and Vaquero. How does it shoot for other people?