You might check the shape and protrusion of your firing pin, it’s fit in the hole it passes through as well. If the firing pin is too sharp or fits sloppily on the bushing it can give you trouble such as you describe as pressures rise.

I load 95 and 100gr SWCs in my 32 mag, over either 5.5gr of AA5 or 6gr of PowerPistol. My revolver is an S&W K frame 16-4.