Sounds to me as if you aren't generating enough recoil. I'd bet the farm on it.
I took a look at my VV manual, looks like 4.2 gr of N310 only gives 770 fps. Find a slower burning powder, HP38/231, Unique, Universal, Herco, Power Pistol, WSF and probably 20 others will allow you to get close to the 830 - 850 fps that is the GI standard for the .45 ACP.
I'd also give them a slight taper crimp. If your RCBS dies were made in the last 30 years they should do taper, not roll so use them, just adjust the crimp die down a bit more. If that fails, buy a Lee die, they do indeed work well.
