You could be getting inconsistent pressure on your barrel as the barrel heats and cools. Before you try shimming it out of the stock, try putting a single shim between the pressure point and the barrel and try that.

Aim for the exit hole.