I am not an expert, but in my experience a gun with a fair amount of recoil, a flimsy fore end and off a bipod is a hard combo to get to shoot consistently. The barrel has shot good in the past , but has been rechambered. I believe you need to either address the fore end and shoot off a good rest , to work out the other variables. Then integrate the bipod back into the setup. Bipods tend to magnify an issue such as yours, and the recoil of a .358 Norma working against the bipod also does.
