Follow up question.
If POI for different bullet weights and types is several inches apart left and right, what can be deducted from this?
The rifle starts to recoil as the bullet starts to move, and so POI is affected by, among other things, how much recoil and how much time between the bullet starting to move and it leaving the muzzle. Just to complicate things, the rifle doesn't recoil straight back, but tends to pivot up at the muzzle (because the recoil is transferred via the butt, below the line of the bore on most rifles). It can also pivot laterally due to things like cast and the effect of the shooter's cheek and other aspects of how the rifle's held. In fact it is for this reason that you can get different POI from different shooting positions, even without changing loads.
As well, with all of this movement the barrel is not infinitely stiff, and it tends to flex as all of this is going on, and this in turn can be affected by contact with the fore-end.
All of these things can contribute to different POI, including lateral shift.
Usually all of this doesn't matter all that much: you zero for the load you'll use, and (if you're smart) from the position you'll favour for the longer shots, and leave it at that. It only really starts to show up if you are shooting different loads, positional matches, or trying to regulate a double rifle.