We try not to change the sights on a fixed sighted gun until we've found the load that we like the best, then we compensate if necessary by moving the sights. After the sights have been drifted for our chosen load, we leave them alone. Even if we are shooting another load that won't hit perfectly to point of aim, we leave them alone. It's our chosen load that we worry about.
Left and right are normally a sign of changing grip mechanics. Cast lead reduces friction in the bore, but you maintained that the cast lead load was hotter. This could result in some leading, which could effect time in bore. Time in bore isn't only responsible for elevation changes, but if you consider torque to the platform by a bullet encountering left or right rifling, you can encounter windage changes too, if your grip isn't totally consistent.
Weaker grips cause this. Or, grips too high, too low, too much trigger, too much thumb...etc. It goes on and on.