A good barrel is not going to "walk" that much. If a barrel does walk a lot on you, it is likely "stressed " from the machining process. Any amount of uneven pressure exerted on the barrel, as it warms up from shooting, will also cause your poi to wander.
