The brass has spring to it. So when you start it moving it won't necessarily linearly follow the amount the die is turned into the press.
Yes, it's probably a little unpredictable in that way but it still should be a fairly close approximation...but the point is that screwing the die in *just a little bit more* can push the shoulder back much further than what you intend.
I would imagine that annealing reduces the "spring back" effect just as it affects bullet pull (I dislike the term "neck tension").