If the brass starts out with different sizes before resizing this can also be a culprit. When shooting different powder loads you will have different expansion of the brass especially when new. Brass springs back. So cases that start larger stay larger than cases that start smaller because it is trying to spring back to it's original size. At least this has been my experience.
I have the redding competition die set. I know what my maximum case expansion is. But I might use 3 different shell holders to get all my brass back to the same size. Well at least within .001.
