You've got to have the fire burning before the bullet starts to move. You have a magnum, with a mighty primer, and there's a chance that some bullets move with primer firing, that's "upset" and a "variable" that hurts beyond the speed loss. You are on the right track in your thinking.
Mandrel diameter has an effect on your pull, if you have only 2 thousandths, that's on the low end for everyone except benchrest, and THOSE guys put their bullets right against the riflling. Keep in mind that just one thousandth in neck wall thickness is HALF of your total, or TWICE your total "pull." That's why the BR people turn necks.
Further, you've got a kicker, so I have tio wonder if your loads in the magazine are shifting enough to mess stuff up, too.
Either get a bushing necker and the bushings that give you four thousandth's (or so) or buy a couple of extra balls and spin them down to the point where you've got a firm, and CONSISTENT pull. Also keep in mind that more tension is more forgiving than loose necks up to a point.
You should be able to feel pull as you seat, the amount of force, especially if you use your fingertips on the loading lever handle. That has been sufficient for me, if something feels wrong, it usually is.
