Does it bother you if you need to drift your rear sight (to sight in your rifle) to the point it is noticeably off center at a glance?

Yes. If it's rather noticeable I drift the front sight a little so i don't have to go as far with the rear sight.