I generally hunt elk in Colorado at 10,500 to 12,000 ft though most recently in northern BC at ??? ft. While doing so I have lived in the Bay Area of Californa, West Texas, the Gulf Coast Texas, SE Kansas and Calgary, Alberta. I have never tweaked my sights after zeroing them where I lived and have had zero problems with deer or elk out to ~300 yards. My ability to shoot dictates whether I hit or not so I don't worry about altitude.
