I've hunted buff 3X so I'm not expert by any means, but on the 2X I've followed up buff, I've not removed my scope - just left it set it to 1.5X each time and it's been great.

Good hunting.