Is your rest one of those cradle things? The rifle may be moving out of position after the first two shots. Have you tried it with just a front rest?
I use these bags. And yes, the rifle moves out of position after every shot, and always need to re-aim it.