I've shot a few VTR's, and own one, a .204. I haven't seen that with any I've shot, and in fact my .204 will put 10 shots into 3/4" or so, shot as rapidly as possible.

But that doesn't mean an individual barrel can't have a kink in it that unwinds when it gets hot, or that there's some sort of bedding problem.

Have you tried letting the barrel cool between the 3rd and 4th shots? Are you checking for bullet runout?

“Montana seems to me to be what a small boy would think Texas is like from hearing Texans.”
John Steinbeck