Though if the barrel is a good one, it should be just as accurate whether shot hot or cold.
One factor that can affect group size in cooler weather, especially on calmer days, is warm air from a heated barrel roiling the view through the scope.
Many other factors tend to increase group size as more shots are fired. It's rare to see shooters put out wind flags when testing loads for accuracy, but a 3-mph "gust" can cause so-called fliers even at 100 yards. I don't do serious accuracy testing without putting out at least one or two flags.
Also, a lot of shooters don't check the parallax on their scopes. A little bit of parallax can result in fliers when more shots are fired, due to the shooter not having his eye in exactly the same position behind the scope for each shot.
Then there are variations in holding the rifle for each shot, or just in pulling the trigger.
All of those factors can add to impression sporter-weight barrels "always" starts to wander as it warms up.
