The first detriment would be the restriction on reaching the safety. Most bolt action rifles have the safety located to be reached by the trigger finger or thumb at or near the top of the action. Having your hand wrapped inside the stock makes that more of a reach.
I suppose technically, at thumbhole would mean more wood on the stock and thus more rifle weight. Not sure if this matters to you.
I had a thumb hole on a 22-250 varmit rifle. Great for that application as the whole rifle weighted a ton anyway and I was not "snap shooting" so I had lots to time to reach the safety, if I bothered to walk with a round chambered.
On the plus side, the thumbhole allows your trigger hand to provide some measure of recoil control and absorbtion.
Personally for the safety activation issue alone, I would not choose a thumbhole stock for a deer or similar rifle.
