For big game hunting the 308 is excellent and the 243 is recoil matched for small folks or hunting pests.
Have you looked at other rifles besides the Cooper? I prefer the CRF and M70 type safety on the Kimbers.
I'm 6'2" and 210 pounds, but my size hasn't fooled me into thinking it takes a lot of gun to hunt deer.