I see no reason to cheap-out by using inadequate ammunition, either by construction or caliber, when planning a hunting trip simply because it's what you have on hand. Buy some game bullets and leave the varmint bullets at home.
I've seen bigger and heavier bullets than the 85gr. CT frag on game animals and the animal need tracking to be put in the bag.
