I hunt deer every fall with a pistol in one flavor or another. I have killed deer with .357s, .44 mags and 10mm. There are lots of options out there for weapon platforms. Find a platform that fits you and start there. Practice, practice, practice. Shot placement is more important then knockdown power. You don't need a 460 or .500 to kill a deer. I could see using the larger 460 X Frame and using .45 LC to take the recoil until you get some practice under your belt, but keep in mind that the .454 is no slouch.
A .44 mag is a good middle of the road starting point if you like wheel guns and can utilize .44 specials for low recoil practice. If you are into autos check out the 10mm. But keep in mind that all 10mm ammo is not created equally. I have really enjoyed hunting with my 10mms the last few years and have to say that the round itself is my favorite pistol round.

Freedom is never more than one generation away from extinction. We didn't pass it to our children in the bloodstream. It must be fought for, protected, and handed on for them to do the same... President Ronald Reagan