I've been fortunate to shoot many big bore single actions from both FA and custom ruger bisleys in 454, 475 and 500 Linebaugh. No matter what your experience with rifles, handguns are an entirely different matter. My shooting buddy can handle handgun recoil levels that leave me flinching all over the place, whereas I'll shoot rifles w/o flinching he refuses to shoot.

My advice would be to a get a mdl 83 in 45 colt, and handload to whatever level you want. I'll also add that IMHO, the majority of handgun shooters cannot accurately shoot a 475 or 500, well unless you build the gun very heavy and add a muzzlebreak. Extended shooting with the 475 and 500 has caused permanent injury to shooters wrists, and more than once the unaware has had the front sight planted into their foreheads.

My limit is the 480 ruger, 400 gr @ 1200 fps will take anything I'll hunt with a handgun, and it's shootable (I have about 4000 rounds through my srh)