I'd hate to be so limited but were it to occur, there are many variables to consider. A handgun, to my way of looking at this topic, would primarily be for self-defense. So I'd look at the utility of rifle first. I'd want a rifle that can harvest most North American game. I'd go with a .454 Casull. I'd use very stout rounds in it. For my handgun, I'd use .45LC ammo, not necessarily anything other than standard factory stuff. In self-defense applications, sight picture retention and quick follow-up shots are mandatory.
A very good argument could support either a .44 Rem Mag or .41 Rem Mag. I wouldn't consider a .357 Mag. But that's merely my opinion.
In any case, one would have to become a damned good hunter were one limited to using a rifle chambered for a handgun round.
