I think you’ll be fine not carrying a pistol for backup. 😉
I wouldn’t throw my rifle down and Wild Bill Hickok a charging bear.
I never carry a handgun if I'm carrying a rifle. OTOH when we go back to camp after shooting game and get all the stuff to clean etc... I swap the rifle for a handgun...