As a baseline I would ask myself this question:
What is the smallest handgun (size and cartridge wise) that I am willing to get into a gunfight with, against an armed adversary/adversaries?
That is my thought process. The Glock 43 met that as the barest minimum for me, with two spare +1 magazines, and only in the rare instances when I cannot carry more gun. Of course, at places like Disney World, it's a lot more bare than that.