As a baseline I would ask myself this question:
What is the smallest handgun (size and cartridge wise) that I am willing to get into a gunfight with, against an armed adversary/adversaries?
Spot on.
This is a fun discussion, and a good exchanged of experiences. But IMO time would also be well spent on personal assessments (careers, weather, body type, etc), to find solutions to concealing proven platforms.
With today's eclectic fashion, and a market flooded with holster options, the right combination is out there for the determined searcher.
I'd much rather make sacrifices to my personal fashion than my pistol choices.