I was coming into the muzzleloader scene in the early 80s, just as the modern inline thing was about to break. At the time, anything 45-54 cal was considered kosher for deer. The .54 in a round ball gun was considered optimal. That's what I went with.
It's been 40+ years, and I was out of the scene for most of that time. I went back to Friendship for the fall shoot this year-- first time in 35 years or so. I'm currently working up a Brown Bess (.75 cal) for deer, turkey and squirrel. However, I've already started thinking about a full-stock flinter for deer, and I'm asking the same sorts of questions.
I think the bias towards .54 cal back in the day was that it was seen as a slightly heavier ball that would do better on a deer. When conicals and faster twists came about, you could get all that in a .50 cal. I later swapped out my round ball barrel for a Green Mountain 1-28" .54 cal drop-in and that thing pole-axed deer.
Just this year, I pulled that barrel off and went back to roundball.
If I had to resurrect the thoughts I was having in 1984, as I was shopping for my first muzzleloader, the advice I had rolling in my head was this:
1) .54 cal was the best caliber for deer and bear for normal hunting
2) .50 cal was better if distance was the issue
3) .45 cal was better if weight was the issue
I've put that forward to the current crop of smokepole mavens and about half agree and half disagree. I'm not saying my ideas are worth anything. I'm just telling you how I made the decision to buy a TC Hawken .54 cal in 1984.
