I would tend to think that increased energy with the same bullet makes a difference. If energy did not mean much, then a cowboy action load with hardcast should stop an animal the same as a fire breathing 454 load. We know that is not the case. There is a point when too much energy can decrease penetration when the bullet is not up to the task. The bullet simply will break apart. I know that I would feel better with a 454 than the same bullet in a colt. There is nothing wrong with the colt but there is a reason for more powerful cartridges.
Then I would ask if you have ever hunted with a handgun. The reason I ask is not to be a jerk, but to see if you have killed any game with a .454 or a .45 Colt. If you have, you will see that game just isn't impressed with the increase in velocity -- not in the least. Energy is nonsense.