I think their reputation for toughness stems from being a herd animal and their strong will to stay with the group. They will immediately take off at the shot and run till they drop dead in order to keep up with the herd.
Any deer-appropriate caliber will easily put them down and I've killed a bunch with 7mm-08 and .308.
Don't overthink it.
