When I hear somebody say, "Yeah, the .500's not so bad," I have to question what they're relating it to, what kind of loads their shooting, or what kind of experience they're really had with it.
That's fine, I can understand your skepticism, but I have found that those who flocked to the .500 S&W as if it was an answer to some unanswered question, probably weren't familiar with the .475 Linebaugh, .500 Linebaugh, .500 Maximum, etc. as they would have seen the .500 Smith for what it was, a slick marketing ploy and not much else. Many folks bought into the muzzle energy myth and I credit S&W for at least getting more folks interested in handgun hunting.
However, keep in mind that .475 Linebaughs, etc. are typically built on 3-lb Ruger Blackhawks. The .500 Smith is a 4 1/2 + revolver empty, with a muzzle brake. The mass alone eats up much of the unpleasantness. I also get that for the uninitiated particularly, it's a pretty brutal gun, and relatively speaking it really is.
I have a .500 Maximum that weighs just a hair more than 3-lbs that is very unpleasant to shoot even for me, and I don't even need to go into the .50 Alaskan that jwp mentioned as that is in its own universe of brutality. And here it is, weighing in at 4-lbs, slinging 525s at right around 1,600 fps. Not fun in the least.