I've been hunting with braked rifles, even non-magnums, almost exclusively since the late 1980s. I wear comfortable in-the-ear-canal noise-dampening hearing protection all the time. No regrets.
I agree with Jordan above. Being able to stay on target, even to see the reaction on impact (even, sometimes, discerning a clean miss vs. a hit) is well worth the extra noise to me.
Invest in good hearing protection. IMHO, don't worry about the brake.
In my experience, POI may or may not be affected, when replacing the brake with a thread protector, depending on the rifle/load.
