A .44-40 from a 26" barrel ought to be about as quiet a deer rifle as you can get, barring suppressors.

A professor at Ball State University was to write a book on how various rifle types affect hearing damage. Some dB data which he sent me, as measured at the shooter's head, was surprising. The .22 LR was still pretty bad for your ears. Sadly, I don't think he ever finished the book, and I haven't heard from him in years.