But not as bad as a .577...
I inherited Dad's .458 #1, which has the same contour barrel, and is a basically the same rifle. It's energetic with full power loads, but tolerable.
Now the Ruger Rigby can take warmer loads than factory, basically you can load it much like a Weatherby. It will kick the same, too.
Barnes has a 350gr TTSX, that oughta shoot pretty flat in a Rigby.