IMO, being fit & flexible matters more than size in terms of recoil tolerance. 'Tis much less punishing to be fit and be able to "move" under recoil than to be huge / stiff and absorb the full blow with your body.

YMMV.