If the recoil isn't a issue stick with your current load.
Being familiar with the trajectory of one load that will work from small to big is a profitable thing.
If you want a better deer round I'd change cartridges rather than the bullet/charge.