I can't tell any real difference between them until you get up to the 180's in the '06.
However, I have a load for the .270 that uses a 110 grain ProHunter over a starting load of IM 4350 that is accurate, deadly on deer and has very low recoil.
Maybe a .243 would be an option?