At 20" a 308 may not lose enough speed to matter. The fastest 308 I own is a 20" Tikka CTR. It shoots 20-30 fps faster than my 22" guns. A 30-06 will start with more speed. But when cut will lose more. At the end of the day, I'd still guess an 18-20" 30-06 will still be faster than an 18-20" 308. But the difference won't be as much as it would be if both had 22-24" barrels.
It wouldn't be worth 20-25% more recoil from the 30-06 plus it will be louder. I'd stay with 308 for what you want to do. To my way of thinking 30-06 doesn't start to be better than 308 until you get to 180 gr and heavier bullets.
