I have a .25-06 AI, used to have a .25-06. I also have (2) 6.5X55's. I would go with the 6.5X55. It's not so much an issue of recoil as muzzle blast. Neither will give you a great deal of recoil, but the (to me) considerably greater blast of the .25-06 will be more intimidating to an infrequent shooter. I've never hunted elk, but if I were going tomorrow I would be happy taking either, but would more likely take the 6.5X55.
