I can't tell you what you should do, but the reality is that small eastern deer at short ranges is an unconstrained problem unless a) you need a really short rifle for woods use or b) there's a straight wall requirement. Literally anything that will work for elk in terms of rifle and load will work otherwise. So I'd select an elk rifle and shoot deer with it most of the time.
Since a good chunk of elk territory is also bear territory, I like at least a 7mm and arguably a 30 or larger. Nothing wrong with hunting elk with a 6.5mm other than it being a less than ideal bear gun. The .25 and .27 bores are basically crap unless fast twisted and then bullets are still limited. Since I think .30 has the best elk bullet right now (200gr Terminal Ascent) I have one .300 WSM with a 24" barrel, and then a .375 WSM with a 20" barrel that uses the 300gr A-Frame. I'd probably use the .375 on eastern woods whitetails since it would waste less meat. Both are scoped with NSX 2.5-10s which again would be huge overkill in most of the east but on 2.5x would work just fine.
