As someone who owns four Ruger #1's & four 1885's, I would recommend avoiding the #1 itch. It can be addictive. I generally don't shoot 1" but that has more to do with my level of skill than the quality of the rifle. I can put deer in the freezer but I'm not going to win any target shooting contests. My continued success with these rifles is based mainly on the moderate ~150 meter shots I usually take in the southeastern woods.
As you probably know, the Ruger #1 was chambered in lots of calibers. Based on your criteria, good choices might be the 6mm Remington, 243 Winchester, the 25-06 you mentioned, 250 Savage, 257 Roberts (rarely inexpensive) & an assortment of 6.5's. Some people have had good luck hunting deer with smaller rounds like the 22-250 & the 223 but personally I favor larger calibers. The smallest #1 I have is a 6.5x55 SE in a 1A model which I really like a lot.
The 25-06 you were considering is often found in the B or V models or even the KB model in stainless with a laminate stock. If you can locate one in an AH variant, then that could be a really nice lightweight option. As someone mentioned, Browning made the B-78 in 25-06 as well but all of the ones I've seen have a right-hand buttstock with a cheek weld so just be aware of this if that makes a difference. Good luck with your search.
