I might be the only one on this website to ever admit this: I have 1 rifle and I use factory ammunition.
I only use factory ammo. For years I only had one centerfire rifle (also had a 22 LR). It is a Remington 700 30-06. I could have never bought another rifle and would not have had any less success. I have fun playing with different rifles but if you buy a versatile chambering, you only need one.
