That's because the bullet is a relatively smaller part of loaded ammo, and producing hundreds of thousands of bullets, in a relatively limited caliber/weight array for common factory ammo, also lowered per-bullet costs.
Also, handloaders shoot a lot more than hunters who buy factory ammo, and most deer-hunting handloaders aren't willing to pay even a few bucks more for 100 bullets when there are cup-and-cores right next to 'em on the shelf that cost less.
It may seem odd to Campfire members, but a lot of handloaders still buy whatever 150-grain spitzer is cheapest. And they kill plenty of deer.