Do what you want. Since my shoulder replacements, 99% of my shooting has been 22LR/223/9mm. I'd keep 1-2 in the others that you have a feeling for and dump the rest. I disagree somewhat on keeping oddball calibers around for tough times when you can't find the regulars. As an example, if you take the money that you would've spent on a .40 and used it to stock up, you'd have a spare 1-2 cases of 9mm sitting around. Just make it your emergency supply.
