People worry about this stuff, but have they ever tossed an old couch or living room chair on a bonfire and saw how fast all the plastic based materials it's made of go up?
Or, better yet, been involved with the fire service and been through the training or a few house fires?
Unless your powder and ammo is THE seat of the fire, what amount most reloaders have in storage won't make a tinker's damn to the outcome of the situation. Your entire house is full of potential accelerants.
