I don't even allow WD-40, ArmorAll, or anything else with silicone in it to be brought into my shop. Even though I use WD-40 as an initial water displacer (hence its name) after the last carding in a rust bluing job, I take it outside for that, and then go straight home with the parts so as not to allow even one molecule of the stuff to invade my workspace.
I have used ultrafine bronze wool to rub finishes out, but that stuff is really hard to find. I usually resort to good old rottenstone, but there's a learning curve to its use for uniform results.
