I use Imperial dry neck lube as well and only tumble clean the brass when I have to.
I've wondered about the guys that use a bronze brush to remove the powder residue inside the case neck but I don't bother so I won't find out for myself.
"Pick your battles" and "Don't make unnecessary work for yourself" come to mind.