Mine stays in. It's one of those with three ball sizes on it and the only time it comes out is to rotate it from 2" to 2 5/16". I didn't know anybody ever took them out. It got put in as soon as I got the truck home from the dealership and hasn't been removed since.

As for somebody damaging their vehicle by hitting my hitch, that's their problem, not mine. They shouldn't be anywhere near that close to my truck.