I prefer a sight pusher. I've seen people damage them with a punch. However, you can do it with a brass or nylon punch if you're careful.