Similar experience here, started with vise block for upper and then bought reaction rod. No comparison.
Roll of blue painters tape to protect finish.
Absolutely use job specific roll pin punches as suggested.
Roll pin pusher for trigger guard is a good tool also.
roll pin pusher

Slightly off original question, it is wise to order more small parts than your build requires. Meaning roll pins springs detents etc.
There are dimension variances for every part and from every brand.