Go for it.
I had 3 good bucks I had to let go the last day of deer season because while I could see horns with the binoculars I couldn't see them with the iron sights on my .45-70 marlin. I was in heavy brush, there were bucks and does both, milling around, changing positions. The first buck i saw, I put horns on, good rack. dropped the 'nocs, brought up the gun, decided to slow down and take a 2nd look to confirm. Wasn't a buck anymore, they'd changed positions and I'd dang near shot a doe I didn't have a tag for. After that I got even more cautious. Never could get a shot I was confident of. Even with a 4X they'd have been easy, real easy.
So, IMHO it's not only ok, I'd recommend it. Personally, I'm done with iron sights on rifles.
Tom