I am getting new iron sights on two rifles of mine.
I am looking at Fiber optic front an rear sight.. and they come in several heights.

Is there a formula or something to figure out how tall I should order?

https://dawsonprecision.com/sight-calculator/

https://www.brownells.com/aspx/learn/learndetail.aspx?lid=13186

If the notch or aperture of the rear sight is the same height above the bore centerline as the front sight your sight line will be parallel to the barrel and you will hit low at all ranges. In this case you would need enough up adjustment in the rear sight to climb out of the low zone. Try to deal with someone who knows which sights go with your particular rife.

Didn't your high school trigonometry tell you that some day trig would come in handy?