I’d depends -
1. do you or your wife like cleaning .... if not don’t buy a bigger house.
2. Does your wife buy so much crap (Hoarder) until you are pilling stuff on top of each other ... smaller is better.
3. Big open spaces = more comfort inside, but high heating & Air Conditioning bills
We have up sized and down sized a couple times, I like the bigger open spaces but things can bite you in the arse.