Our land progress over the past 40 years started at 3 acres, then 40 (with a 1000 acre Trappist Abbey forest on a couple of sides) and currently 300 acres with the real BLM on two sides and no near neighbors. We would buy more if we could.
When looking for land (and we looked for 5 years for our current place) we focused on a secluded build-able location, water, easements and neighbors. In retrospect, we wished we had purchased the 80 acres to our south (about 1,3 miles from our house), but we were too conservative.