This is one of those lose lose situations you get when you don't work together with your neighbors in a rural area. You let your neighbors know your planned use, ask them what you can do to make it work for them (limited flights and hours and avoiding homes on the path, the Fred a ride in a medical emergency) feed them at a BBQ and see if that works. At worst your neighbors think you are an ok guy and don't look to cause problems for you.
If that doesn't work screw them and build a long strait road to your new big barn that happens to fit a plane.
