We get some pretty big paper wasp nests around here. Had one a few years ago, 15 inch diameter and 8 feet off the ground. When October rolled around, the wasps all left. The nest was vacant. It is possible that will happen with your nest. Just wait for late Autumn and the bugs might just go away.