The only potential down-side might be getting crudded-up or frozen in foul weather, but I've never had that problem, I just go home.
And here friends, is an experienced woodsman that knows up from down. There are times to hunt, and times to put your feet up by the fire.