A lotta times if I'm moving the jacket ends up becoming unneeded weight in my pack.
That's true, however...this is assuming you can keep moving. A sprained ankle in a cold place will require a warm jacket in your pack if you hope to get home. A puffy jacket, either down or synthetic, doesn't add much weight and will compact a lot. it's good insurance.
