The first key to actually staying warm when it is actually cold out is stopping the wind from robbing your body heat. Stop the wind with a light external layer, then build what you need for insulation under that.