I have a blue hooded sweat jacket I wear in cold weather, in GA that is below 40 degrees. I would estimate I've killed over 30 deer in the last ten years with that same old blue sweat jacket.
Back in the 70's when most folks wore Army surplus BDUs, I had a buddy that wore blue jeans and a grey sweat shirt. I've seen him kill many deer that way.
I'm of the opinion that scent, and movement are more detrimental than color.
