In my mind "casual wear" is not the same as work wear, so many of the suggestions above would be off base a bit. Casual would be puttering around in the unheated garage, raking leaves, watching a high school football game...stuff like that. It just needs to be warm and somewhat wind- and water-resistant.

Cotton is out if the weather gets wet. Synthetics are out if there's a chance of sparks. Wool is resistant to both, so it's always a good choice.

With an appropriate thickness shirt, I find that a good vest usually handles things. I've owned and liked puffy ones, canvas ones, wool ones, and fleece ones. I like fleece best, but have a warm spot (puns intended) for a Filson wool one. Vests zip or button to regulate temp, have two to four good pockets, and don't restrict arm motion while keeping your core toasty. Some are windproof in addition to warm. Canvas ones can be treated to be at least water resistant.

If it gets too cold or wet for a vest, just slip a coat or rain jacket on over the top. Now you have everything, except a good hat. But that's a whole different thread.

Cleverly disguised as a responsible adult.