In the situation the OP is referring to I usually just have a plaid shirt, with the heaviness dependent upon time of year.
If rain is expected I will put on a heavy wool shirt with raingear (usually cheaper stuff, but I am expanding my horizons). Kifaru Lost Park Parka if it is windy and cold.
I am intrigued to see what others use and recommend though.
