Agree with Mr. Raab - a vest does everything you wanted.
LL Bean has wool one, I've worn mine for years and when it is
wet weather pull a rain coat over it.