Bought a woolen mix pullover from Cabelas with windstopper material in it years ag. Great article of clothing but probably would be a bit heavy for me where and when you are describing.
Have a lightweight jacket from cabelas also with windstopper that is good. Exterior is similar to a fleece so it is very quiet.
Unfortunately, cabelas stuff has gone down the crapper in the last 5+ years. But I am sure there are other brands that would fill the need.
I would look for a fleece jacket or pullover (your preference) with windstopper. I just searched and came up with a ton of results. If it is fleece exterior, it should be plenty quiet. Brands that I use these days include arcteryx, marmot, mammut, kuhl, sitka and kuiu. Wife sometimes slips a patagonia into the house against my wishes but they do make decent stuff.
Good luck
