That's because unless you have a really smooth barrel, patching isn't going to get everything out. So the brush knocks some fouling loose.

BTW, are you cleaning the brushes between uses?