When I have used the brush, it was because I wanted to remove a fresh charge of priming powder from my pan, or had (somehow) made a mess while priming and needed to clean up that area of the lock.
Shooting down here in Florida, I've developed a regular routine between shots.
I use a clean patch to wipe the pan, wipe the face of the frizzen and wipe the edge of the flint. Between EVERY shot.