Once I started removing, cleaning, and oiling, my problems went way down. There was never anything inside, but flushing off the frizzen pivot, pan, roller, etc., removed stuff better than wiping or brushing. The other thing I do that is mostly superstition is run fine sandpaper over the frizzen a few times with each cleaning rather than waiting for scarring to become a problem. Since I started doing this, I get a lot more shots per flint and need to knap less, and both of my locks, a durs egg and a chambers late ketland, seem to be more consistent and less picky about flints.
I always remove barrels and clean in a bucket, except in camp, and sometimes then.
Just my experience, do whatever seems to work for you.