At the start of a shooting session, have you been performing the ritual usually called "popping a cap" to verify that the nipple and flash channels are open before you load with powder & shot? If so, do the caps you're using provide enough 'oomph' to displace a little dirt or dry leaves on the ground?
Seems like either (1) your percussion caps are too anemic, or (2) the nipple has become plugged or is possibly defective, or (3) the flash channel in the breech has become clogged, or (4) like Knifemaker suggested you're wiping between shots and leaving too much moisture down there.
(1)To rule out the caps, I'd either try another brand or pop one of the caps you have in another long gun and compare how the cap makes dirt and leaves fly in that gun versus in your shotgun.
(2) For the nipple, hold up to the light and look into the flash hole. You should see a circle of daylight all the way through. I believe a hole diameter around 0.030" in the threaded end is pretty much standard. If you still have the old nipple, put it back in and see if you still have the problem.
(3)For the breech flash channel, with the nipple out and gun unloaded, you should be able to blow down the barrel and feel essentially no resistance to the air flow. In the absence of a brush such as TOF suggested, I'd run a pipe cleaner well soaked in solvent down through where the nipple threads in. With the nipple removed (and gun unloaded) maybe apply a blast of compressed air. If the barrel removes from the stock easily, soak the breech end of the barrel (nipple removed) in Hoppes 9+ or Mr. Flintlock or LVL or some moose milk concoction overnight, then hit it with compressed air, pipe cleaner, etc.
As to number (4), sounds like you've already considered and ruled that out.
I had a rifle with about 25 years service on its record. Over a 2 month period its ignition became more and more unreliable and eventually got to where popping a cap caused no reaction down at the muzzle. Took a lot of breech channel cleaning to get the fire flowing again, but when I finally got a couple of black flakes out, the difference was like night and day. Good luck, SCL