Hanshi's suggestion of an overpowder wad or alternatively, about 30 gr by volume of cornmeal, between powder and patched ball would be worth trying. Check the condition of the recovered patches. They shouldn't look scorched at all. If they're cut or torn that would suggest there's sharp metal somewhere compromising the integrity of the patch material as it is pushed down or shot out of the barrel.
I recall reading somewhere that if the patch holds together like it should, hot gasses can't get past, the hot gas produced by the powder burning can't move any faster than the ball, i.e. somewhere around 1700 fps for a typical load. But if the patch fails, the hot gas can get past the ball at a much higher velocity, maybe 10000 fps, in a narrow jet, and a narrow stream of high velocity hot gas can be rough on the barrel, regardless of the effect on accuracy. An overpowder wad or cornmeal largely eliminates that possibility, but at the cost of the inconvenience of an extra component in your load.
BTW, I have a .40 that ate patches like they were popcorn. After trying different thicknesses of ticking, different lubes (7:1 water to olive oil was my usual recipe for my other guns) I found a spit patch would survive. Barring the patches getting cut on sharp metal, I expect you just need to keep experimenting to find what the gun likes. Good luck, SCL