Deeper grooves common to round ball guns will usually not do well with grooved lubricated bullets, let alone paper patched ones.
As Taylor has noted, paper patching is a skill and process unto it's own. Let's assume your bore is .400", with grooves .010" deep
making a groove to groove diameter of .420". About the only bullet that I can think of to work, would be the .38/40 bullet, which is
nominally .408", but likely cast .410" in pure lead. Th is bullet would have to be pre engraved in the rifling - using a section of the barrel
as a 'form die'. I've done this and it worked well, however takes some experience in this sort of thing.
I cannot 'see" this paper patching experiment proving anything about this barrel.
What the guys said about the bullets is spot-on. .38 bullets are .357/8" in diameter. 9mm bullets are .356". Bullets and round balls for
a .36 cap and ball pistol are usually .375". All of these are way too small to to patched to fit the .400" bore. With a .375" bullet, double
wrapping in heavy .003" paper will only bring them up to .387". Triple layer wrapping with .003" should being them to .399".
If you used a tight fitting wad underneath, THAT might work, but I doubt it.
You should be trying cloth patched round balls IF the rate of twist is 38" or slower.