I don't think the lead has to take the rifling for the rifle to be accurate. Roundball's experiments with brass balls are pretty convincing, and many report good results with wheel weight cast balls. Then there are the period accounts of using "scratch" rifling to cheat in smoothbore matches. Apparently any spin is adequate for a round ball. With this rifle, I would try a slightly bigger ball and a slightly thinner silk patch based on what I've read from the caplock era. It doesn't take much to fill the narrow grooves and excessively thick material just gets in the way. Just an observation.
Also, Taylor is likely correct about the rifling not being as deep as it looks at the muzzle. It is unlikely one of the biggest makers of rifles in the period produced barrels that weren't any good at all!
Finally, the 75gr. group is starting to show promise. By eyeball, it appears that it is around 2" if you assume the one at upper right is a flyer. 70-80 gr. 2F has always been a good spot for my .50's. 75gr. 3F should be even "more adequate". Given that it is an old barrel (though it looks plenty good and sturdy), I'd look for a better patch and ball rather than going much higher in powder charge.