The gun hits to the left at 25 and 50 yds with a 50 grain charge, but to the right at 100 yds with an 80 gr charge? Glad you got that problem and not me!
I'd say, take a deep breath, clear your mind, and start all over. Weigh out a couple dozen roundballs so you know they're within a grain of each other in weight. Shoot 3 shots off a rest at 25 yds with 50 gr, then 3 shots off rest at 50 yds w/ 50 gr, then 3 shots with 50 gr at 100. Center hold for all shots. Label the holes. Bump the charge to 80 grains and repeat. Maybe even repeat the test with an intermediate charge, maybe 65 grains. Using the same charge and hold at all three ranges will give you a hardcopy record of how the ball is moving R/L and U/D with range.
Recover a few patches and examine for any signs of them getting blown or burned. Anything suspicious, try different patching or lube. Or, try 50 gr by volume of Cream of wheat between powder and patched ball. You're looking for something that will make a difference - find it and you can maybe figure out why it makes a difference and fix it.
Don't know what you'll see, but some pattern may present itself. If nothing else, you'll have some organized data on paper you could scan and send to Rice Barrel Co to see if they can suggest something.
Good luck, SCL