Nice gun. Yes, the angle of the sun on the sights is well documented. Luckily for load development we are looking at group size, not placement.
How far was the target?
Have you tried no patch, just placing the ball between a lubed wad and a wad or card to hold it in? One of my smoothbores prefers a p.r.b. of .600 ball and .010 patch. The other would not shoot what I felt was an acceptable group at 50 yards with any patch combo I tried, okay groups at 25, then out at 50 they fell apart. But, oddly, the loosest p.r.b. combo I tried shot best, so I tried the wad and ball method I'd heard so many speak of.