Sounds like you need to file a little off the front sight and tap the rear sight to the right a little. However, first I would settle on a load. For a 54 caliber, I would shoot somewhere between 75 and 85 grains of FFg, or if using FFFg maybe around 70. Keep using your .530 roundball and try a pillow tick patch. If you are trying to sight this in for deer hunting, you want it stick with what you will use to deer hunt with. Use a patch lube that is not liquid. (no spit) Lastly, if sighting it in for 50 yards, you want it hitting in the 12 o'clock position.
When trying to work up a load, get it in the ball park at 25 yards before moving out. Also, you dd not want to make any sight adjustments until you can cover a 3 shot group with a quarter at 25 yards. Once your groups are consistent, then that proves the bugs are worked out.
Brian