I don't want to be the voice of "not so great news", but no one has yet mentioned barrel runout or a bent barrel.
What range are you shooting when it is 4.5" off? How long is the sight radius? If you are 4.5" off at 25 yards, I don't think loads will make a difference.
Did you make the rifle? If so, did you check for runout before building?
Please forgive me if you already know this, but runout is where the bore is not centered within the outside dimensions of the barrel. In other words, the outside of the barrel is pointed at the target, but the hole down the middle isn't.
Makers stamp the barrels so that the runout can be oriented up or down. That is no big deal at all, as the height of the sights will take care of that alignment. The makers' mark usually goes on the bottom flat.
If the runout is side-to-side, you'll end up with the sights not centered on the barrel top flat. In minor runout cases, the adjustment of the sights on the top flat is barely notable. In bad cases, the sights are hanging way off to one side or the other. Another fix is to file the sight notch offset, but that rarely looks good either.
You can check the math this way:
Sight radius (in inches) is to rear sight movement, as the distance to target (in inches) is to the target error in inches. So, for a 30" sight radius, 900" to target, and 4.5" off of aiming point.....
30" x 4.5" divided by 900" equals move the rear sight right by .15 inches. Not a small amount. You'll see that.
30" x 4.5" divided by 1800" (or 50 yards) is logically .075" rear sight movement. See why the distance matters?
I hope this helps a little, and I sincerely hope you do not have a case of runout.
Best wishes, and God Bless, Marc