Brokenhand, a few thoughts come to mind. I took a weekend class with the Marine Corps pistol team back 30 years ago, and took away some useful offhand tips for rifles or pistols. Here are a few (these do not apply generally to fowlers):
Stare at the front sight. When you can see the corners clearly, you are concentrating enough on the front sight. You cannot focus on three things at once, and the front sight is the key to see. Young eyes can shift back and forth fast enough that it seems like one can focus on multiple planes, but eyes cannot once older.
Accept the wobble. Everyone wobbles offhand. Accept it as the size your groups will be on that day. If you try to "catch it going by", you'll yank the trigger and pull off target. Rifles can hide this somewhat, but there is no hiding from trigger yanking with a flintlock pistol.
To follow that up, press the trigger directly to the rear by increasing pressure. The break should come as a surprise. The desire is to release the sear without disturbing the sight picture. I will say this is easier with a lighter trigger release.
Ignore any shakes. Like from too much coffee. There are shooters who win matches with the shakes. It saps your confidence, but does not hurt your group size. Ignore the shakes.
Don't lean back when shooting offhand. Lean forward. This steadies up the back, and allows you to ride with recoil.
Now for some of my own thoughts:
Concentrate on hitting your target, not the flash or report. I have been blessed to help some newcomers to muzzleloading. I always start them on a big plastic bottle up close. They are thinking about hitting the bottle and making it jump, not the pan flash. Once a rifle/pistol is sighted in, I leave the bench and practice offhand. I like clay birds on the berm at 50 yards as targets. I don't always hit them, but I have fun and it makes for useful skills. Starting a newcomer on paper (in my experience) makes them think about the flash and recoil, and develops a flinch.
Glasses. I went to CVS when my eyes started to fade at 40. I bought an assortment of inexpensive reading glasses. I have them in a baggie, and use whichever pair allows me to see the front sight best for a given gun. I select the one that "just" allows me to see the front sight. That allows for a less blurry target than a stronger-than-needed pair. A trick for purchasing the right pair is to take a yardstick shopping with you. Hold it at arms length, and tape a small sign to the yardstick at the same distance from your eye as the front sight (do this at home first). At the store, try on different glasses until you get the right pair that allows you to see the small sign's lettering. I also then buy one strength up and one strength down.
The reason I love swamped barrels is that they seem to "hang there" offhand on-target all on their own. I cannot explain it any other way.
My last thought is to make sure you have a good, fast lock in your gun. A touch hole of about 1/16". From Pletch's high-speed photography test, I use 4f in the pan as it ignites faster than coarser grains. NullB is a little faster, but it plates my pan and lock in graphite fouling. I need all of the speed I can get to have the ball exit the muzzle FAST when I am shooting offhand. Yes, 2f will set off a charge, but will be slower to ignite the charge by a fraction of a second. That fraction may allow me to wobble off target before the ball leaves the muzzle. I know, I know. There are folks who will argue their gun does just fine with 2f in the pan. I have not seen any of these folks say they have done scientific testing that says 2f is as fast as 4f in the pan. Pletch did the testing. I am just sharing the info.
I hope this is helpful to you in some way. Not trying to stir up any animosity. God Bless, Marc