Ken I wrote a reply but somehow it didn't send. Oh, well, here goes again.
Your accuracy seem very fine. As to the changing zero I can only guess. first, I and many others, don't recommend wiping between shots. It puts in another variable and is "artificial" in hunting. I suppose it's okay for targets though unnecessary.
Start by checking your barrel to see if it has any looseness in the stock, even a tiny amount makes a big difference. Try wiggling it and look for any movement. If the barrel checks out check all screws & parts for looseness.
Next make sure your loading routine is the same all the time; same charge, lube, patch, loading pressure, etc. Try to remove as many variables as possible.
It's common for rifles and handguns to have a lower poi when fired from off hand or from field positions than when fired from a rest. Check your shooting technique. it should be the same every time. Another thing, you MUST use follow through when shooting a flintlock! Keep the gun mounted after you pull the trigger as if you HAVEN'T pulled it! Keep everything the same for an extra couple moments. Flintlocks need follow through more than capguns do. This is a common mistake with first time flint shooters and those use to capguns.
If your front sight is brass or silver it can "disappear" depending on how sunlight strikes it. Glare can make it look shorter, taller or make it look like it's to one side or the other. Shade it if you can.
If you try all these and still have the same problem, kill a goat and sacrifice it. It will be out of your hands. Good shooting.
