If the gun is held differently shot to shot, the ball will impact accordingly. The longer the barrel, the greater the need for perfect follow through.
Normally when shooting off a bench, even though held in both hands, the flinch disappears - once a fellow is used to shooting a flint rifle. I find using the bench technique as noted above, I can trust my sights if sighted to the bench grouping.
Once sighted in, you can try resting the gun at different locations along the stock, letting the gun rebound and observe the point of impact change, if any.
It is possible there will be none, depending on the particulars of the stock, your own body and how they fit together. If the butt is positioned too far out on the arm, ie: stock too long, this alone can have the effect of throwing shots to the right (right handed) as the gun 'throws' or 'moves' that way under the start of recoil. This is why I like to hold flat or shotgun butted stocks on my pectoral muscle or between pec and shoulder, resting on muscle, not arm - recoil comes straight back and gun stocks of shorter length are easily shot accurately. Once recoil is taken up by the arm rather than heavy body muscle, point of impact can shift- easily and quickly.
Another rule, both feet flat on the ground. Think solid.
I still pull the odd shot off when shooting on the bench, but it is rare now. There is no solid reason for it, just a twitch when it shouldn't happen - and usually on the 5th or 10th shot of an otherwise stellar group.