"It" happens. To some with less experience in shooting, "It" happens more often.
Sounds as if you are doing everything right & I agree with not checking the group until you are finished.
Patch condition before and after is of vital importance. I personally have not checked ball weights since the 70's
when I decided at the ranges normally shot with round balls, it was likely a waste of time. Now, if in competition at
200yards, I might weigh the balls, if I remembered to do that, which at this stage of the game, not likely.
Muzzle condition/shape to not cut the thick patch upon starting - check ALL spent patches to ensure patch integrity.
Sounds as if you are doing everything right. Some folks need specific sight shapes to shoot well. Different bull sizes
makes a LOT more difference in the final groups size than one might think.
Usually the smallest bullseye you can see over your sights - having to REALLY concentrate on the perfect sight picture,
will lend the smallest groups- at any range. Double bagging helps, with some rifles being really picky where the rifle
is rested on the bags. I used to be able to shoot well at range, with hand held with my hand rested on the bags. Now
I shot better groups with the forend and underside of the butt on the bags. Heart rate (side of the cheek against the stock)
is a bummer, only seen if using as scope & is good for 2" extra on the group size if the shot is taken at the wrong time.
It's aggravating.
Here's a reproduction of a group at 100 meters with my 14 bore rifle made by Taylor GRRW barrel, 66" twist. Fliers happen.
This one is from an original Joseph Lang 16 bore rifle, 48" twist.