If you are losing powder out the vent when seating the ball, perhaps it is leaving the charge, further from the inside of the vent.
Taylor had a rifle that needed a new vent, as we notices a difference in it's sound, shot to shot. Of course it did not shoot well, either.
A new vent fixed everything. I like vents from 1/16" up to about .70".
Try loading with the cock at 1/2 bent and the frizzen closed. Most people will frown at this, but it was the way ALL military muskets were loaded with paper ctgs, "by the numbers".
Cock to 1/2 bent, tear base of ctg. off with teeth, prime then close frizzen, insert the ctg. into the muzzle and ram it home, throwing the rod on the 'charge' 3 times.
Something like that, but in that order.
If you are losing prime as you seat a ball, then after loading with the frizzen closed, you should find the pan full. If there are only a few granules of powder in the pan, NP - carry on.