I wouldn't sweat it. I did the same thing on one of my rifles. The drill bit walked and the hole ended up higher on the barrel flat than my punch mark. I chalked it up to experience and finished the rifle. I figured I could correct the position with an off-center vent liner if I had trouble with ignition. Not only does the rifle fire reliably, but it may be the fastest firing of all my flintlocks. Now I drill a pilot hole first. Lesson learned.
BTW, I like that vise you are using to build the rifle. I've never seen one like it. What kind is it and where did you get it?
Thanks for posting the videos.