My two cents. I’m currently on gun number ten from a blank; so I feel like I might actually know what I’m doing one of these days.
Leave all the wood while you drill.
I keep my drill bit lightly lubed with beeswax. And I only drill about ½ inch before withdrawing the bit to clear out the chips. It’s important to keep the bit from wandering. Once I get 3 inches drilled, I drill 1/16” feeler holes in the bottom center of the barrel channel (about every inch and a half) as I progress. They allow me to monitor the web thickness and any wandering. In ten guns from a blank, I've only had to finish the ramrod groove from the barrel channel on one of my early guns. All other ramrod holes have ended up within 1/16" of where they're supposed to... if they're off at all.
You didn’t mention your ramrod groove. For me, the key to a successful ramrod hole is a precise ramrod groove. Centered, straight (no high or low spots), with precise web dimensions all along the groove.
I don’t know what others do; but I don’t install barrel lugs and pins until my trigger plate is in and the tang bolt installed. So, for my sequence, there’s a lot of work between the barrel and tang and the barrel lugs.