I do it both ways, depending on how I'm feeling that day, or if I get excited about another aspect of the build in which case, the pins get left behind for a while. Once the forearm is shaped, you can eliminate the wandering drill tip if you have driven a prick punch into the spot where the drill will go, and running the drill bit at high speed, introduce it very gently against the wood. It will want to follow the punch mark, and once its started, away you go. There is always a little more to take off the forearm after drilling, so the holes clean up very nicely.
On a longrifle that is getting moldings along the forearm, I sometimes use the pins left extended, to support the straight edge that I use to guide the scribe for the first cut for the moldings, going from the entry pipe to the muzzle. I don't use one of those molding scrapers I've seen illustrated here, but I see they work really well too.