If you are cutting molding lines down the forearm, having the pins out of alignment can be a problem. Other than that, it is just a little flaw that irritates some.
I drill my holes for barrel retaining pins to fit a finishing nail that measures .073" with a #46 drill bit. I use 1/16" welding rod pins for thimbles. I drill my holes all in one go using the highest speed my drill press will generate, and advance slowly, clearing chips frequently. I just did the two barrel pins on a Jaeger build yesterday, having marked both sides of the stock for location, and the drill came out exactly on the marks on the far side in both cases. A sharp drill, and a light touch, and high speed, is the ticket, for me, at least. Oh, and I drill the forearm while it is still square, when I'm building from the plank.