Make a rod from the same type of wood as the stock, if there's a risk of it showing through when final shaping. Fit it into the hole so it is somewhat snug, but definitely not tight. Drill a few air relief holes down from the barrel channel to let air and glue out. Apply Titebond II or III glue, and drive the rod in all the way (the glue will swell it all up, which is why it can't be a very tight fit). Once it is all dry, you can either try drilling again, or cut the ramrod hole down from inside the barrel channel.
If you drill again, drill a few witness holes down from inside the barrel channel. Calculate, and measure VERY carefully precisely where the ramrod hole should be. Drill about an inch and check. Drill about two or three inches and check. I find that if the rod is going to be off, it's going to go off quick and within the first three inches, I know if it's straight or not, and then I can correct it before it gets too out of hand.
To cut down from the barrel channel is not too difficult. Basically, rout it out the way some precarve stocks used to be. Drill it out with a series of holes almost all the way (when doing this, I still like to normally drill the first inch or so, and the last couple of inches or so. I don't like cutting down through the barrel channel very close to the breech). Clean it up and fit the rod in to proper depth with gouges and round files, etc. When that is done, then a piece of wood can be fitted to fill in the slot between the barrel and the rod. Fit it pretty snugly (with plenty of extra wood on top to provide you a handle which will be cut off later). Wax up a ramrod or a steel rod and put it in the hole, and apply glue to the sides of the filler piece and stock and press in the filler on top of the rod. Tap it in so it is snug on top of the rod, but not tight, so you can still get the rod back out. Let that dry then drill the last bit of rod hole all the way to the breech. Finally, cut down the extra wood to bottom back out the barrel channel. That may sound super complicated, but it's not. Sometimes (often) I just absolutely cannot get a rod hole to drill straight, and this is the only way I can do it.
The finished product: