Most the time its because of the bit not being sharpened correctly , if its not started straight is another problem, putting too much pressure on the drill (ie ) pushing too hard and trying to bore the hole faster than the bit is cutting, and not cleaning the hole from wood chips periodically can contribute to the bit walking or so in my experience .
A lot of the ramrod drills i have are twist drills that have been threaded on the end and have a extension rod added for the length, and if they are not sharpened correctly they will walk a lot.
If your bit is dull and you are pushing too much the stock might flex enough to maybe be causing the problem
I normally leave the bottom of the stock heavy until after I drill the ramrod hole but a half stock plains rifle has a lot more wood to play with than a long rifle.
I would think that a good machine shop could sharpen your bit