Mauser06... Just guessing here, but I am betting your ramrod channel and hole are not parallel with the bore of your barrel. I think most of us, at least when beginning to build rifles, have experienced this. This is not catastrophic, but it does require thinking about how to approach the situation.
As some have suggested, you do have some metal you can take away, however, I would be careful of this. If this turns out to be an appropriate solution for you, you should probably solder a tongue on the tail so you make sure it doesn't move.
Myself, I would try and find what the actual issue is. I does sound like you have more wood in the area than is called for, and if that is the case, perhaps more work on the ramrod channel is called for. If so, it is likely that your ramrod hole is going to be off a bit too, but if you properly taper your ramrod, this may be less of an issue than you may think. I use a pair of dividers to determine my ramrod channel depth along the length of the barrel. This is easy when using a straight barrel, but with a swamped barrel, you will need to modify your dividers to allow for the thick area near the waist of the barrel. I put a simple notch in mine so I can go exactly one turn in one direction or another.
Probably the least acceptable solution is to inlet the pipe deeper, though it may be in your case the best one. Make sure the tail of the pipe is exactly bent correctly, and inlet it deeper, using a test ramrod to make sure everything is acceptable. A sixteenth of an inch here is quite a lot, considering most tails are only .030-.040". Perhaps more pictures could give us all a better idea of the situation you are facing.
Matt