Getting a ramrod drill will make life the simplest. They are only $34 bucks at TOW and are well worth the investment
+1 on that advice.
I had your exact same issue, rmatt, on my first 3 builds with PR precarves.
I fiddled, fudged, filed, and confabilated with my entry pipe till I couldn't stop cussin at it, never made much difference.
went and made a 3/8" RR drill (took about $10 & 10 minutes if you can MIG weld, or have a buddy who can.)
a quick pass with the RR drill in the precarve RR hole cleaned up the hole slightly and my RRs go in smooth and snug now.
RR hole
clean-out drill (this twist bit-type not good for drilling the initial hole I'm told).
(1) line up the pieces (bit and round stock - all right in Lowes/HD) in channel
(2) tack 'em together
(3) clean up the weld and make sure she's all straight.