I don't think it's the oblong pin holes at all - both my longrifles have round pin holes that are a snug fit to the pins - no wondering or changing of point of impact happens in a string of shots.
As a .530 bal with .018" is a snug fit - RB is correct I think in surmising it is the crown that is preventing easy loading.
Blackdog - watch the video- This is Taylor loading his .50 Virginia with a Rice barrel with rounded rifling - the load is 85gr. 2F, with a .021" denim patch and a .495" ball. The ball is only .005" smaller than the bore, yet it is easy loading. He is using a range rod, however loading is virtually identical using the rifle's rod.
Also- some time ago, Dan Pharis posted a short video of himself loading his .54 - using a .530" or .535" ball and .022" patch, without using a short starter - merely choking up on the rod and pushing it in. His crown was such that it alowed the ball and patch to swage into the rifling- just as Taylor's does, except Taylor just gives his a smack with the starter knob.
Factory crowns need to be smoothed about 99% of the time. Emery and your thumb are all that's needed to take the sharp edges off the corners.
Here's another one- I am loading a .445" ball with .0220" patch in my GM .45 barrel- .010" rilfing & using the 3/8" hickory rod. I-too have a nicely radiused crown - smooth corners that allows the ball and patch to enter easily.
And yet, another one loading the .58 double rifle. The load is an easy loading .562" ball with .0215" patch. Accuracy is OK - best of 1 1/2" at 50 yards, both barrels for 5 shots (figure that out), but better since I switched to a .570" ball and the same patch. Rifling is only .008" deep. It will shoot an inch even with 10gr. less than the best load, which opens the group to 2" wide, both barrels shooting an inch, parallel to each other.