It requires a bit of investment, but a few taper pin reamers of various sizes come in handy for a variety of hole size adjustment issues.  The ebony loading blocks below are for .32 and .50 cal round balls.  I drill the holes undersize and then use a taper pin reamer, from both sides, to precisely adjust the hole size to grip the ball (any size ball and any patch thickness) the way I want.  

I also use these reamers to open up the bores of cast ram rod pipes, etc.  The reamer in the photos below is a Number 8.  I use #s 7,8, & 9 the most and those three have been worth more than what they cost over the years in making hole adjustments quick, easy and precise.
