My rule of thumb is to match ball diameter. It's just easier to gauge beforehand, turning out just right since I cut my patching "at the muzzle" while loading the boards. By the time you seat the ball a little below the board for the reasons others have described here, the top is set down just the right distance for cutting the patch. I do my thinning using the disk sander on my bench, BTW.
As for hole diameter, I want a pretty snug fit- requiring a little more than thumb pressure for seating. That keeps the balls from getting knocked out on a hunt, and meanwhile greatly eases seating into the barrel. If you're careful how you seat them in the block, you can be pretty rough and tumble about how you drive them into the bore in the field, yet still get a good load.