There is no loss of accuracy and there is no buildup inside the bore. The combination I shoot gives considerable compression in the bottom of the grooves and wipes the bore as the ball is pushed down.
If there was a span of 30 or more minutes between shots in a contest, where the last shot's fouling might dry and become crusty and effect accuracy, (there is only one shot's fouling in the bore at any one time, mind you), I might wipe the bore - or not. It's an easy loading combination, as you can see. When shooting groups at the range, walking down to post new targets, measure shot ones, uses time, but I've not seen any 'change' in accuracy, however if the next load goes down feeling a big crunchy- I'll just shoot it off down range and load afresh - with no accuracy change.
Gus - when I first started in this game in 1972 I bought a .50 Cal TC Hawken with .004" rifling. Before 7 months had passed, I had settled on a load of 80gr. 2f GOEX, a .020 'brushed' denim patch (calipers) and a .495" ball. I had also picked up some "cut rifled" Bauska round ball barrels from Les (friend of Les Bauska) and by late 1973, I had one fitted to that rifle, with the stock modified to look less "California" made.
By mid '73', I had also discovered the nicely radiused and polished crown which allowed easily seating using a flat topped wooden starter with which I smacked the ball flush with one blow - no damage to the ball which was then nicely conformed to the bore. With the TC's .004" deep button rifled bore being .508" and the combination I was using making .535", the combo I used was a snug fit but managed to hold the skinny rifling and shoot very well indeed - generally a 1" diameter hole at 50 yards - I was trying to emulate the 'buffalo' shooters at Friendship with their heavy .50's and shot off sticks. I made the good groups but for me, it too a bench. I had exceptionally good mentors in the late Lester H. Hawkes of Telkwa, BC as well as the writings of Ned Roberts (the ML Caplock rifle) to go on. Both of these great mean taught me the path of using ball and patch combinations that did not need wiping while they were being shot - = because - When you load the next one, you are wiping the last one and to do that, the combinations had to be larger than the groove diameter.