You shouldn't have to brush & swab between shots, if your loading procedures are good.
SMALLER BORES CAN BE MORE DIFFICULT THAN LARGE BORES.
But not always.
To shoot well with any firearm, you must be consistant.
In the case of a muzzleloader, you start with your loading procedures.
Each shot must be the same as the last.
So if your powder, patch, ball, and ramming procedure are not consistant, your groups will not.
So if you powder and ball are the same from shot to shot, so must your patch and ball be the same. Meaning the fit of the ball to the bore, which is governed by the patch, with some form of gas seal, cleaning of fowling from shot to shot.
If your patch & ball combination is too loose, the patch will burn through, resulting in little or no seal, and increasing the fowling.
Many shooters will actually put some form of material between powder and ball to prevent burn through and assist lubrication.
If your patch and ball combination is correct, you will actually impart pattern of rifeling and patch pattern on the ball when loading.
A properly crowned barrel and short starter will greatly help loading.
The Dutch Shultz (spelling) loading process will make a you a pro at shooting your Pedersoli.
All the best!
Old Ford