Are you patching them? If you want to have an easy loading combination and still use a thick patch, perhaps a .580" ball would be better. Generally ..020" to .030" smaller, for a smoothbore.
We run .005" normally for rifles, but odd time, depending on the dimensions and moulds available, I've gone to .562" in a SxS .58, as well as .682" in a .69 rifle.
If wanting to use harder than pure lead, then a slightly smaller ball is advantageous, even in a rifle. I use a .675" hard ball in my .69 calibre Rifle (with .034" patch) & have even tested .662" balls in it, by reverting to using doubled patches.
The accuracy was actually quite useable, at 50yards. That would measure about 1 1/2" for 5 shots.