The entries in my old Lyman Black Powder Handbook for muzzle velocity vs powder charge for a Colt Baby Dragoon, .31 cal, show that they used a .319 diameter roundball.
The 'General Info' section of Dixie GW catalog lists .320 as the diameter for a .31 Colt. In the product description section of the catalog, for a Uberti replica, the load recommended is 12gr 3F, a revolver wad, and a .321 diameter RB.
A No. '0' buckshot might be worth trying.