If you can't see what has happened where the load rested, perhaps you should unbreech the barrel so you can look at the area in question in detail. I doubt the load has done structural damage making it unsafe to shoot but there is no substitute for "knowing" for certain.
Also, unless you have a religious issue with them, a CO2 powered ball discharger is a much safer way to unload most guns rather than pulling the ball. Or try an air compressor. On a percussion gun a grease zert in place of the nipple can be used with a manual grease gun to force a load out as well. If you must pull the ball, use a T- handled rod so you can tie the rod handle to a tree or other anchor point and you pull from the butt end of the rifle. (I have a friend who experienced a load going off while he was trying get the screw into the ball to pull it - so I know it can and has happened.)
The next time you need to kill a powder charge, remove the nipple, remove the barrel from the stock and sit the breech end of the barrel in a bucket of water for an hour or so.