Daryl, I've shot most calibers from .32 to .69 on targets - informally and at rendezvous. I would say the .45 or .50 is the best for punching steel plates at rendezvous.
If shooting paper past 50yards, I'd give the .50 the nod.
I really liked the .40 as an accurate shooting rifle, but did lose some 'points' because the deaf guys with me, didn't hear the little "clink" of it's 94gr. ball.
The .45 does hit harder and the diffference can be heard over the .40. The .50's 185gr. ball is very decisive.
Around here, the .54's really shine, but not as brightly as my old .69.
The .36 and .32 were worse for losing points, of course and not as good shooting at 50yards and beyond, as the larger bores.
An old timer here used to be @!*% hard to beat, especially when we were shooting at gongs at 200meters.
So- pick a calibre and learn to shoot it.
I would suggest .45 or .50 if going with a flinter.