Although I like the little guys, 32 and .40, along with the .45- I'd go with a 10 pound .50 cal. as a do-everything gun. A .50 has a theoretical advantage with the longer targets. Many of the shoots have longer targets and for this, the .50 is a better choice. I am referring to targets past 80 yards, to perhaps 200.
For cheap shooting, hardly anything beats the .32, it's 42gr. ball with 35gr. of 3F. It rings targets quite well, but doesn't move the larger or heavier steel ones well and a close eye has to be made by those scoring hits. Binocs can help. Our own trail walk shows the .32 gives up little to the .40's and even .50's - except on 100 yard and further targets. Hearing the hits is difficult as the velocity is quite high with gun sound and impact being close together makes hit registration difficult. A .45 might be a good comprimise, but a .50 uses very little more power for a perceived range advantage- especially in the wind. It must be fed more than 60 gr. to show any advantage over a smaller bore.