I've got 32 and 36 cal, and when you get the load dialed in, there's really no difference in them for cleaning. I can go all day without swabbing, sometimes running up close to 20 shots. With any of them, it's a headshot-only deal, simply to save meat. The accuracy has to be there, and for the occasional shot that wanders down into the neck, it's really worth exploring reduced loads in all calibers. I'm down to 20 grains of 3f in both my rifles, and 15 grains is next. A neck shot with 20 grains in either the 32 or the 36 usually ruins the front shoulders too, and you don't even want to think about a body shot with 20 grains.
It's worth tossing out that our game laws don't restrict caliber for small game or large, so as often as I use my 32 or 36, I use a 50 or 54 just as much with 30-35 grains of 3f. I haven't used one of my 58 simply because I haven't worked up reduced loads yet. It really doesn't matter which you use if you're taking head shots.
I'm real fascinated right now with the idea of making a 45 as a combined small game/deer rifle. I think it will be more versatile in that role than a 40, but I kinda have the twitch for a 40, as well. Heck, I've got the twitch for a 69 rifle too, and when I get it, I bet I use that for small game too.