I've never hunted deer with anything less then .45 PRB and have also used the .50 and .54 PRB for taking deer and antelope. They all work and shot placement is everything. Roundball goes for the heart, I like to go for the lungs. Just two different schools of thought on where to hit 'em, but both work provided we've done our part and put the ball where it needs to go within our chosen range of good marksmanship acceptance.
I spend a lot of time at the range shooting my .45, so I do prefer my .45. I've yet to make (in my book) a poor shot on deer size game with any of the three, .45, .50, and .54,,, knock on wood, and I do feel a successful hunt is directly related to range time with any caliber the hunter has chosen to hunt with.
I'm not saying I will never make a poor shot because it can happen in a heart beat to any of us. I just feel if you're a hunter and prefer a certain caliber round ball over another and you have confidence in your shooting ability from your range sessions, there's no reason not to use what is legal in your state, be it a .38, .40,
.44, .45, and so on. Confidence in your chosen muzzle loader and your shooting ability has a lot to do with making the good shot that will put meat on the table.
Just my thoughts.