While the .40 is light for deer hunting, the problem in using it isn't the ability of the hunter to put a shot in the boiler room; if one can't do that, a .54 won't help. Shot distance should be the primary consideration.
this harks back to those who'll bring up the aleutians killing moose with a .22 rimfire.
lightweight calibers increase the need to drop the pretenses and recognize yer worth as a hunter who can get really close to their prey, and be a good, ethical shot whilst in those hunting venue conditions.
a .40 for mule deer will require a lot more hunting skills and flat out luck. one could put in the time and effort to train with a .40, as well as recognize some shots will need to be passed up. one might also wanna compare the cost of such a hunt versus its expected outcome.