Does your state have laws/regulations regarding minimum and maximum caliber size for small and large game? For instance, here in CT. I can't use a muzzleloader larger than .36 caliber for small game, and not smaller than .45 caliber for large game, and can't use a rifle at all for birds (turkeys or partridge), making a .40 almost useless here. But, I can hunt anything with a 20 gauge smoothbore, loaded with a single roundball for deer or shot for birds and small game.
If I had no restrictions like I do, I'd go with a .45, but I'd be hunting deer too. No deer on the agenda, I'd go with the .36 as I don't want to fuss with anything smaller, especially when it's cold.