I find that TVM, Tip Curtis and the like fills a (void) in the muzzle loading rifle community between the "off the shelf" offerings by Cabela's, Lyman etc and the custom builds that can take many many months or years from the "one off" builders.
Please don't read that I'm knocking them - I'm not. They produce a "generic" rifle in an (almost) semi-production style. An early Virginia (or late Lancaster - pick the tag) from one of these does incorporate a number of the characteristics from a number of builders in the area but does not look much like a Dickert or a Sheets etc.
I sometimes wonder how they make any money from their builds. The cost of parts is barely below the cost of their "no frills" offerings. I think you will find that just like many "package deals" that if you start "adding on" or "upgrading" the cost will quickly grow and could well equal what some of the custom builders charge.
I personally would be hard pressed to recommend "going all out" with a TVM build. Yes, it might have a swamped Rice barrel and very fancy wood (at significant extra cost), but it will essentially look "exactly" like the lowest priced model except the wood will be fancy (with a custom builder, even if they make 10 Lancasters they will all be "unique").
Your TVM (place tag here) will look like every other TVM (with the same tag) which is why I would not pay a premium price. I would suggest your "best value" is in their (less fancy) package offering. If you are willing to drop more than 2K (which it could quickly add up to), find a custom builder and get a "unique" rifle.
(enough preaching)
You mulled between 50 and 54 and ask about 45 cal.
While I own a 50 it's stays mostly in the gun safe. The 45 is my "big bore"
- my everyday rifle is a 40 cal and I will be replacing that (soon hopefully) with a 38 cal - and YES, I will use that for deer because where I hunt it is legal and the way I hunt, it is way more than enough caliber.
Many consider sub-45 cal inadequate for deer (and I hunt Eastern Canadian Whitetails - so not deer the size of Golden Retrievers, we have some Big Bambi's), but hunting medium game with small calibers is a kind of finesse hunting that you work your way into and I wouldn't feel right saying that everyone can or even should try hunting deer with a sub-45 cal rifle.
On the other hand, a 45 cal is all you will ever need unless you hunt bear/moose. And even then I personally think a larger caliber would only really give you a "mental advantage" - but that's not really what you are asking and really is a whole other discussion.
A 42" "B" weight 45 makes a nice slender rifle - A "C" weight 54 would be my next choice.
I only own a 50 cal because that is about the only caliber that you can find supplies for in Canada "off the shelf" in the handful of places that carry anything for black powder at all (other cals require long drives or mail order) - otherwise, if supplies to feed it is not an issue, a 50 gives little to no advantage over a 45, so if you are going bigger, may as well go to the 54 cal which will result in a lighter barrel in the same profile.
And if you carry the rifle any distance while hunting, a half pound can be quite substantial.