I'd stay with the .54. I don't see any advantage to going bigger than that.
The "Hawken"style I made years ago was .54 and was a fierce performer at 100 yards.Bill Large barrel and the rest was mine.
I think that ball weighed 230 grains and 125 grains of 3fg got it from under the front sight.Sold it to Wes Kindig and don;t know where
it is now.The 54,I agree will do about anything that needs done with around ball.In the early 1960's I gave Turner Kirkland $65 for a Colt Special Model military rifle with a 40" barrel that had been reamed smooth.I took it to Bill Large's shop and made a 54 caliber barrel and
grafted the original breech to it and it would reach out and touch the ground hogs in the Smoke Hole area here in WVa.The last trip to that area was in 1965 and the Colt+the Len Meadows SMR and the Whitworth were all involved then.
Bob Roller