Assuming it's a decently heavy breech's barrel, I'd have no qualms of shooting up to about 130/140gr. 2F. I would not use more than about 115/120gr.2F in a 15/16th inch flintlock, .54 by Rice.
More powder SHOULD improve accurcy along with flattening trajectory. If more powder does not improve accuracy, the patch probably needs to be thicker, ie: a tighter load. I would use none other than a .535" ball with such a load.
Your 80gr. charge would be hard pressued to exceed 1,400fps. The heavier charges I mentioned should develope a good 1,850fps which will give you a 130 yard p[oint blank range - where the ball is no more than 3" above nor 3" below the line of the sights.
To shoot accurately at longer range than point blank, one needs to feed the rifle. Even at 50 & 100yards, the heavier loads should shoot more accurately. Anything over 80gr. is an improvement for longer range shooting.
What does Rice have to say about loads for your barrel. I'd ask them first.