I've done rust bluing for 50+ yrs and have never had a problem with the finish being easy to wear off.
I generally use Slow Rust (Cold Rust) Blue, but still on occasion use Express Blue on some projects.
Express Blue will show a tendency IMO to wear quicker over time than Slow Rust blue.
I think that is just from the nature of how the rust itself is formed on the metal.
In Express Rust it's an almost immediate reaction after the application of the soln to the warm metal surface that produces the 'rust',,and then the part is then put into the boiling water for the conversion to blue/black.
Slow Rust blue the rust is formed naturally over a longer time,,hrs even if you quicken the pace with a damp box. The rust has to have more of a foot or bite into the steel than with Express Bluing.
If the blue is wearing off the metal very quickly after being finished, then there is likely something wrong with the bluing process, soln, water, contamination, ect. Lots of variables.
Assuming you use all the same soln, application methods, cleaning methods, ect,,,,The only difference betw the rust brown that wears excl't,,and the same rust blue that doesn't wear well at all...it the fact that the Bluing went thru a boiling water treatment to convert the Brown rust to Black rust.
I'd look for something in that portion of the process that is contaminating or damaging the blue.
The type of steel you are bluing has a lot to do with it as well.
Some of the more modern steels with alloys of nickle and other elements that do not rust are resisting the very process you are trying to impose on the steel.
Different rusting solns can make a different too. One will work well on a particular steel where another that worked fine in the past will be a bust on a new project if the steel isn't the same.
That shows up most in the Express Rust process.
The solns themselves are not as a general rule interchangeable.
What works as an Express Blue soln will not work as a Cold Rust blue soln,,and vise-versa.
I've worked full time at restoration gunsmithing over that time and have blued and reblued hundreds if not more bbls and parts for different shops and customers. Everything from Purdey, H&H, Parker & Fox and on and on. Rifle and handguns as well. Never had any complaints. Just plenty of work.
If the process didn't wear well, all those beautifully Arsenal rust blued Mausers, Luger and Mannlicher commercial firearms from the late 1800's and early 1900's would certainly have worn to gray/white metal by now. But they haven't.