Likely one (or both) of two things:
1. Crown is too sharp (cutting patches on the way in).
2. Wire edges on the lands.
I would get a green Scotchbright pad, wind it around your ramrod, & work it up and down the bore until its starts to loosen, replace & repeat until you get up to around 100 strokes or so. The Scotchbrite will not hurt the barrel, but will polish it up nicely. Don Getz advocated this method. Shoot it and see what happens.
If that doesn't do it, take some 400 grit wet or dry and press it down on the muzzle with your thumb, while turning the barrel. Turn the barrel equal numbers of turns clockwise and counter clockwise. When you replace the sandpaper, repeat the process starting from another flat. Do this until you smooth the bevel of the crown nicely & go shoot it. One thing I WOULD NOT do is go after the crown with a carburundum "rock" in a hand drill. You would be amazed how much crown you can take off in a second with that technique. You can easily wind up with a lopsided crown that way, but you will just polish it with your thumb.
Mr. Rayle makes fine barrels & yours will shoot with a little love & attention. Back in the day, we didn't expect a cut rifled barrel to shoot well for around 100 shots or so because the lands were too sharp. This was referred to as "shooting in" a barrel which also frequently required some crown polishing as well. If all else fails, give Ed a call; he will stand behind his work.
I routinely shoot .395 balls with eighteen to twenty thousandths thick patching in my .40s with any charge of powder without problems. As a matter of fact, I usually shoot a 5 thousandths under bore size ball with the same patch thickness for any caliber.
Roger B.