I do quite agree that hickory (or any other common rammer wood) itself doesn't wear any metal at all, same as the paper sheets underneath the glued-on oxides for sanding purposes doesn't wear on anything. Both of them only hold the substances that do the cutting.
Ever run your hand across a wooden railing or handle that's had 75 or 100 years of service, with very few if any refinishes? It's smooth, smooth like butter if it's like the ones I've experienced*. It's way easy to see the effect on an object that gets handled more at one place and less otherwise. Human skin, even calloused, is softer than wood but it was hands that provided that polish on the wooden surface. It just takes time and a little bit of contamination. Perfectly clean hands wouldn't polish wood, same as a perfectly clean rammer wouldn't polish or reshape a muzzle. Problem is we and our rammers are rarely so clean.
*I have to retell this one once again, sorry if it's a an old story already: When surveying one day we had the opportunity to enter an old barn. The interior doors didn't have handles of any sort on them but a single hole about 1.25" (30mm) across, without a thought I poked my finger in that hole and pulled the door open. I was immediately shocked by the absolute fantastic smoothness of the inside diameter of the hole and the wood around it. How many 10's of 1,000's of times had a finger or glove (soft but dirty) been poked into that hole? A million?
Unfortunately I didn't take pics or measurements--that hole could have been worn wonky too!
But I'll never forget how silky slick it was. And knew immediately that no furniture maker had polished it out, that it was made that way completely unintentionally.