If the hammer has been case hardened, it will have to be annealed down...best as you can. I doubt it has been, though. If it is soft, serrations could be filed onto the hammer spur with reasonable ease with good, sharp cornered three-square files. Might not be perfectly even, but functional. A checkering file is an expensive tool, and good for laying out the fine serrations on a modern pistol sight or other such things. It won't cut fully on its own, but it starts your cuts and gets them evenly spaced....which may not work all that well on such a curved surface anyway. I would not recommend getting the checkering file solely for this operation (I'm not sure it would be very advantageous to have at all anyway). It just takes a keen eye, and a steady hand.
Serrations would probably be sufficient, but actual checkering could be done by filing again lengthwise, over the crossways serrations. Not so easy, but can be done.