Dennis,
I went back to read your original question again. I get the best, most even twist if I use the round tip pliers, like I said above. And I pick a set of pliars that has the narrowest part of the tip equal to the inside diameter of the helix size I want. I first twist the tines up as tight as they will go around the pliars (thereby using the pliars as a mandrel). That makes the curl even, makes everything nice and round, and gets both sides of the helix to match perfectly. Once it's all tight and in line with the shank, then I untwist it just a bit, so as to get some gap between the tines. I then use a screwdriver to space the gaps out evenly. This the best technique I have been able to discover thus far. (I once tried to space the tines out by wrapping a wire up with them, so as to use the wire as a spacer, but all I managed to do was burn my fingers!). Wrapping them up as tight as they will go with the round tips of the pliers as a mandrel gets them nice and even.
I decided to forgo my cup of coffee this AM and instead when to the shop and twisted up a couple of nails, just so as to make a quick illustration. I didn't twist them all the way down for the simple reason that I didn't want to risk ruining the temper on my vise grips. But you get the idea. If you are actually making a worm, you can heat the tines all the way down to their base and twist them up nicely.
Here's a photo to show the even, round curl. This is looking down into the helix.(sorry its blurry)
Here's a photo of the pliers I'm describing. The left most is a modern set, flat with teeth (teeth can be ground off if making a set for using in worm making). The next is an antique pair, flat (square tip) with no teeth, great for getting. the curl started. The third is a round tip pair (the one I used to twist up these nails). These produced a helix that would work for a .50 cal. You would need a set with finer tips if you needed a smaller worm. (By the way, I try to avoid using pliers with teeth, as the teeth cut into the hot tines and leave burrs.)
Finally, here's a picture of set #3 with one side inside the helix, as if I was twisting the tines. This helps illustrate how one side of the pliers goes inside and serves as a mandrel for the tines to wrap around tightly as you make the twist.
In this last picture, the tines have already been untwisted slightly and spaced out. When first making the curl, they would be touching and tight.
I hope we eventually get to hear how Jim Web does this. I find it facinating to see how different smiths solve these problems in various ways.
Whetrock
PS: for those guys who want to try their hand at this, be sure to polish ALL the forge scale off the outside surface of the helix. The scale is very hard and will scratch up your bore. Some historians suspect that these worms were a major cause of rifling getting worn down in old rifles.