Couple things, Bill. Use a constant, consistent feed, no starting/stopping. While watching, I noticed that each time the drill stopped, it was leaving the flatter side away from the blade.
Was also thinking that you might be better spinning the rod against something like 16-grit paper cupped in one hand; that would certainly avoid the chunking of that single blade on the dowel-making tool.
You alluded to a pencil sharpener, but they have two opposing cutter blades.
Kind of relegated to sitting and watching any more.