I am not familiar with the brand chamfer tool you have, however I have been chamfering holes for 35 plus years and can offer some tips:
#1 -Slow Down, often the slower you turn the tool, the better the cut.
#2- look at the bit and see if it looks sharp and clean any chips in the teeth.
#3 - use cutting oil, most Any oil will help, thicker is better than thinner.
#4 use fairly heavy feed down, or do not just let the tool rub, chamfering tools require firm down pressure.
When I say slow, often Less than 150 RPM, if you are wanting a specific hole to turn out nice (like on a rifle) it's not a bad idea to drill a couple practice pilot holes same size as in rifle, and in simular type metal to test your speed and feed rates, often going VERY slow RPM is the best fix.
Finally, some chamfering tools are poorly designed, it's not a tool you want to go cheap on....hope this helps.