How hot did you get the metal when drilling the holes? If you got it really hot it might have hardened it.
You may need to anneal it again in order to save your bits.
Just heat up to a good red heat and cool very slowly. I use hardwood ash for the cooling. Warm the ash with another chunk of hot metal, slip the extra chunk of metal out of the ash and slide the but plate in where it came out. Leave it alone to cool completely and the next day it should be dead soft.
If you don't want to anneal it, you may be able to grind the chamfers instead.
