Sounds like you plugged the hole with the end grain up and out. Suggest you cut a small square hole, tapered top to bottom, fill with an appropriately sized (a little oversized) piece of the same wood with the grain running in the same direction. Stain and glue with ca glue, file off the excess, sand, more color and finish. Do it right and the repair should be nearly invisable. Good luck. Wormey