Take a quarter inch steel rod of sufficient length, cut a slot in the end of it, about an inch deep, and drill a cross hole in it. You can then tear off a piece of sandpaper, fold it over and slip it into the slot. Take a small nail and shove it through the hole to hold the sandpaper in, and you can run it up and down inside the hole with a cordless drill to sand the hole out some.
That said, if you're trying to stick a 5/16" ramrod into a 5/16" hole, it WON'T FIT. Two objects can't occupy the same space. To get a 5/16" ramrod to fit into a 5/16" hole, it must be taken down so it is smaller. Same with a 3/8" hole. To get a 3/8" rod to fit into a 3/8" hole, the rod must be made smaller.