That the muzzle was drilled after separation or a different piece of steel is likely why it is smooth. The bore of the false muzzle being .003" smaller than the rifle's bore (do you mean groove?) is
puzzling. If the FM's bore is smaller than the rifle's bore, the false muzzle is doing nothing. Might as well shoot a .003" undersized ball or bullet with card wads.
If you mean the bore of the FM is .003" smaller than the groove diameter, then a slightly crown would be all that is necessary on the muzzle.
Flat bottomed cutters for cutting grooves was used in the late 1800's for bullet firing rifles. The grooves would be deeper in the corners, than the middle of the groove- like Henry or Pope rifling.
Henry had sharp corners in his rifling, however Pope's was rounded, but still deeper than in the middle, due to the flat bottomed rifling.
If the patch is tight enough on the rod, even a hickory rod can be used to 'find' the rate of twist. The rod will WANT to twist with the rifling, just don't try to stop it form turning.
Your Dewey rod should work perfectly. Taylor and I used my .35 cal. and up Dewey rod to cast a plug and lap a barrel. We did this twice & salvaged a barrel that was badly pitted from using boiling water for cleaning by the owner.
If the grooves are shallower at the land corner, then the rifling cutter was rounded. Deeper in the middle would be impossible, unless there was also a radius like Pope barrels and you are measuring on that radius.