Terry, I polish all my metal parts down to about 220 during the build, and so when it comes time to finish the wood, sanding with 220 and 320 right over the metal does no harm. I whisker the wood with all the hardware still in place too, so I don't ruin inletting. Once I'm satisfied that the stock is ready for stain, I apply Ferric Nitrtate to maple still with metal in place, so that when I burnish the wood, again no inletting is damaged.
Naturally, the lock and trigger guard are off the rifle during these processes, otherwise getting around them would be difficult. Careful attention needs to be used around lock mortise and trigger guard inlet.