Hi Ray,
To be honest with you, I fuss over this decision on every one of the 35+ guns gun I've built over the years. Right now I install all the decorative inlays, ramrod thimbles, patchbox, sideplate, and buttplate before I do the final coats of finish. I put on the trigger, trigger guard, and lock after I am done. You have to clean excess finish out of the mortices because it builds up and also swells the wood so be careful. A funny thing, Ray, I always hate my work when I get to the stage of putting on the last finish. I inevitably think they look like $#@*. When the finish is done and polished or smoothed, I like them again. Go figure.
dave