I would say that .005" is a good inletting job. Those gaps will close up, either actually or visually, by the time you finish the gun. As you cut away the wood around the barrel, it will tend to warp toward the barrel closing up those little gaps. Also, the wiskering and AQ application will tend to close up those gaps. The important thing is to may sure the gap is uniform along the barrel. Those will close up and disappear. It is the little digs that won't disappear. Even a tiny ding from a stray chisel or file will stand out like a sore thumb if the fit around it is uniform and tight. I feel that it is best to smooth our the sidewall of the barrel channel with a square rasp even if it opens up the gap a little more. As long as the gap is even, it won't be noticed so much.
If you have an even, noticeable, gap, you can close it up some by soaking he INSIDE ONLY, of the barrel channel with water, replacing the barrel and letting it dry. It will at first push the sidewalls out, but when it dries, it will dry tighter.
It is best to leave all this to the end. Wood moves all over the place. Before you finish the gun, it will bend, twist, swell, and shrink. Over the course of a year it is normal for a stock to swell and shrink 1/16" laterally and 1/8" longitudinally. You are not cutting plastic or metal that only responds to temperature. Wood can be a real pain to inlet because of this. I keep track of the humidity in my shop and will not do any inletting when the humidity is really high, like over 70%; definitely no inletting during a thunderstorm. Around here the humidity can change 30%-40% in a matter of minutes when a summer thunderstorm rolls through. I was inletting a barrel once when one of these rolled through and the barrel was in the stock. I thought that I had accidentally super glued it in. I could not remove that barrel no matter how hard I tried. It just wouldn't budge. I had to wait for the humidity to drop a bit to get the barrel out.