Rich's drawing is a view looking down on the top of the barrel channel, and though the grain direction can be tricky there, in my experience, the grain usually flows along the length of the wood without too much run-out. Where it gets tricky and important, is along the side grain where the grain normally or often at least, runs unhill from the butt to the forend tip. In this case you must cut in the same direction, that is, from the breech end toward the muzzle. If you try to cut in the other direction the chisel will follow the grain and not shear the wood off, but rather, split and chip out the wood sometimes catastrophically. I use a large gouge and a large wooden mallet to take out most of the wood, then flat and skew chisels for the flats.