couple of rules about using sandpaper that have to followed for a decent finish.
first, keep your sandpaper fresh and sharp,..... change it out often. as soon as you see it not producing the dust that it did when it was brand new, get a new piece. dull sandpaper only burnishes the wood fibers, closing the pores that are expected to take finish. this reduces the vivid grain colors and variations in grain direction that gives your stock it's beauty. this is the reason scraping is preferred over sanding.
second, the finer you go in grit, the more important it is to use progressively finer grits in small increments.