I have a cheap little drill press like what you describe. It is a bottom rung of drill presses. Even so, mine gets a fair amount of use. The accuracy is far better than any hand held drill. Holes go straight, to depth, and stay round. As far as drill presses, go bigger, older, and you get better. If accuracy mattered I would never use a hand held drill. Lot of folks do fine with hand tools though. Folk art quality of build where the holes are a little off but work, you'll be fine.
Ones expectation of precision is the deciding factor. I am quite obsessive and want things to be correct the first try. I have no interest in building using 200 year old backwoods tools and methods. So, when real accuracy is needed I use a milling machine with a drill chuck. It does not even need to be modern or particularly nice. I have one of these:
https://www.astromart.com/images/classifieds/278000-278999/278559-1.jpg For instance, drilling and tapping a touch hole liner. I set it up in the mill. Spot a center hole, drill a pilot hole, drill to size, leave the barrel in the mill and don't move anything, lower the table, put a pilot rod in the chuck to guide the tap, tap the barrel, done. Drilling and tapping fine thread holes can be sketchy with a hand held drill and eyeball method for the tap. I don't want to ruin any barrels.
Somebody mentioned using a drill press with sanding drums and such. Be careful, the chuck is probably pressed onto a tapered spindle. A side load can work it loose and damage the taper.