If you don't use a vise, the drill can grab the work causing it to spin. I saw a guy try to grab his spinning work (to save it from flying across the room?). I don't know what he was thinking. Bashed hand up pretty bad.
When the work jumps, it usually causes the hole to get messed up.
If you use a free-floating vise, it's easy to align the work under the drill. Just put a pin or bolt on the table somewhere to keep the vise from spinning should it ever spin on you. Don't go grabbing the durn thing, though.
The tilt fixtures, cross slides, etc, are nice, but they add tremendous weight to the table. They also use up height capacity between chuck and table. These units are best for floor model presses, where you have lots of daylight between chuck and table. Often you have to move the work up or down when you go from long bits to short ones. A drill press with a rack and pinion on the column is very helpful, if not vital, when using a cross slide or heavy vise. Always give a table a lift as you crank up or down.