Rick
I truly believe that if it fits your head currently and you like it, you could go with out. In fact, I do a small number of unlined hats a year for people that don't want to spend as much. Two of which will be headed to Dixons Gun Fair next week (two Cocked hats). I will often leave a liner out of a hat if I pull it of the dry board and find a flaw in the felt (a thin spot or color problem etc). Once I get a hat to that point there's no sense in not taking it the rest of the way. I just sell it for less (as what i like to call a "Beater Hat") and mark on the tag what the flaw is.
Remember that if you sew in a liner of cotton or linen that you will loose approximately a quarter inch in size and will have to refit and stretch the hat.
Cheers
Morgan Shea