Banana depends on the individual barrel, and how much stress is in the steel, or is created by your milling cutter.
I have milled barrels on one side, and when looking through the hole, I could no longer see thru it, there was so much bend. But when I milled the opposite side, it was perfectly straight again.
I'm not saying this will happen with YOUR barrel, but it's nice insurance to be able to throw it out and drill and ream a new blank on the spot. If you don't have that luxury, then be aware of the risks involved!
Some barrels are almost stress free, or have been normalized to get rid of the stress. You would probably have better luck with one of those.
But how do you know what kind of barrel you have? You don't, until you cut into it.