Unless it is something like heat treated & double tempered 4140 or 4150, that barrel steel will have some level of residual stress.
So you will need to file the same amount off each side to keep it from bending. Like a little from one side, then a little from the opposite, all the way around.
As a metallurgist, I have known this for decades. But, like the shoemaker's kids, I had to watch a miniature rifle barrel I was swamping suddenly "kink" at the swamped end. Golly Ned I wish I hadn't done that. Bent it back straight again, relieved the stresses with a torch & finished filing the barrel. It didn't have to be accurate, 1/3 scale smooth anyway, but had to at least LOOK straight. You have touchier requirements to keep your barrel accurate.