Rich, the barrel isn't going to fly through your skull. In the worst case--say, if you forgot to put the tang screw and pins in--the barrel will probably sorta pivot up and probably damage the stock around the tang.
Now if you did a bad job of breeching the barrel, it might be a different matter. but you didn't do that, did you?
I had a terrible flinch for a while with my first rifle, for much the same reason. My mentor pointed out that I shot original guns, and guns made by other people, without hesitation. It was just the rifle I built that was making me nervous. One afternoon he shot a match with the rifle while I watched, and after that I was OK.
Give it a little time for you to get used to shooting the rifle, and I think you'll be OK.