I was concerned about recoil when I was buying my kit from Jim Chambers. Because I wanted to hunt deer, he recommended either a 50 or 54 caliber and he said the 54 caliber barrel is lighter (presumably because more metal is removed from the bore). That convinced me to go with 54 because I'd be carrying the rifle around in the woods.
You can see my post about my shooting below... I built an Isaac Haines kit and when I shoot off bags, I just get the butt properly seated in the joint of my shoulder and I don't even put my front hand on the stock. It kicks and you can feel it but without a scope, you have no worry of hitting yourself in the eye. When I did shoot it once, I had my face pretty buried in the cheek rest and it kind of knocked me a bit in the face but wasn't so bad.
I shoot a lot of sporting clays with my 20 gauge and I honestly didn't find it much different.