Right Pete. The POI can be different.
That is what I was thinking when I read the OP's post. If you only shoot from the bench, you don't know where it will hit offhand. If the gun wasn't sighted for offhand, the OP's shooting could have been fine, but he could still miss.
That is why Daryl and I have pointed out that when shooting from the bench for a gun to be used elsewhere, hold the gun like it will be used in the field. Hold the foreend with your hand, and support your hand on the rest. Also, sit up straight. The POI will not vary offhand as much.
One other thing to keep in mind.....the OP notes his gun shoots cloverleafs at 20 yards. Even smooth bores will shoot good at 20 yards. Try the gun out at 50 yards, 75 yards, 100 yards. You'll learn a lot, and perhaps find out you have not yet located the optimum load for that gun.
Once sighted, I only shoot offhand. I use clay birds and rubber seal-healing targets at the 50 yard line. I have found swamped barrel longrifles to hang in the hand better offhand than anything modern I've tried. I guess the old guys knew what they were doing.
I hope this helps. God Bless, Marc