OldMtnMan,
The OP says he does do a 'tween shot wipe.
I've noticed the same thing. That first shot from a pristine bore is off the mark. Just running a spit-moistened patch down the bore once-and-out doesn't really clean it that thoroughly. Thus the subsequent shots group in a different spot.
When I get home, before putting my gun up for storage, I will run up to 6 patches or more down the bore to really get it clean enough that I don't think it will rust in storage. Shooting once at an elk, then taking the piece home to thoroughly clean seems tolerable. But trying to run half a dozen patches down the barrel between squirrels seems a bit discouraging. To paraphrase Waylon Jennings, "Are you sure [Dan'l Boone] done it this way?"