My first thought for m1garand_man was to ask, "what is the temperature difference between days when the rifle was dead-on vs 5" high".
But I think the fouling vs humidity issue is key. I have noticed on very humid days that my bore fouls up much faster and thicker. This fouling can (I suspect) lead to higher pressure and faster velocity, thus resulting in variations in elevation of the group.