Your POI will definitely change with light, regardless of what your sight is made of. There is a reason that most ranges have covered firing lines other than rain protection. If you have never experienced this phenomena, you can only speculate.
I first noticed it shooting in a match on a bright sunny day. There were large fluffy clouds drifting over, of the type that cast a shadow on the ground under them. I was facing south. When there were no clouds, I was hitting the bullseye. As a cloud drifted over my POI would drop 3" and hit right 3". As soon as the cloud drifted past, the POI returned to the bullseye. When it finally dawned on me what was happening, I shot a 2nd target to confirm. The results were the same.