The larger the calibre, the greater the POTENTIAL for shooting smaller groups.
The larger the calibre, the greater the recoil which can reduce the shooter's ability to benefit from the better 'terminal' ballistics of the larger calibre, which would then reduce the potential.
Last time out shooting, for example, I found I can no longer 'handle' the recoil of my .69 calibre rifle, thus my 100 yard accuracy suffered greatly. Loss of eye-sight for open sights also plays a part in this equation.
My .50 cal. rifle seems to be performing now and am looking forward to shooting some 100 and 200 yard groups with it this coming spring and summer.
I would not attempt to shoot a rifle less than .50 cal, seriously, at 100yards. This is merely that I have little faith in those lighter balls- it's a mind-set, I guess.
Both my .40 and .45 cal. rifles gave me 1/2", 5 shot groups at 50yards, so the potential is there to do well at 100 - I just never tried them at that range and now they are both sold, I cannot try them. Couple guys needed accurate rifles, so I sold them. Now, I have to compete against those rifles - story of my life- so far, so good.