Regardless of the measurements, you're going to have to go to a thinner ball/patch combo than the .530/.020 you've been using. You can't make it easier to load by going to bigger ball, thicker patch, so you might as well go smaller to find a snug but loadable combo, and see what accuracy you get. It can't be any fun fighting that .530/.020 load downbore, especially if it's not delivering one hole groups. You think you've eliminated some of the problem factors, so it's perhaps time to go back and more or less start over. Conventional wisdom says you'll (probably) get best accuracy with the tightest ball/patch combo you can load without deforming the ball (that's the theory, anyway).
It wouldn't hurt to lay hold of a small supply of .520 and .526 balls (Track of Wolf sells cast balls in packages of 50). I'd go with a couple of different patching material: JoAnn Fabrics will sell you a quarter yard of linen or pocket drill pretty cheap. Since you're after easier loading, finding material that will compress down to around .012-.015 sounds like it would be worth trying. You might think about cutting the powder charge back to 50 or 60 grains until you get the gun grouping better. Start with the smallest ball, thinnest patch combo you have available and then start increasing patch thickness or ball diameter and see where you get the best grouping at acceptable level of loading effort.
Good luck. It sounds like you're getting there. SCL