I had this problem with my first build. I drilled straight holes for the pins. The friend I sold the rifle to filed the holes into slots and the problem went away.
I have posed this question here before without this context. The questions being... have you examined original guns, fastened with pins, with slots instead of straight holes in the lugs. The responses seem to indicate straight holes were traditional, with slots being common (but not universal) today. This may be something they just didn't worry about in the past.