That’s part of it, but the biggie is “edge effect.” Most stray light comes from around the edges of lenses, because that’s where it’s hardest to bend, and also where runs into lens mounts. Smaller lenses have more edge per amount of surface, so tend to have more edge effect.
