The only difference in accuracy you'll notice from switching lowers is one that affects your accuracy with trigger pull issues.
If you have a chit trigger on one, you may not be able to as accurate as the lower that has a great trigger.
I had one upper/lower pairing that was loose. Real loose. But it didn't affect accuracy either. That rattle trap shot well.