You should be getting a lighter gun with the 20 ga but recoil will be similar. I would look for a 20 ga semiauto for lighter weight with somewhat lower recoil. My preference is a Beretta 309/391 but there are several others such as Browning, Franchi, Remington, et al that will also do fine.
You will not do as well with a 20 ga as with a 12 ga for trap and sporting clays, expect 3-5 birds per 100 as a general rule depending on course and skill. I don't know a single person who truly believes score doesn't matter- it may not be utmost on their mind but it costs the same to hit the bird as it does to miss and missing sucks. I shoot a 20 ga regularly for sporting clays and my long term average is below that of the 12ga. If shots never get past 25 yards like skeet then there isn't much difference but if they do you will find the 12 ga to pull away.
As for your current gun, more use will make it seem lighter by building up muscles not normally used as would more appropriate loads. Every spring my wife complains about the weight of her Beretta 391 sporting clays gun but after a couple weeks of regular shooting she doesn't mention it. She uses Winchester AA Xtra-lites or the equivalent 1 oz load at 1180 fps for competition with no problems. Those are the same loads I use for sporting clays shoots I think important though I pick up the velocity when shooting FITASC or other event where the shots are long.
