Do you know this because you've done it and formed that opinion? If the answer is yes then I respectfully say we have a difference in opinion.
Well hell I wouldn't just make it up to have something to argue about.
No really I've had a few ultralights with scopes that were a bit much and to me it screws everything up. I put a 3-9x42 meopro on a Forbes 20B and it made me really dislike the way it felt and handled.
Currently I have a pair of Montanas......the .308 wears a VX-6 2-12x42 and the 7-08 wears a 2.5-8 VX-3. The 308 isn't nearly as nimble or handy to me as the 7-08. As a matter of fact, when you handle them both side by side the .308 starts to feel (to me) like a boat paddle. It even lays a lot better in bags (to me) than the .308. I've yet to figure that out but I can control it much easier and actually shoot better with it.
As far as the scopes go I prefer the VX-6 100 times Over the VX-3 but I tend to leave it at home as I much prefer carrying the rifle with a bit more suitable scope for its weight.
As mentioned earlier, I think I am one of those guys that is really sensitive about particulars in firearms so I tend to notice little things a lot more.
On a weatherby though I'm not sure I would notice much difference whatsoever. Once you start dropping below 6 lbs on a rifle every couple ounces starts changing the feel and it seems to increase exponentially the farther you go.
