fwiw...
work up loads and then decide. shooting is better than wondering about it forevermore...
Personally have never had any more trouble with a 6# rifle than shooting heavier rifles. The only negative variable for me is recoil.
I doubt the weight of the piece is the issue. To verify, borrow a known 6# class shooter and see how you do with it.