Of the two dozen or so rifles that I have tinkered with over the years, I generally got better accuracy out of a rifle after bedding the action and free floating the barrel. Cannot recall ever having a decrease in accuracy.

Having said that, there is a gentleman in WV who builds light accurate rifles for a living. And my understanding, he full length beds every rifle he builds. Given that his rifles have a proven track record for being accurate, and he has a back log of work, I'd take my experience with a big grain of salt...