Unless you are looking at a custom rifle built for, and used by, say, Jack O'Connor, the seller is never likely to recover the initial build price of the rifle.
If you look at the prices commanded by some of the top rifle makers today, you are in the tens of thousands in some instances. For that kind of dough, a customer wants something built specifically for him, and is not likely to spend those kind of bucks to buy something built for someone else.
IMO, unless you are buying at dirt-cheap prices, it's not an investment. Consider it as buying art, doing so because you really like it.
