A lot depends on how they're fitted. A good kydex rig that only has a minimum of friction points will wear very little, whereas a leather holster with a lot of boned fitting will wear on the finish rather quickly.
As noted, keeping them both free of debris will go along way to minimizing wear and tear on the finish. That's a very easy thing to do with kydex.
