If you are trying to sight in a rifle, and know that you have accidentally produced a flier, then it probably should not be included at all. So I think you are right in that.
But what about the self-induced flier you didn't catch? If you are sighting in a rifle, you probably want that ignored too. In that case, the median is the better choice.
There is practically no downside to using the median.
There is no one right way. A lot of it boils down to understanding how things work and choosing the way you like the best.
