I think individual taste buds have a lot to do with this. As I mentioned earlier, I absolutely can't tell the difference between mule deer and elk, nor whitetail and antelope for that matter. Same for moose, caribou and whatever other wild red meat.
I know people that prefer mule deer to anything else. I also know people who won't touch whitetail, but like everything else. How it was taken care of and how it was cooked I am sure has something to do with some of this, especially with a bad taste that was etched in stone in a person's mind when trying a critter for the first time. I guess my only advice to the OP is to try it, and if it is too gamey or nasty either give it away or grind it into sausage. I think New Mexico allows game meat to be made into dog food, so that's possibly another option.
