I frequently see hunters at airports wearing their camo, fly fisherman wearing their fly fishing gear on a flight to Alaska etc, BJJ wrestlers wearing their uniform on the way to and from their training. Who cares whether you wear whatever clothing is suitable for the activity you do, outside of that activity, on the way to or from home? If it is convenient to do so, why not? I think we are moving away from the idea that you don't wear a specific style of clothing for a specific activity just because some people might think it's gay. If you are not gay, you shouldn't need to worry that some people might think that you are gay just because of the sport specific clothing that you wear.