if someone asks you to remove your hat at
their residence, common courtesy would dictate
that you do so.

a related aside - don't wear spurs in the store.
contractors don't wear their tool belts, and
doctors don't wear surgical masks and stethoscopes
etc.
anyway, if you have a horse that you need to use
spurs with it wasn't trained properly or needs to
be taken to the packing house.