My experience is that unless you are shining a white light directly into the eyes of deer they do not seem to bother at all about the light. Keep it on them where you get the reflection of their eyes and especially with an LED with some power and they leave. I think it hurts their eyes just like it does with ours. Keep it out of their eyes and they do not seem to associate it with people or danger.

LEDs with some power can for sure ruin your night vision for a period though. I only use mine full power when blood trailing deer.