It's because they exude the ability to tear you up good if they ever decided to, and that freaks some folks out
Just for clarity, I assure you that my 125 lb pup could do a whole hell of a lot more physical damage to you than my neighbors 60lb pit could. Most Pit vs human fatalities are usually involving small children, women and people that cant fend them off very well, or when they attack in numbers.