Two main factors IMO, LBS's "Great Society" was a political scam (LBJ stated the the Negroes will be voting Democrat for the next 200 years.... he used a more derogatory term though) along with aptitude described in The Bell Curve.... No one is born as a blank slate..... some have a better aptitude for athletics and others pick up advanced math as easy falling off a log....
