Scientists tested a number of methods in an effort to develop an efficient and inexpensive procedure for the mass sterilisation 

View This Term in the Glossary

 and other groups Nazi leaders considered to be racially or genetically undesirable.

Importantly, when you think of data, you might think of formal studies in which demographics and representation are carefully considered, limitations are weighed, and then the results are peer-reviewed. That’s not necessarily the case with the AI-based systems that might be used to make a decision about you. Let’s take one source of data everyone has access to: the internet. One study found that, by teaching an artificial intelligence to crawl through the internet — and just reading what humans have already written — the system would produce prejudices against black people and women. Another example of how training data can produce sexism in an algorithm occurred a few years ago, when Amazon tried to use AI to build a résumé-screening tool. According to Reuters, the company’s hope was that technology could make the process of sorting through job applications more efficient. It built a screening algorithm using résumés the company had collected for a decade, but those résumés tended to come from men. That meant the system, in the end, learned to discriminate against women. It also ended up factoring in proxies for gender, like whether an applicant went to a women’s college. (Amazon says the tool was never used and that it was nonfunctional for several reasons.)

*Food for thought: Given that it is known that there is bias in algorithms, why do we (instituitions, companies, governments etc.) still continue to use (and rely on) them?

 Just because a technology is accurate doesn’t make it fair or ethical. For instance, the Chinese government has used artificial intelligence to track and racially profile its largely Muslim Uighur minority, about 1 million of whom are believed to be living in internment camps.

Relevant controversial issues:


Also consider Intersectionality  (intersecting and overlapping social identities- which could prove to be empowering for some, yet oppressing for others)