# data[].attributes.body
In this interview, Browne expands on her research, including insights on police technology and racial profiling. It gives helpful context to her academic work by showing her thought process.

# data[].attributes.attachment_link.description
“Constantly Dreaming of Concrete” features an installation, glass sculptures, graffiti, and traditional painted watercolors and canvases by the artist known as EGS, contrasting the quasi-illegality of his street art activity…

# data[].attributes.headline
Week 4 – MIT Technology Review (Joy Buolamwini)

# data[].attributes.subject
Week 4 – MIT Technology Review (Joy Buolamwini)

# data[].attributes.body
This article outlines how Black Americans have been subject to intense surveillance historically. It complements Simone Browne’s “Dark Matters” by connecting her theory of biometric surveillance to practical cases in the U.S.

# data[].attributes.attachment_link.title
Characterizing Profiles: Data Surveillance and Literature in the Twenty-First Century

# data[].attributes.attachment_link.description
Author(s): Johnston, Katherine Danielle | Advisor(s): Kinney, Katherine A | Abstract: Data profiles have come to proliferate across the private and public sectors as digitization has engendered new technologies and techniques for gathering, aggregating, and sharing data. My dissertation interrogates the epistemologies, affects, and power-relations of this increasingly ubiquitous data surveillance.

# data[].attributes.body
This article profiles Joy Buolamwini’s work exposing racial bias in facial recognition. It supports the course’s critique of race-neutral tech.

# data[].attributes.attachment_link.provider_name
MIT News | Massachusetts Institute of Technology

# data[].attributes.attachment_link.readable_text
Three commercially released facial-analysis programs from major technology companies demonstrate both skin-type and gender biases, according to a new paper researchers from MIT and Stanford University will present later this month at the Conference on Fairness, Accountability, and Transparency.

In the researchers' experiments, the three programs' error rates in determining the gender of light-skinned men were never worse than 0.8 percent. For darker-skinned women, however, the error rates ballooned - to more than 20 percent in one case and more than 34 percent in the other two.

The findings raise questions about how today's neural networks, which learn to perform computational tasks by looking for patterns in huge data sets, are trained and evaluated. For instance, according to the paper, researchers at a major U.S. technology company claimed an accuracy rate of more than 97 percent for a face-recognition system they'd designed. But the data set used to assess its performance was more than 77 percent male and more than 83 percent white.

"What's really important here is the method and how that method applies to other applications," says Joy Buolamwini, a researcher in the MIT Media Lab's Civic Media group and first author on the new paper. "The same data-centric techniques that can be used to try to determine somebody's gender are also used to identify a person when you're looking for a criminal suspect or to unlock your phone. And it's not just about computer vision. I'm really hopeful that this will spur more work into looking at [other] disparities."

Buolamwini is joined on the paper by Timnit Gebru, who was a graduate student at Stanford when the work was done and is now a postdoc at Microsoft Research.
Chance discoveries
The three programs that Buolamwini and Gebru investigated were general-purpose facial-analysis systems, which could be used to match faces in different photos as well as to assess characteristics such as gender, age, and mood. All three systems treated gender classification as a binary decision - male or female - which made their performance on that task particularly easy to assess statistically. But the same types of bias probably afflict the programs' performance on other tasks, too.

Indeed, it was the chance discovery of apparent bias in face-tracking by one of the programs that prompted Buolamwini's investigation in the first place.

Several years ago, as a graduate student at the Media Lab, Buolamwini was working on a system she called Upbeat Walls, an interactive, multimedia art installation that allowed users to control colorful patterns projected on a reflective surface by moving their heads. To track the user's movements, the system used a commercial facial-analysis program.

The team that Buolamwini assembled to work on the project was ethnically diverse, but the researchers found that, when it came time to present the device in public, they had to rely on one of the lighter-skinned team members to demonstrate it. The system just didn't seem to work reliably with darker-skinned users.

Curious, Buolamwini, who is black, began submitting photos of herself to commercial facial-recognition programs. In several cases, the programs failed to recognize the photos as featuring a human face at all. When they did, they consistently misclassified Buolamwini's gender.
Quantitative standards
To begin investigating the programs' biases systematically, Buolamwini first assembled a set of images in which women and people with dark skin are much better-represented than they are in the data sets typically used to evaluate face-analysis systems. The final set contained more than 1,200 images.

Next, she worked with a dermatologic surgeon to code the images according to the Fitzpatrick scale of skin tones, a six-point scale, from light to dark, originally developed by dermatologists as a means of assessing risk of sunburn.

Then she applied three commercial facial-analysis systems from major technology companies to her newly constructed data set. Across all three, the error rates for gender classification were consistently higher for females than they were for males, and for darker-skinned subjects than for lighter-skinned subjects.

For darker-skinned women - those assigned scores of IV, V, or VI on the Fitzpatrick scale - the error rates were 20.8 percent, 34.5 percent, and 34.7. But with two of the systems, the error rates for the darkest-skinned women in the data set - those assigned a score of VI - were worse still: 46.5 percent and 46.8 percent. Essentially, for those women, the system might as well have been guessing gender at random.

"To fail on one in three, in a commercial system, on something that's been reduced to a binary classification task, you have to ask, would that have been permitted if those failure rates were in a different subgroup?" Buolamwini says. "The other big lesson ... is that our benchmarks, the standards by which we measure success, themselves can give us a false sense of progress."

"This is an area where the data sets have a large influence on what happens to the model," says Ruchir Puri, chief architect of IBM's Watson artificial-intelligence system. "We have a new model now that we brought out that is much more balanced in terms of accuracy across the benchmark that Joy was looking at. It has a half a million images with balanced types, and we have a different underlying neural network that is much more robust."

"It takes time for us to do these things," he adds. "We've been working on this roughly eight to nine months. The model isn't specifically a response to her paper, but we took it upon ourselves to address the questions she had raised directly, including her benchmark. She was bringing up some very important points, and we should look at how our new work stands up to them."

# data[].attributes.attachment_link.title
Study finds gender and skin-type bias in commercial artificial-intelligence systems

# data[].attributes.attachment_link.description
A new paper from the MIT Media Lab's Joy Buolamwini shows that three commercial facial-analysis programs demonstrate gender and skin-type biases, and suggests a new, more accurate method for evaluating the performance of such machine-learning systems.

# data[].attributes.body
This real-world case illustrates how racially biased algorithms operate in healthcare. It makes clear the life-or-death stakes of algorithmic racism.

# data[].attributes.attachment_link.readable_text
Artificial intelligence (AI) and algorithmic decision-making systems - algorithms that analyze massive amounts of data and make predictions about the future - are increasingly affecting Americans' daily lives. People are compelled to include buzzwords in their resumes to get past AI-driven hiring software. Algorithms are deciding who will get housing or financial loan opportunities. And biased testing software is forcing students of color and students with disabilities to grapple with increased anxiety that they may be locked out of their exams or flagged for cheating. But there's another frontier of AI and algorithms that should worry us greatly: the use of these systems in medical care and treatment.

The use of AI and algorithmic decision-making systems in medicine are increasing even though current regulation may be insufficient to detect harmful racial biases in these tools. Details about the tools' development are largely unknown to clinicians and the public - a lack of transparency that threatens to automate and worsen racism in the health care system. Last week, the FDA issued guidance significantly broadening the scope of the tools it plans to regulate. This broadening guidance emphasizes that more must be done to combat bias and promote equity amid the growing number and increasing use of AI and algorithmic tools.

Bias in Medical and Public Health Tools

In 2019, a bombshell study found that a clinical algorithm many hospitals were using to decide which patients need care was showing racial bias - Black patients had to be deemed much sicker than white patients to be recommended for the same care. This happened because the algorithm had been trained on past data on health care spending, which reflects a history in which Black patients had less to spend on their health care compared to white patients, due to longstanding wealth and income disparities. While this algorithm's bias was eventually detected and corrected, the incident raises the question of how many more clinical and medical tools may be similarly discriminatory.

Another algorithm, created to determine how many hours of aid Arkansas residents with disabilities would receive each week, was criticized after making extreme cuts to in-home care. Some residents attributed extreme disruptions to their lives and even hospitalization to the sudden cuts. A resulting lawsuit found that several errors in the algorithm - errors in how it characterized the medical needs of people with certain disabilities - were directly to blame for inappropriate cuts made. Despite this outcry, the group that developed the flawed algorithm still creates tools used in health care settings in nearly half of U.S. states as well as internationally.

One recent study found that an AI tool trained on medical images, like x-rays and CT scans, had unexpectedly learned to discern patients' self-reported race. It learned to do this even when it was trained only with the goal of helping clinicians diagnose patient images. This technology's ability to tell patients' race - even when their doctor cannot - could be abused in the future, or unintentionally direct worse care to communities of color without detection or intervention.

Tools Used in Health Care Can Escape Regulation

Some algorithms used in the clinical space are severely under-regulated in the U.S. The U.S Department of Health and Human Services (HHS) and its subagency the Food and Drug Administration (FDA) are tasked with regulating medical devices - with devices ranging from a tongue depressor to a pacemaker and now, medical AI systems. While some of these medical devices (including AI) and tools that aid physicians in treatment and diagnosis are regulated, other algorithmic decision-making tools used in clinical, administrative, and public health settings - such as those that predict risk of mortality, likelihood of readmission, and in-home care needs - are not required to be reviewed or regulated by the FDA or any regulatory body.

This lack of oversight can lead to biased algorithms being used widely by hospitals and state public health systems, contributing to increased discrimination against Black and Brown patients, people with disabilities, and other marginalized communities. In some cases, this failure to regulate can lead to wasted money and lives lost. One such AI tool, developed to detect sepsis early, is used by more than 170 hospitals and health systems. But a recent study revealed the tool failed to predict this life-threatening illness in 67 percent of patients who developed it, and generated false sepsis alerts on thousands of patients who did not. Acknowledging this failure was the result of under-regulation, the FDA's new guidelines point to these tools as examples of products it will now regulate as medical devices.

The FDA's approach to regulating drugs, which involves publicly shared data that is scrutinized by review panels for adverse effects and events contrasts to its approach to regulating medical AI and algorithmic tools. Regulating medical AI presents a novel issue and will require considerations that differ from those applicable to the hardware devices the FDA is used to regulating. These devices include pulse oximeters, thermal thermometers, and scalp electrodes -each of which have been found to reflect racial or ethnic bias in how well they function in subgroups. News of these biases only underscores how vital it is to properly regulate these tools and ensure they don't perpetuate bias against vulnerable racial and ethnic groups.

A Lack of Transparency/Biased Data

While the FDA suggests that device manufacturers test their devices for racial and ethnic biases before marketing to the general public, this step is not required. Perhaps more important than assessments after a device is developed is transparency during its development. A STAT+ News study found many AI tools approved or cleared by the FDA do not include information about the diversity of the data on which the AI was trained, and that the number of these tools being cleared is increasing rapidly. Another study found AI tools "consistently and selectively under-diagnosed under-served patient populations," finding the under-diagnosis rate was higher for marginalized communities who disproportionately don't have access to medical care. This is unacceptable when these tools may make decisions that have life or death consequences.

Equitable treatment by the health care system is a civil rights issue. The COVID-19 pandemic has laid bare the many ways in which existing societal inequities produce health care inequities - a complex reality that humans can attempt to comprehend, but that is difficult to accurately reflect in an algorithm. The promise of AI in medicine was that it could help remove bias from a deeply biased institution and improve health care outcomes; instead, it threatens to automate this bias.

Policy changes and collaboration among key stakeholders, including state and federal regulators, medical, public health, and clinical advocacy groups and organizations, are needed to address these gaps and inefficiencies. To start, as detailed in a new ACLU white paper:

Public reporting of demographic information should be required.

The FDA should require an impact assessment of any differences in device performance by racial or ethnic subgroup as part of the clearance or approval process.

Device labels should reflect the results of this impact assessment.

The FTC should collaborate with HHS and other federal bodies to establish best practices that device manufacturers not under FDA regulation should follow to lessen the risk of racial or ethnic bias in their tools.

Rather than learn of racial and ethnic bias embedded in clinical and medical algorithms and devices from bombshell publications revealing what amounts to medical and clinical malpractice, the HHS and FDA and other stakeholders must work to ensure that medical racism becomes a relic of the past rather than a certainty of the future.

# data[].attributes.attachment_link.title
Algorithms Are Making Decisions About Health Care, Which May Only Worsen Medical Racism | ACLU

# data[].attributes.attachment_link.description
Unclear regulation and a lack of transparency increase the risk that AI and algorithmic tools that exacerbate racial biases will be used in medical settings.

# data[].attributes.body
This piece gives a strong overview of Afrofuturism and its cultural significance. It helps us connect course readings to art, literature, and science fiction.

# data[].attributes.attachment_link.description
Verifying you are human. This may take a few seconds.

# data[].attributes.body
This explainer discusses Black Panther and other media that connect technology with Black liberation. It helps visualize what a decolonial future might look like.

# data[].attributes.attachment_link.description
Vox is a general interest news site for the 21st century. Its mission: to help everyone understand our complicated world, so that we can all help shape it. In text, video and audio, our reporters explain politics, policy, world affairs, technology, culture, science, the climate crisis, money, health and everything else that matters. Our goal is to ensure that everyone, regardless of income or status, can access accurate information that empowers them.

# data[].attributes.body
This review of “Parable of the Sower” adds analysis to Butler’s depiction of race, ecology, and collapse, which we discussed in class.

# data[].attributes.attachment_link.provider_name
Journal of Intersectional Social Justice

# data[].attributes.attachment_link.title
Analysing the Disproportionate Impact of Climate Disaster on Social Minorities as Represented in Climate Fiction: Octavia E. Butler’s Parable of the Sower and Paolo Bacigalupi’s The Water Knife

# data[].attributes.attachment_link.description
Essay completed in the course of a degree

# data[].attributes.body
Reinforces themes of surveillance and medical racism. It ties into our classroom discussions about whose bodies are made visible or invisible in tech.

# data[].attributes.attachment_link.description
Reporting from the frontiers of health and medicine

# data[].attributes.body
This article about Black women being under-treated for pain provides a personal and systemic view of healthcare neglect. It supports critical questions about data, trust, and health.

# data[].attributes.attachment_link.title
Five Facts About Black Women’s Experiences in Health Care | KFF

# data[].attributes.attachment_link.description
This brief examines Black women’s experiences in health care, including unfair treatment by providers due to race and their health outcomes as a result of this treatment. The brief also explores the association between racially concordant providers and positive health care experiences among Black women.

# data[].attributes.body
A great overview of how Indigenous and other communities are reclaiming and reshaping technology. It pairs well with course discussions on decolonial design.

# data[].attributes.body
it explains the “Land Back” movement in simple, clear language. It adds a political and historical context to the notion of reclaiming land and rights.

# data[].attributes.body
Indigenous cartography reframes how we understand space and ownership. This entry supports themes of reimagining knowledge systems.

# data[].attributes.attachment_link.title
Indigenous Cartography & Decolonizing Mapmaking

# data[].attributes.attachment_link.description
Note: It’s June 2020, and while the Coronavirus Pandemic is disproportionately affecting people of color around the world, here in the…

# data[].attributes.headline
Week 9: Final Reflections / Cross-topic Synthesis

# data[].attributes.subject
Week 9: Final Reflections / Cross-topic Synthesis

# data[].attributes.body
it encourages interdisciplinary thinking. The article argues for ethical, critical approaches to tech—something emphasized throughout our course.

# data[].attributes.headline
Week 9: Final Reflections / Cross-topic Synthesis

# data[].attributes.subject
Week 9: Final Reflections / Cross-topic Synthesis

# data[].attributes.body
It shows how digital platforms can be used for political resistance and community building. Ties in with themes of activism and tech.

# data[].attributes.attachment_link.description
from JSTOR, nonprofit library for the intellectually curious

# data[].attributes.headline
Week 9: Final Reflections / Cross-topic Synthesis

# data[].attributes.subject
Week 9: Final Reflections / Cross-topic Synthesis

# data[].attributes.body
so it offers the policy recommendations and an academic summary of racial bias in AI. It gives concrete tools to imagine better tech futures.

# data[].attributes.attachment_link.description
The Brookings Institution is a nonprofit organization based in Washington, D.C. Our mission is to conduct in-depth, nonpartisan research to improve policy and governance at local, national, and global levels.