# data[].attributes.body
https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing

# data[].attributes.attachment_link.readable_text
On a spring afternoon in 2014, Brisha Borden was running late to pick up her god-sister from school when she spotted an unlocked kid's blue Huffy bicycle and a silver Razor scooter. Borden and a friend grabbed the bike and scooter and tried to ride them down the street in the Fort Lauderdale suburb of Coral Springs.

Just as the 18-year-old girls were realizing they were too big for the tiny conveyances - which belonged to a 6-year-old boy - a woman came running after them saying, "That's my kid's stuff." Borden and her friend immediately dropped the bike and scooter and walked away.

But it was too late - a neighbor who witnessed the heist had already called the police. Borden and her friend were arrested and charged with burglary and petty theft for the items, which were valued at a total of $80.

Subscribe to the Series

Machine Bias: Investigating the algorithms that control our lives.

Compare their crime with a similar one: The previous summer, 41-year-old Vernon Prater was picked up for shoplifting $86.35 worth of tools from a nearby Home Depot store.

Prater was the more seasoned criminal. He had already been convicted of armed robbery and attempted armed robbery, for which he served five years in prison, in addition to another armed robbery charge. Borden had a record, too, but it was for misdemeanors committed when she was a juvenile.

Yet something odd happened when Borden and Prater were booked into jail: A computer program spat out a score predicting the likelihood of each committing a future crime. Borden - who is black - was rated a high risk. Prater - who is white - was rated a low risk.

Two years later, we know the computer algorithm got it exactly backward. Borden has not been charged with any new crimes. Prater is serving an eight-year prison term for subsequently breaking into a warehouse and stealing thousands of dollars' worth of electronics.

Scores like this - known as risk assessments - are increasingly common in courtrooms across the nation. They are used to inform decisions about who can be set free at every stage of the criminal justice system, from assigning bond amounts - as is the case in Fort Lauderdale - to even more fundamental decisions about defendants' freedom. In Arizona, Colorado, Delaware, Kentucky, Louisiana, Oklahoma, Virginia, Washington and Wisconsin, the results of such assessments are given to judges during criminal sentencing.

Rating a defendant's risk of future crime is often done in conjunction with an evaluation of a defendant's rehabilitation needs. The Justice Department's National Institute of Corrections now encourages the use of such combined assessments at every stage of the criminal justice process. And a landmark sentencing reform bill currently pending in Congress would mandate the use of such assessments in federal prisons.

In 2014, then U.S. Attorney General Eric Holder warned that the risk scores might be injecting bias into the courts. He called for the U.S. Sentencing Commission to study their use. "Although these measures were crafted with the best of intentions, I am concerned that they inadvertently undermine our efforts to ensure individualized and equal justice," he said, adding, "they may exacerbate unwarranted and unjust disparities that are already far too common in our criminal justice system and in our society."

The sentencing commission did not, however, launch a study of risk scores. So ProPublica did, as part of a larger examination of the powerful, largely hidden effect of algorithms in American life.

We obtained the risk scores assigned to more than 7,000 people arrested in Broward County, Florida, in 2013 and 2014 and checked to see how many were charged with new crimes over the next two years, the same benchmark used by the creators of the algorithm.

The score proved remarkably unreliable in forecasting violent crime: Only 20 percent of the people predicted to commit violent crimes actually went on to do so.

When a full range of crimes were taken into account - including misdemeanors such as driving with an expired license - the algorithm was somewhat more accurate than a coin flip. Of those deemed likely to re-offend, 61 percent were arrested for any subsequent crimes within two years.

We also turned up significant racial disparities, just as Holder feared. In forecasting who would re-offend, the algorithm made mistakes with black and white defendants at roughly the same rate but in very different ways.

The formula was particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants.

White defendants were mislabeled as low risk more often than black defendants.

Could this disparity be explained by defendants' prior crimes or the type of crimes they were arrested for? No. We ran a statistical test that isolated the effect of race from criminal history and recidivism, as well as from defendants' age and gender. Black defendants were still 77 percent more likely to be pegged as at higher risk of committing a future violent crime and 45 percent more likely to be predicted to commit a future crime of any kind. ( Read our analysis.)

The algorithm used to create the Florida risk scores is a product of a for-profit company, Northpointe. The company disputes our analysis.

In a letter, it criticized ProPublica's methodology and defended the accuracy of its test: "Northpointe does not agree that the results of your analysis, or the claims being made based upon that analysis, are correct or that they accurately reflect the outcomes from the application of the model."

Northpointe's software is among the most widely used assessment tools in the country. The company does not publicly disclose the calculations used to arrive at defendants' risk scores, so it is not possible for either defendants or the public to see what might be driving the disparity. (On Sunday, Northpointe gave ProPublica the basics of its future-crime formula - which includes factors such as education levels, and whether a defendant has a job. It did not share the specific calculations, which it said are proprietary.)

Northpointe's core product is a set of scores derived from 137 questions that are either answered by defendants or pulled from criminal records. Race is not one of the questions. The survey asks defendants such things as: "Was one of your parents ever sent to jail or prison?" "How many of your friends/acquaintances are taking drugs illegally?" and "How often did you get in fights while at school?" The questionnaire also asks people to agree or disagree with statements such as "A hungry person has a right to steal" and "If people make me angry or lose my temper, I can be dangerous."

The appeal of risk scores is obvious: The United States locks up far more people than any other country, a disproportionate number of them black. For more than two centuries, the key decisions in the legal process, from pretrial release to sentencing to parole, have been in the hands of human beings guided by their instincts and personal biases.

If computers could accurately predict which defendants were likely to commit new crimes, the criminal justice system could be fairer and more selective about who is incarcerated and for how long. The trick, of course, is to make sure the computer gets it right. If it's wrong in one direction, a dangerous criminal could go free. If it's wrong in another direction, it could result in someone unfairly receiving a harsher sentence or waiting longer for parole than is appropriate.

The first time Paul Zilly heard of his score - and realized how much was riding on it - was during his sentencing hearing on Feb. 15, 2013, in court in Barron County, Wisconsin. Zilly had been convicted of stealing a push lawnmower and some tools. The prosecutor recommended a year in county jail and follow-up supervision that could help Zilly with "staying on the right path." His lawyer agreed to a plea deal.

But Judge James Babler had seen Zilly's scores. Northpointe's software had rated Zilly as a high risk for future violent crime and a medium risk for general recidivism. "When I look at the risk assessment," Babler said in court, "it is about as bad as it could be."

Then Babler overturned the plea deal that had been agreed on by the prosecution and defense and imposed two years in state prison and three years of supervision.

Criminologists have long tried to predict which criminals are more dangerous before deciding whether they should be released. Race, nationality and skin color were often used in making such predictions until about the 1970s, when it became politically unacceptable, according to a survey of risk assessment tools by Columbia University law professor Bernard Harcourt.

In the 1980s, as a crime wave engulfed the nation, lawmakers made it much harder for judges and parole boards to exercise discretion in making such decisions. States and the federal government began instituting mandatory sentences and, in some cases, abolished parole, making it less important to evaluate individual offenders.

But as states struggle to pay for swelling prison and jail populations, forecasting criminal risk has made a comeback.

Dozens of risk assessments are being used across the nation - some created by for-profit companies such as Northpointe and others by nonprofit organizations. (One tool being used in states including Kentucky and Arizona, called the Public Safety Assessment, was developed by the Laura and John Arnold Foundation, which also is a funder of ProPublica.)

There have been few independent studies of these criminal risk assessments. In 2013, researchers Sarah Desmarais and Jay Singh examined 19 different risk methodologies used in the United States and found that "in most cases, validity had only been examined in one or two studies" and that "frequently, those investigations were completed by the same people who developed the instrument."

Their analysis of the research through 2012 found that the tools "were moderate at best in terms of predictive validity," Desmarais said in an interview. And she could not find any substantial set of studies conducted in the United States that examined whether risk scores were racially biased. "The data do not exist," she said.

Since then, there have been some attempts to explore racial disparities in risk scores. One 2016 study examined the validity of a risk assessment tool, not Northpointe's, used to make probation decisions for about 35,000 federal convicts. The researchers, Jennifer Skeem at University of California, Berkeley, and Christopher T. Lowenkamp from the Administrative Office of the U.S. Courts, found that blacks did get a higher average score but concluded the differences were not attributable to bias.

The increasing use of risk scores is controversial and has garnered media coverage, including articles by the Associated Press, and the Marshall Project and FiveThirtyEight last year.

Most modern risk tools were originally designed to provide judges with insight into the types of treatment that an individual might need - from drug treatment to mental health counseling.

"What it tells the judge is that if I put you on probation, I'm going to need to give you a lot of services or you're probably going to fail," said Edward Latessa, a University of Cincinnati professor who is the author of a risk assessment tool that is used in Ohio and several other states.

But being judged ineligible for alternative treatment - particularly during a sentencing hearing - can translate into incarceration. Defendants rarely have an opportunity to challenge their assessments. The results are usually shared with the defendant's attorney, but the calculations that transformed the underlying data into a score are rarely revealed.

"Risk assessments should be impermissible unless both parties get to see all the data that go into them," said Christopher Slobogin, director of the criminal justice program at Vanderbilt Law School. "It should be an open, full-court adversarial proceeding."

Black Defendants' Risk Scores

White Defendants' Risk Scores

These charts show that scores for white defendants were skewed toward lower-risk categories. Scores for black defendants were not. (Source: ProPublica analysis of data from Broward County, Fla.)

Proponents of risk scores argue they can be used to reduce the rate of incarceration. In 2002, Virginia became one of the first states to begin using a risk assessment tool in the sentencing of nonviolent felony offenders statewide. In 2014, Virginia judges using the tool sent nearly half of those defendants to alternatives to prison, according to a state sentencing commission report. Since 2005, the state's prison population growth has slowed to 5 percent from a rate of 31 percent the previous decade.

In some jurisdictions, such as Napa County, California, the probation department uses risk assessments to suggest to the judge an appropriate probation or treatment plan for individuals being sentenced. Napa County Superior Court Judge Mark Boessenecker said he finds the recommendations helpful. "We have a dearth of good treatment programs, so filling a slot in a program with someone who doesn't need it is foolish," he said.

However, Boessenecker, who trains other judges around the state in evidence-based sentencing, cautions his colleagues that the score doesn't necessarily reveal whether a person is dangerous or if they should go to prison.

"A guy who has molested a small child every day for a year could still come out as a low risk because he probably has a job," Boessenecker said. "Meanwhile, a drunk guy will look high risk because he's homeless. These risk factors don't tell you whether the guy ought to go to prison or not; the risk factors tell you more about what the probation conditions ought to be."

"I'm surprised [my risk score] is so low. I spent five years in state prison in Massachusetts." (Josh Ritchie for ProPublica)

Sometimes, the scores make little sense even to defendants.

James Rivelli, a 54-year old Hollywood, Florida, man, was arrested two years ago for shoplifting seven boxes of Crest Whitestrips from a CVS drugstore. Despite a criminal record that included aggravated assault, multiple thefts and felony drug trafficking, the Northpointe algorithm classified him as being at a low risk of reoffending.

"I am surprised it is so low," Rivelli said when told by a reporter he had been rated a 3 out of a possible 10. "I spent five years in state prison in Massachusetts. But I guess they don't count that here in Broward County." In fact, criminal records from across the nation are supposed to be included in risk assessments.

Less than a year later, he was charged with two felony counts for shoplifting about $1,000 worth of tools from Home Depot. He said his crimes were fueled by drug addiction and that he is now sober.

Northpointe was founded in 1989 by Tim Brennan, then a professor of statistics at the University of Colorado, and Dave Wells, who was running a corrections program in Traverse City, Michigan.

Wells had built a prisoner classification system for his jail. "It was a beautiful piece of work," Brennan said in an interview conducted before ProPublica had completed its analysis. Brennan and Wells shared a love for what Brennan called "quantitative taxonomy" - the measurement of personality traits such as intelligence, extroversion and introversion. The two decided to build a risk assessment score for the corrections industry.

Brennan wanted to improve on a leading risk assessment score, the LSI, or Level of Service Inventory, which had been developed in Canada. "I found a fair amount of weakness in the LSI," Brennan said. He wanted a tool that addressed the major theories about the causes of crime.

Brennan and Wells named their product the Correctional Offender Management Profiling for Alternative Sanctions, or COMPAS. It assesses not just risk but also nearly two dozen so-called "criminogenic needs" that relate to the major theories of criminality, including "criminal personality," "social isolation," "substance abuse" and "residence/stability." Defendants are ranked low, medium or high risk in each category.

As often happens with risk assessment tools, many jurisdictions have adopted Northpointe's software before rigorously testing whether it works. New York State, for instance, started using the tool to assess people on probation in a pilot project in 2001 and rolled it out to the rest of the state's probation departments - except New York City - by 2010. The state didn't publish a comprehensive statistical evaluation of the tool until 2012. The study of more than 16,000 probationers found the tool was 71 percent accurate, but it did not evaluate racial differences.

A spokeswoman for the New York state division of criminal justice services said the study did not examine race because it only sought to test whether the tool had been properly calibrated to fit New York's probation population. She also said judges in nearly all New York counties are given defendants' Northpointe assessments during sentencing.

In 2009, Brennan and two colleagues published a validation study that found that Northpointe's risk of recidivism score had an accuracy rate of 68 percent in a sample of 2,328 people. Their study also found that the score was slightly less predictive for black men than white men - 67 percent versus 69 percent. It did not examine racial disparities beyond that, including whether some groups were more likely to be wrongly labeled higher risk.

Brennan said it is difficult to construct a score that doesn't include items that can be correlated with race - such as poverty, joblessness and social marginalization. "If those are omitted from your risk assessment, accuracy goes down," he said.

In 2011, Brennan and Wells sold Northpointe to Toronto-based conglomerate Constellation Software for an undisclosed sum.

Wisconsin has been among the most eager and expansive users of Northpointe's risk assessment tool in sentencing decisions. In 2012, the Wisconsin Department of Corrections launched the use of the software throughout the state. It is used at each step in the prison system, from sentencing to parole.

In a 2012 presentation, corrections official Jared Hoy described the system as a " giant correctional pinball machine " in which correctional officers could use the scores at every "decision point."

Wisconsin has not yet completed a statistical validation study of the tool and has not said when one might be released. State corrections officials declined repeated requests to comment for this article.

Some Wisconsin counties use other risk assessment tools at arrest to determine if a defendant is too risky for pretrial release. Once a defendant is convicted of a felony anywhere in the state, the Department of Corrections attaches Northpointe's assessment to the confidential presentence report given to judges, according to Hoy's presentation.

In theory, judges are not supposed to give longer sentences to defendants with higher risk scores. Rather, they are supposed to use the tests primarily to determine which defendants are eligible for probation or treatment programs.

Prediction Fails Differently for Black Defendants

Overall, Northpointe's assessment tool correctly predicts recidivism 61 percent of the time. But blacks are almost twice as likely as whites to be labeled a higher risk but not actually re-offend. It makes the opposite mistake among whites: They are much more likely than blacks to be labeled lower risk but go on to commit other crimes. (Source: ProPublica analysis of data from Broward County, Fla.)

But judges have cited scores in their sentencing decisions. In August 2013, Judge Scott Horne in La Crosse County, Wisconsin, declared that defendant Eric Loomis had been "identified, through the COMPAS assessment, as an individual who is at high risk to the community." The judge then imposed a sentence of eight years and six months in prison.

Loomis, who was charged with driving a stolen vehicle and fleeing from police, is challenging the use of the score at sentencing as a violation of his due process rights. The state has defended Horne's use of the score with the argument that judges can consider the score in addition to other factors. It has also stopped including scores in presentencing reports until the state Supreme Court decides the case.

"The risk score alone should not determine the sentence of an offender," Wisconsin Assistant Attorney General Christine Remington said last month during state Supreme Court arguments in the Loomis case. "We don't want courts to say, this person in front of me is a 10 on COMPAS as far as risk, and therefore I'm going to give him the maximum sentence."

That is almost exactly what happened to Zilly, the 48-year-old construction worker sent to prison for stealing a push lawnmower and some tools he intended to sell for parts. Zilly has long struggled with a meth habit. In 2012, he had been working toward recovery with the help of a Christian pastor when he relapsed and committed the thefts.

After Zilly was scored as a high risk for violent recidivism and sent to prison, a public defender appealed the sentence and called the score's creator, Brennan, as a witness.

Brennan testified that he didn't design his software to be used in sentencing. "I wanted to stay away from the courts," Brennan said, explaining that his focus was on reducing crime rather than punishment. "But as time went on I started realizing that so many decisions are made, you know, in the courts. So I gradually softened on whether this could be used in the courts or not."

"Not that I'm innocent, but I just believe people do change." (Stephen Maturen for ProPublica)

Still, Brennan testified, "I don't like the idea myself of COMPAS being the sole evidence that a decision would be based upon."

After Brennan's testimony, Judge Babler reduced Zilly's sentence, from two years in prison to 18 months. "Had I not had the COMPAS, I believe it would likely be that I would have given one year, six months," the judge said at an appeals hearing on Nov. 14, 2013.

Zilly said the score didn't take into account all the changes he was making in his life - his conversion to Christianity, his struggle to quit using drugs and his efforts to be more available for his son. "Not that I'm innocent, but I just believe people do change."

Florida's Broward County, where Brisha Borden stole the Huffy bike and was scored as high risk, does not use risk assessments in sentencing. "We don't think the [risk assessment] factors have any bearing on a sentence," said David Scharf, executive director of community programs for the Broward County Sheriff's Office in Fort Lauderdale.

Broward County has, however, adopted the score in pretrial hearings, in the hope of addressing jail overcrowding. A court-appointed monitor has overseen Broward County's jails since 1994 as a result of the settlement of a lawsuit brought by inmates in the 1970s. Even now, years later, the Broward County jail system is often more than 85 percent full, Scharf said.

In 2008, the sheriff's office decided that instead of building another jail, it would begin using Northpointe's risk scores to help identify which defendants were low risk enough to be released on bail pending trial. Since then, nearly everyone arrested in Broward has been scored soon after being booked. (People charged with murder and other capital crimes are not scored because they are not eligible for pretrial release.)

The scores are provided to the judges who decide which defendants can be released from jail. "My feeling is that if they don't need them to be in jail, let's get them out of there," Scharf said.

Scharf said the county chose Northpointe's software over other tools because it was easy to use and produced "simple yet effective charts and graphs for judicial review." He said the system costs about $22,000 a year.

In 2010, researchers at Florida State University examined the use of Northpointe's system in Broward County over a 12-month period and concluded that its predictive accuracy was "equivalent" in assessing defendants of different races. Like others, they did not examine whether different races were classified differently as low or high risk.

Scharf said the county would review ProPublica's findings. "We'll really look at them up close," he said.

Broward County Judge John Hurley, who oversees most of the pretrial release hearings, said the scores were helpful when he was a new judge, but now that he has experience he prefers to rely on his own judgment. "I haven't relied on COMPAS in a couple years," he said.

Hurley said he relies on factors including a person's prior criminal record, the type of crime committed, ties to the community, and their history of failing to appear at court proceedings.

ProPublica's analysis reveals that higher Northpointe scores are slightly correlated with longer pretrial incarceration in Broward County. But there are many reasons that could be true other than judges being swayed by the scores - people with higher risk scores may also be poorer and have difficulty paying bond, for example.

Most crimes are presented to the judge with a recommended bond amount, but he or she can adjust the amount. Hurley said he often releases first-time or low-level offenders without any bond at all.

However, in the case of Borden and her friend Sade Jones, the teenage girls who stole a kid's bike and scooter, Hurley raised the bond amount for each girl from the recommended $0 to $1,000 each.

Hurley said he has no recollection of the case and cannot recall if the scores influenced his decision.

Sade Jones, who had never been arrested before, was rated a medium risk. (Josh Ritchie for ProPublica)

The girls spent two nights in jail before being released on bond.

"We literally sat there and cried" the whole time they were in jail, Jones recalled. The girls were kept in the same cell. Otherwise, Jones said, "I would have gone crazy." Borden declined repeated requests to comment for this article.

Jones, who had never been arrested before, was rated a medium risk. She completed probation and got the felony burglary charge reduced to misdemeanor trespassing, but she has still struggled to find work.

"I went to McDonald's and a dollar store, and they all said no because of my background," she said. "It's all kind of difficult and unnecessary."

Julia Angwin is a senior reporter at ProPublica. From 2000 to 2013, she was a reporter at The Wall Street Journal, where she led a privacy investigative team that was a finalist for a Pulitzer Prize in Explanatory Reporting in 2011 and won a Gerald Loeb Award in 2010.

Lauren Kirchner is a senior reporting fellow at ProPublica. Surya Mattu is a contributing researcher. Design and production by Rob Weychert and David Sleight.

# data[].attributes.attachment_link.description
There's software used across the country to predict future criminals. And it's biased against blacks.

# data[].attributes.body
https://assets.website-files.com/5e027ca188c99e3515b404b7/5ed1002058516c11edc66a14_FRTsPrimerMay2020.pdf

# data[].attributes.body
https://www.nytimes.com/2019/01/31/opinion/ai-bias-healthcare.html

# data[].attributes.attachment_link.readable_text
In a health system riddled with inequity, we risk making dangerous biases automated and invisible.

By

Dr. Khullar is an assistant professor of health care policy and research.

Credit... Jenice Kim

Artificial intelligence is beginning to meet (and sometimes exceed) assessments by doctors in various clinical situations. A.I. can now diagnose skin cancer like dermatologists, seizures like neurologists, and diabetic retinopathy like ophthalmologists. Algorithms are being developed to predict which patients will get diarrhea or end up in the ICU, and the FDA recently approved the first machine learning algorithm to measure how much blood flows through the heart - a tedious, time-consuming calculation traditionally done by cardiologists.

It's enough to make doctors like myself wonder why we spent a decade in medical training learning the art of diagnosis and treatment.

There are many questions about whether A.I. actually works in medicine, and where it works: can it pick up pneumonia, detect cancer, predict death? But those questions focus on the technical, not the ethical. And in a health system riddled with inequity, we have to ask: Could the use of A.I. in medicine worsen health disparities?

There are at least three reasons to believe it might.

The first is a training problem. A.I. must learn to diagnose disease on large data sets, and if that data doesn't include enough patients from a particular background, it won't be as reliable for them. Evidence from other fields suggests this isn't just a theoretical concern. A recent study found that some facial recognition programs incorrectly classify less than 1 percent of light-skinned men but more than one-third of dark-skinned women. What happens when we rely on such algorithms to diagnose melanoma on light versus dark skin?

Medicine has long struggled to include enough women and minorities in research, despite knowing they have different risk factors for and manifestations of disease. Many genetic studies suffer from a dearth of black patients, leading to erroneous conclusions. Women often experience different symptoms when having a heart attack, causing delays in treatment. Perhaps the most widely used cardiovascular risk score, developed using data from mostly white patients, can be less precise for minorities.

Will using A.I. to tell us who might have a stroke, or which patients will benefit a clinical trial, codify these concerns into algorithms that prove less effective for underrepresented groups?

Second, because A.I. is trained on real-world data, it risks incorporating, entrenching and perpetuating the economic and social biases that contribute to health disparities in the first place. Again, evidence from other fields is instructive. A.I. programs used to help judges predict which criminals are most likely to reoffend have shown troubling racial biases, as have those designed to help child protective services decide which calls require further investigation.

In medicine, unchecked A.I. could create self-fulfilling prophesies that confirm our pre-existing biases, especially when used for conditions with complex trade-offs and high degrees of uncertainty. If, for example, poorer patients do worse after organ transplantation or after receiving chemotherapy for end-stage cancer, machine learning algorithms may conclude such patients are less likely to benefit from further treatment - and recommend against it.

Finally, even ostensibly fair, neutral A.I. has the potential to worsen disparities if its implementation has disproportionate effects for certain groups. Consider a program that helps doctors decide whether a patient should go home or to a rehab facility after knee surgery. It's a decision imbued with uncertainty but has real consequences: Evidence suggests discharge to an institution is associated with higher costs and higher risk of readmission. If an algorithm incorporates residence in a low-income neighborhood as a marker for poor social support, it may recommend minority patients go to nursing facilities instead of receive home-based physical therapy. Worse yet, a program designed to maximize efficiency or lower medical costs might discourage operating on those patients altogether.

To some extent, all these problems already exist in medicine. American health care has always struggled with income- and race-based inequities rooted in various forms of bias. The risk with A.I. is that these biases become automated and invisible - that we begin to accept the wisdom of machines over the wisdom of our own clinical and moral intuition. Many A.I. programs are black boxes: We don't know exactly what's going on inside and why they produce the output they do. But we may increasingly be expected to honor their recommendations.

In my practice, I've often seen how any tool can quickly become a crutch - an excuse to outsource decision making to someone or something else. Medical students struggling to interpret an EKG inevitably peek at the computer-generated output at the top of the sheet. I myself am often swayed by the report provided alongside a chest X-ray or CT scan. As automation becomes pervasive, will we catch that spell-check autocorrected "they're" to "there" when we meant "their"?

Still, A.I. holds tremendous potential to improve medicine. It may well make care more efficient, more accurate and - if properly deployed - more equitable. But realizing this promise requires being aware of the potential for bias and guarding against it. It means regularly monitoring both the output of algorithms and the downstream consequences. In some cases, this will necessitate counter-bias algorithms that hunt for and correct subtle, systematic discrimination.

But most fundamentally, it means recognizing that humans, not machines, are still responsible for caring for patients. It is our duty to ensure that we're using AI as another tool at our disposal - not the other way around.

Dhruv Khullar (@DhruvKhullar) is a doctor at NewYork-Presbyterian Hospital, an assistant professor in the departments of medicine and health care policy at Weill Cornell Medicine, and director of policy dissemination at the Physicians Foundation Center for the Study of Physician Practice and Leadership.
Follow The New York Times Opinion section on Facebook, Twitter (@NYTopinion) and Instagram.

# data[].attributes.attachment_link.title
Opinion | A.I. Could Worsen Health Disparities (Published 2019)

# data[].attributes.attachment_link.description
In a health system riddled with inequity, we risk making dangerous biases automated and invisible. By Dr. Khullar is an assistant professor of health care policy and research. Artificial intelligence is beginning to meet (and sometimes exceed) assessments by doctors in various clinical situations. A.I.

# data[].attributes.body
(19:22-21:26)

# data[].attributes.attachment_link.title
Artificial Intelligence: Last Week Tonight with John Oliver (HBO)

# data[].attributes.attachment_link.description
Experience an integrated media property for tech workers—latest news, explainers and market insights to help stay ahead of the curve.

# data[].attributes.attachment_link.readable_text
As a step toward improving our ability to identify and manage the harmful effects of bias in artificial intelligence (AI) systems, researchers at the National Institute of Standards and Technology (NIST) recommend widening the scope of where we look for the source of these biases - beyond the machine learning processes and data used to train AI software to the broader societal factors that influence how technology is developed.

The recommendation is a core message of a revised NIST publication, Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (NIST Special Publication 1270), which reflects public comments the agency received on its draft version released last summer. As part of a larger effort to support the development of trustworthy and responsible AI, the document offers guidance connected to the AI Risk Management Framework that NIST is developing.

According to NIST's Reva Schwartz, the main distinction between the draft and final versions of the publication is the new emphasis on how bias manifests itself not only in AI algorithms and the data used to train them, but also in the societal context in which AI systems are used.

"Context is everything," said Schwartz, principal investigator for AI bias and one of the report's authors. "AI systems do not operate in isolation. They help people make decisions that directly affect other people's lives. If we are to develop trustworthy AI systems, we need to consider all the factors that can chip away at the public's trust in AI. Many of these factors go beyond the technology itself to the impacts of the technology, and the comments we received from a wide range of people and organizations emphasized this point."

Bias in AI can harm humans. AI can make decisions that affect whether a person is admitted into a school, authorized for a bank loan or accepted as a rental applicant. It is relatively common knowledge that AI systems can exhibit biases that stem from their programming and data sources; for example, machine learning software could be trained on a dataset that underrepresents a particular gender or ethnic group. The revised NIST publication acknowledges that while these computational and statistical sources of bias remain highly important, they do not represent the full picture.

A more complete understanding of bias must take into account human and systemic biases, which figure significantly in the new version. Systemic biases result from institutions operating in ways that disadvantage certain social groups, such as discriminating against individuals based on their race. Human biases can relate to how people use data to fill in missing information, such as a person's neighborhood of residence influencing how likely authorities would consider the person to be a crime suspect. When human, systemic and computational biases combine, they can form a pernicious mixture - especially when explicit guidance is lacking for addressing the risks associated with using AI systems.

"If we are to develop trustworthy AI systems, we need to consider all the factors that can chip away at the public's trust in AI. Many of these factors go beyond the technology itself to the impacts of the technology." -Reva Schwartz, principal investigator for AI bias

To address these issues, the NIST authors make the case for a "socio-technical" approach to mitigating bias in AI. This approach involves a recognition that AI operates in a larger social context - and that purely technically based efforts to solve the problem of bias will come up short.

"Organizations often default to overly technical solutions for AI bias issues," Schwartz said. "But these approaches do not adequately capture the societal impact of AI systems. The expansion of AI into many aspects of public life requires extending our view to consider AI within the larger social system in which it operates."

Socio-technical approaches in AI are an emerging area, Schwartz said, and identifying measurement techniques to take these factors into consideration will require a broad set of disciplines and stakeholders.

"It's important to bring in experts from various fields - not just engineering - and to listen to other organizations and communities about the impact of AI," she said.

NIST is planning a series of public workshops over the next few months aimed at drafting a technical report for addressing AI bias and connecting the report with the AI Risk Management Framework. For more information and to register, visit the AI RMF workshop page.

# data[].attributes.attachment_link.title
There's More to AI Bias Than Biased Data, NIST Report Highlights

# data[].attributes.attachment_link.description
As a step toward improving our ability to identify and manage the harmful effects of bias in artificial intelligence (AI) systems, researchers at the National Institute of Standards and Technology (NIST) recommend widening the scope of where we look for the source of these biases - beyond the machine learning processes and data used to train AI software to the broader societal factors that influence how technology is developed.

# data[].attributes.attachment_link.readable_text
Artificial intelligence image tools have a tendency to spin up disturbing clichés: Asian women are hypersexual. Africans are primitive. Europeans are worldly. Leaders are men. Prisoners are Black.

These stereotypes don't reflect the real world; they stem from the data that trains the technology. Grabbed from the internet, these troves can be toxic - rife with pornography, misogyny, violence and bigotry.

Every image in this story shows something that doesn't exist in the physical world and was generated using Stable Diffusion, a text-to-image artificial intelligence model.

Stability AI, maker of the popular image generator Stable Diffusion XL, told The Washington Post it had made a significant investment in reducing bias in its latest model, which was released in July. But these efforts haven't stoppedit from defaulting to cartoonish tropes. The Post found that despite improvements,the tool amplifies outdated Western stereotypes, transferring sometimes bizarre clichés to basic objects, such as toys or homes.

"They're sort of playing whack-a-mole and responding to what people draw the most attention to," said Pratyusha Kalluri, an AI researcher at Stanford University.

Christoph Schuhmann, co-founder of LAION, a nonprofit behind Stable Diffusion's data, argues that image generators naturallyreflect the world of White people because the nonprofit that provides data to many companies, including LAION, doesn't focus on China and India, the largest population of web users.

[ Inside the secret list of websites that make AI like ChatGPT sound smart]

When we asked Stable Diffusion XL to produce a house in various countries, it returned clichéd concepts for each location: classical curved roof homes for China, rather than Shanghai's high-rise apartments; idealized American houses with trim lawns and ample porches; dusty clay structures on dirt roads in India, home to more than 160 billionaires, as well as Mumbai, the world's 15th richest city.

United States

"This will give you the average stereotype of what an average person from North America or Europe thinks," Schuhmann said. "You don't need a data science degree to infer this."

Stable Diffusion is not alone in this orientation. In recently released documents, OpenAI said its latest image generator, DALL-E 3, displays "a tendency toward a Western point-of-view" with images that "disproportionately represent individuals who appear White, female, and youthful."

As synthetic images spread across the web, they could give new life to outdated and offensive stereotypes, encoding abandoned ideals around body type, gender and race into the future of image-making.

Predicting the next pixel

Like ChatGPT, AI image tools learn about the world through gargantuan amounts of training data. Instead of billions of words, they are fed billions of pairs of images and their captions, also scraped from the web.

Tech companies have grown increasingly secretive about the contents of these data sets, partially because the text and images included often contain copyrighted, inaccurate or even obscene material. In contrast, Stable Diffusion and LAION, are open source projects, enabling outsiders to inspect details of the model.

Stability AI chief executive Emad Mostaque said his company views transparency as key to scrutinizing and eliminating bias. "Stability AI believes fundamentally that open source models are necessary for extending the highest standards in safety, fairness, and representation," he said in a statement.

Images in LAION, like many data sets, were selected because they contain code called "alt-text," which helps software describe images to blind people. Though alt-text is cheaper and easier than adding captions, it's notoriously unreliable - filled with offensive descriptions and unrelated terms intended to help images rank high in search.

[ AI can now create images out of thin air. See how it works. ]

Image generators spin up pictures based on the most likely pixel, drawing connections between words in the captions and the images associated with them. These probabilistic pairings help explain some of the bizarre mashups churned out by Stable Diffusion XL, such as Iraqi toys that look like U.S. tankers and troops. That's not a stereotype: it reflects America's inextricable association between Iraq and war.

Misses biases

Despite the improvements in SD XL, The Post was able to generate tropes about race, class, gender, wealth, intelligence, religion and other cultures by requesting depictions of routine activities, common personality traits or the name of another country. In many instances, the racial disparities depicted in these images are more extreme than in the real world.

For example, in 2020, 63 percent of food stamp recipients were White and 27 percent were Black, according to the latest data from the Census Bureau's Survey of Income and Program Participation. Yet, when we prompted the technology to generate a photo of a person receiving social services, it generated only non-White and primarily darker-skinned people. Results for a "productive person," meanwhile, were uniformly male, majority White, and dressed in suits for corporate jobs.

a person at social services

a productive person

Last fall, Kalluri and her colleagues also discovered that the tools defaulted to stereotypes. Asked to provide an image of "an attractive person," the tool generated light-skinned, light-eyed, thin people with European features. A request for a "a happy family" produced images of mostly smiling, White, heterosexual couples with kids posing on manicured lawns.

Kalluri and the others also found the tools distorted real world statistics. Jobs with higher incomes like "software developer" produced representations that skewed more White and male than data from the Bureau of Labor Statistics would suggest. White-appearing people also appear in the majority of images for "chef," a more prestigious food preparation role, while non-White people appear in most images of "cooks" - though the Labor Bureau's statistics show that a higher percentage of "cooks" self-identify as White than "chefs."

Cleaner data, cleaner results

Companies have long known about issues with the data behind this technology. ImageNet, a pivotal 2009 training set of 14 million images, was in use for more than a decade before researchers found disturbing content, including nonconsensual sexual images, in which women were sometimes easily identifiable. Some images were sorted into categories labeled with slurs such as "Closet Queen," "Failure," "mulatto," "nonperson," "pervert," and "Schizophrenic."

ImageNet's authors eliminated most of the categories, but many contemporary data sets are built the same way, using images obtained without consent and categorizing people like objects.

Efforts to detoxify AI image tools have focused on a few seemingly fruitful interventions: filtering data sets, finessing the final stages of development, and encoding rules to address issues that earned the company bad PR.

For example, Stable Diffusion drew negative attention when requests for a "Latina" produced images of women in suggestive poses wearing little to no clothing. A more recent system (version 2.1) generated more innocuous images.

a Latina (version 1.5)

a Latina (version 2.1)

Why the difference? A Post analysis found the training data for the first version contained a lot more pornography.

Of the training images captioned "Latina," 20 percent of captions or URLs also included a pornographic term. More than 30 percent were marked as almost certain to be "unsafe" by a LAION detector for not-safe-for-work content. In subsequent Stable Diffusion models, the training data excluded images marked as possibly "unsafe," producing images that appear markedly less sexual.

The Post's findings track with prior research that found images of sexual abuse and rape in the data set used for Stable Diffusion 1, as well as images that sexualized Black women and fetishized Asian women. In addition to removing "unsafe" images, Ben Brooks, Stability AI's head of public policy, said the company was also careful to block child sexual abuse material (CSAM) and other high-risk imagery for SD2.

Filtering the "bad" stuff out of a data set isn't an easy fix-all for bias, said Sasha Luccioni, a research scientist at Hugging Face, an open source repository for AI and one of LAION's corporate sponsors. Filtering for problematic content using keywords in English, for example, may remove a lot of porn and CSAM, but it may also result in more content overall from the global north, where platforms have a longer history of generating high-quality content and stronger restrictions on posting porn, she said.

"All of these little decisions can actually make cultural bias worse," Luccioni said.

Even prompts to generate photos of everyday activities slipped into tropes. Stable Diffusion XL defaulted to mostly darker-skinned male athletes when we prompted the system to produce images for "soccer," while depicting only women when asked to show people in the act of "cleaning." Many of the women were smiling, happily completing their feminine household chores.

playing soccer

Stability AI argues each country should have its own national image generator, one that reflects national values, with data sets provided by the government and public institutions.

Reflecting the diversity of the web has recently become "an area of active interest" for Common Crawl, a 16-year-old nonprofit that has long provided text scraped from the web for Google, LAION, and many other tech firms, executive director Rich Skrenta told The Post. Its crawler scrapes content based on the organization's internal ranking of what's central to the internet, but is not instructed to focus on a specific language or country.

"If there is some kind of bias in the crawl and if it's not probing as deeply into, say, Indian websites," that is something Common Crawl would like to measure and fix, he said.

The endless task of eradicating bias

The AI field is divided on how to address bias.

For Kalluri, mitigating bias in images is fundamentally different than in text. Any prompt to create a realistic image of a person has to make decisions about age, body, race, hair, background and other visual characteristics, she said. Few of these complications lend themselves to computational solutions, Kalluri said.

Kalluri believes it's important for anyone who interacts with the technology to understand how it operates. "They're just predictive models," she said,portraying things based on the snapshot of the internet in their data set.

[ See why AI like ChatGPT has gotten so good, so fast]

Even using detailed prompts didn't mitigate this bias. When we asked for a photo of a wealthy person in different countries, Stable Diffusion XL still produced a mishmash of stereotypes: African men in Western coats standing in front of thatched huts, Middle Eastern men posed in front of ancient mosques, while European men in slim-fitting suits wandered quaint cobblestone streets.

the Middle East

Abeba Birhane, senior advisor for AI accountability at the Mozilla Foundation, contends that the tools can be improved if companies work hard to improve the data - an outcome she considers unlikely. In the meantime, the impact of these stereotypes will fall most heavily on the same communities harmed during the social media era, she said, adding: "People at the margins of society are continually excluded."

# data[].attributes.attachment_link.title
These fake images reveal how AI amplifies our worst stereotypes

# data[].attributes.attachment_link.description
AI image generators like Stable Diffusion and DALL-E amplify bias in gender, race and beyond, despite efforts to detoxify the data fueling these results.

# data[].attributes.attachment_link.title
Eliminating bias in AI may be impossible – a computer scientist explains how to tame it instead

# data[].attributes.attachment_link.description
Creating bias-free AI systems is easier said than done. A computer scientist explains how controlling bias could lead to fairer AI.

# data[].attributes.body
This reading is about the issue of AI bias when it comes to machine learning. AI bias is when AI produces biased results. This issue is significant because it can result in errors or unfair outcomes. For example, healthcare systems have shown that they have lower accuracy when it comes to minority patients than white patients. This reading does affect the way I think about machine learning because if inaccurate results are being produced, what's the point of using AI. However I do think this is a solvable issue. It can be solved by first removing the bias from ourselves. -Abrar

# data[].attributes.body
This reading is about the risk of using AI and machine learning in the medical field. Using AI in the medical field is not reliable because AI would need a large data set of people from different backgrounds or the diagnosis may be wrong. This issue is significant because people being diagnosed with the wrong health conditions could worsen their health or even lose their lives. This reading does affect the way I think about AI in the medical field. It has made me realize why we can’t use it when it comes to health. -Abrar