# data[].attributes.headline
Ethical Issues in Artificial Intelligence

# data[].attributes.subject
Ethical Issues in Artificial Intelligence

# data[].attributes.attachment_link.readable_text
Faced with an automated future, what moral framework should guide us? Image: Matthew Wiebe

Alumni, Global Shapers Community, Fathom Computing

Stay up to date:

Emerging Technologies

Follow

Don't miss any update on this topic

Create a free account and access your personalized content collection with our latest publications and analyses.

Sign up for free

License and Republishing

World Economic Forum articles may be republished in accordance with the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International Public License, and in accordance with our Terms of Use.

The views expressed in this article are those of the author alone and not the World Economic Forum.

Global Agenda

The Agenda Weekly

A weekly update of the most important issues driving the global agenda

More on Emerging Technologies

See all

# data[].attributes.attachment_link.title
Top 9 ethical issues in artificial intelligence

# data[].attributes.attachment_link.description
The robot revolution is gaining pace, but is it running in line with our values? Here are some of the main ethical issues keeping the AI experts up at night.

# data[].attributes.body
An article by Abeba Birhane and Fred Cummins

# data[].attributes.headline
AI : Does the energy used to power AI outweigh the cost to the environment?

# data[].attributes.subject
AI : Does the energy used to power AI outweigh the cost to the environment?

# data[].attributes.attachment_link.readable_text
Recent advances in natural language processing-an AI technique that helps machines parse, interpret, and generate text-have proven especially power-hungry. A research paper from a team at UMass Amherst found that training a single large NLP model may consume as much energy as a car over its entire lifetime-including the energy needed to build it.

Training a powerful machine-learning algorithm often means running huge banks of computers for days, if not weeks. The fine-tuning required to perfect an algorithm, by for example searching through different neural network architectures to find the best one, can be especially computationally intensive. For all the hand-wringing, though, it remains difficult to measure how much energy AI actually consumes, and even harder to predict how much of a problem it could become.

The Department of Energy estimates that data centers account for about 2 percent of total US electricity usage. Worldwide, data centers consume about 200 terawatt hours of power per year-more than some countries. And the forecast is for significant growth over the next decade, with some predicting that by 2030, computing and communications technology will consume between 8 percent and 20 percent of the world's electricity, with data centers accounting for a third of that.

In recent years, companies offering cloud computing services have sought to address spiraling power consumption and offset carbon emissions with varying measures of success. Google, for example, claims "zero net carbon emissions" for its data centers, thanks to extensive renewable energy purchases. Microsoft last week announced a plan to become "carbon negative" by 2030, meaning it would offset all of the carbon produced by the company over its history. OpenAI signed a deal to use Microsoft's cloud last July.

It isn't clear how the AI boom will fit with the bigger picture of data center energy use, or how it might alter it. Cloud providers do not disclose the overall energy demands of machine-learning systems. Microsoft, Amazon, and Google all declined to comment.

Jonathan Koomey, a researcher and consultant who tracks data center energy use, cautions against drawing too many conclusions from cutting-edge AI demos. He notes that AI algorithms often run on specialized chips that are more efficient, so new chip architectures may offset some of the projected demand for compute power. He also says that the IT industry has in the past offset rising energy demands in one domain by lowering energy use in others. "People are likely to take isolated anecdotes and extrapolate to get eye popping numbers, and these numbers are almost always too high," Koomey says.

Still, as companies and other organizations increasingly use artificial intelligence, experts say it will become important to understand the technology's energy footprint, both in data centers and in other devices and gadgets. "I would agree that the analysis community needs to get a handle on it," says Eric Masanet, a professor at Northwestern University who leads its Energy and Resource Systems Analysis Laboratory.

Some AI researchers aren't waiting for the industry to wake up. Luccioni of Mila helped organize a workshop on climate change last month at an important AI conference, NeurIPS, and she was pleased to find that the event was standing room only. "There's a lot of interest in this," she says.

# data[].attributes.attachment_link.title
AI Can Do Great Things-if It Doesn't Burn the Planet

# data[].attributes.attachment_link.description
Recent advances in natural language processing-an AI technique that helps machines parse, interpret, and generate text-have proven especially power-hungry. A research paper from a team at UMass Amherst found that training a single large NLP model may consume as much energy as a car over its entire lifetime-including the energy needed to build it.

# data[].attributes.headline
Fanify: Using AI to help budding musicians hit the right note with fans

# data[].attributes.subject
Fanify: Using AI to help budding musicians hit the right note with fans

# data[].attributes.attachment_link.readable_text
Dundalk-based Fanify, which raised significant funding last month, is on a mission to help musicians find and grow their fanbase using AI-driven machine learning.

Artificial intelligence has been around for a while now, but only in the last few years has interest in the technology exploded - most recently because of ChatGPT.

AI technologies are in the early stages of revolutionising conventionally human tasks such as writing, software development and even games. And while AI-generated music may take a while to take off, the voguish technology has found ways to make some noise in the music industry.

One such example of using AI in music is the Dundalk-based start-up Fanify. Founded by musician-turned-entrepreneur Pete Rust and business expert Dagan Fleming, Fanify is on a mission to help budding musicians establish and grow their fanbase.

'Go back to playing music'

Using AI-driven machine learning technology, Fanify aims to fully automate the music promotion and fanbase building process for solo artists and bands, allowing them to focus on the creative process while finding them an audience that can connect with their music.

"Fanify finds and tracks fans, people who really engage with a band or artist's content and continues to re-engage them over time growing and solidifying the fan-artist relationship," Rust told SiliconRepublic.com recently.

"Our motto is 'Go back to playing music and let Fanify build your fanbase for you'".

Founded in 2021, Fanify has developed technology that it calls 'new music audience' which uses music promotion data from thousands of bands and artists as well as "the vast marketing and data analytic infrastructure" of Facebook and Google to find fans for users.

It does so by showing their promotional content to people who love to discover new music and are interested in the same type of music as theirs.

"There is a lot of serious advice from music marketing professionals saying that any serious independent musician or band should be spending at least 20 hours a week on their music promotion and marketing. That independent musicians should by and large have a professional level of understanding of arcane marketing tools and techniques," Rust explained.

"This idea goes against the advancements that have been made across the entire music technology sphere. Fanify is here to support musicians, bands and artists and to save them time and effort."

Rust claims that Fanify users need no technical expertise in marketing or technology and that building a fanbase using its technology will require minimal effort.

"Our users' fans are real people. We don't sell pay-for-play views from click farms or bots. Those methods muddy the data, causing harm to your music career," he went on.

"The algorithms of the major video content platforms, YouTube, Facebook and TikTok, are not looking for an audience for your video, they're looking for videos for their users. Fanify is the platform that finds an audience for your videos."

Striking a chord with artists

Having been a professional musician and promoter for more than 12 years, Rust alongside his colleague Fleming, previously founded another music start-up, Dundalk-based studio Lockup Records. Rust described the company as "a cornerstone of the music scene in the north-east of Ireland".

"My experiences in the music industry have given me real world insight into the pain and difficulties that most bands and artists have in trying to promote their music," he said.

"They spend a lot of time and money through the whole process of writing, rehearsing, recording and releasing their music. They will equally invest in music promotion but there is no service currently that fully automates the process of music promotion and growing a fanbase.

"Any avenue available for bands and artists to do this requires a steep learning curve, trying to understand complicated marketing tools and learning digital marketing concepts."

While Rust heads the business as CEO, Fleming leads Fanify's business development. With more than 15 years of experience in project management and sales, Fleming produced hundreds of live events and recording sessions as general manager of Lockup Records.

Just last month, Fanify raised €400,000 in pre-seed funding led by April Ventures. Other backers include Global Perspectives, a vehicle established by Shane Brett, who previously founded fintech company Gecko Governance.

Rust said they are planning a further raise later this year "to continue building out our roadmap and growing our userbase globally".

"We have already been contacted by several potential investors and have had positive preliminary meetings. We are always willing to hear from potential future partners about how we can take Fanify to the next level."
10 things you need to know direct to your inbox every weekday. Sign up for the Daily Brief, Silicon Republic's digest of essential sci-tech news.

# data[].attributes.attachment_link.title
Fanify: Using AI to help budding musicians hit the right note with fans

# data[].attributes.attachment_link.description
Dundalk-based Fanify, which raised significant funding last month, is on a mission to help musicians find and grow their fanbase using AI-driven machine learning. Artificial intelligence has been around for a while now, but only in the last few years has interest in the technology exploded - most recently because of ChatGPT.

# data[].attributes.body
Ireland’s first AI ambassador, Dr Patricia Scanlon, spoke to Jenny Darmody of "Silicon Republic" about where we are in the AI industry and what the future may hold

# data[].attributes.attachment_link.readable_text
Ireland's first AI ambassador, Dr Patricia Scanlon, spoke to Jenny Darmody about where we are in the AI industry and what the future may hold.

With the AI race heating up, I took the opportunity to speak to Dr Patricia Scanlon, Ireland's first AI ambassador.

Scanlon was appointed to the post in May last year, having had a strong presence in the sector for several years.

The Kildare native founded SoapBox Labs in 2013, which develops accurate and secure voice technology for children. Scanlon stepped down as CEO of SoapBox Labs in 2021.

Now in her new role as the AI ambassador, she hopes to help "start a national conversation about AI" as part of the Ireland's AI strategy.

"You can't help but note how much AI is in the news recently and for the past year," she said. "We're all talking about it, but a lot of people still probably struggle with the concept of what AI is, what it's not, what are the benefits, etc."

'Stuff that used to take us weeks now takes hours'
- PATRICIA SCANLON

She also said that while many people are aware of the dangers of AI through the media, movies and science fiction, it's hard to discern the fact from the fiction.

"I do think there's a huge interest in starting to talk about it and then having this role is helping to facilitate that conversation with young people, with people in businesses, across government, and universities."

AI has indeed become a hugely hot topic in recent years. From bias embedded within AI systems to ethical questions around facial recognition and surveillance, it has become a central discussion among both industry players and lawmakers.

The last few years alone saw the unveiling of the EU's AI Act, which aims to implement varying degrees of restrictions on AI systems depending on their risk level.

And now, we see the rise of the AI arms race, as Big Tech players such as Microsoft and Google go head-to-head with their own versions of advanced natural language models.

However, the concept of AI itself is far from new. In fact, Scanlon said it was already a "mature enough area" when she started her PhD back in 2000.

What has changed in the last decade is the advancements and availability of graphics processing units (GPUs), huge processors that are able to crunch the numbers of very large datasets, making it a powerful tool for machine learning.

"Stuff that used to take us weeks now takes hours," said Scanlon. "But also, with the availability of tools and cloud processing on demand cheaply, anybody can start in AI, which is very different to what was 10 or 12 years ago."

The growth of AI chatbots

With those present-day advancements in mind, the sudden explosion of natural language models and ChatGPT in particular has whipped up a storm with millions of users getting the chance to try these tools out.

Scanlon said she couldn't help but marvel at the effect ChatGPT has had because it is essentially "a user interface that was put on something that's been around for a couple of years".

"Suddenly opening up and letting people play with it is just going to unleash a huge amount of use cases that probably even the creators hadn't conceived. So, I think that's where it gets really interesting," she said.

While Scanlon agreed with the general consensus that these chatbots sound very intelligent, it is also very limited in what it can do, at least for now.

"That kind of awareness needs to happen as well. It isn't sentient, it sounds very intelligent. But really the logic or the facts or the thinking behind it isn't there so don't be deceived."

While the growth of these tools have thrown up some cause for concern around the information it spits back out, Scanlon said it has actually been "excellent" in terms of pulling back the curtain on AI.

"I think before, people probably thought there was more mystery and intelligence to it. Now, people are beginning to see the power of it, but they're also seeing very blatantly, the limitations," she said.

"There's countless of examples being thrown up of how it's wrong, how it's limited. To be honest, I think as an education thing, ChatGPT has been excellent in that regard, because it's very easy."

The challenge of bias

Chatbots aren't the only tool getting a massive AI boost. Areas such as recruitment, healthcare and security are increasingly seeing the use of AI and machine learning, but regulating all of this is going to be massive challenge.

"You have to ensure that there isn't bias. You have to deeply think about a problem to know how to test for bias and anti-bias," said Scanlon.

She said that fairness, transparency and explainability are important elements of all AI systems; being able to understand how it came to a certain conclusion, making sure it works the same for all people and ensuring it is free from historical and systemic biases.

'You could create a very objective, non-biased system...if it was trained right'
- PATRICIA SCANLON

"I think it's really important to realise that no AI is biased just like no child is born racist, that has to be cultivated in some way. And if that's the programmers, if that's the data, if that's people really not deeply thinking about how it affects humans, you won't get a fair and unbiased system," she said.

However, the bias challenge within AI is a big one to overcome, Scanlon said it can also be "a holy grail" if trained correctly because it could remove the danger of human bias.

"We have to be careful of [always thinking] that the human loop is the one that's going to solve the bias problem, when we've never really done that before when humans are evaluating CVs or assessments or loans or jail sentencing," she said.

"There is a holy grail here where maybe if AI is built correctly, that you could create a very objective, non-biased system, maybe less bias than any human eventually, if it was trained right. But there's a long way to go to get to that goal."

Ireland 'ahead of the curve' in some ways

With AI becoming such an important area, I asked Scanlon where she thought Ireland was in that area. She said there has been "nonstop conversations" around the technology, the interest, the skills and the country's strategy.

"I'm aware of nonstop efforts being made around skills to make sure we have enough skills, to make sure that people are being trained. Sometimes people talk about the displacement of jobs in AI, but equally we're hearing nothing but the lack of skills in AI and you're going to need non-technical people and technical people to help develop this," she said.

"I think Ireland is ahead of the curve in terms of the attention [AI is] being paid. Not ahead of it completely, but definitely up there with other leading countries in terms of making sure our SMEs and our companies and our Government processes are incorporating AI. It's very early days, but the interest is there."

She added that she doesn't believe any one country is leading the field because the interest and capabilities are so new. She also said that Ireland is in a good position to be one of the leaders, especially if it takes a focused approach on ethical AI.

"Sometimes I think technology is often rushed to market to get a competitive advantage, but doing it more thoughtfully could potentially give us that lead in the future because sometimes when you build massive AI systems and you have them going for years, it's hard to go back and correct it," she said.

"You should focus on the ethical AI foundations because it will be good for society, be good for humans and be good for businesses. I've been impressed with the focus on that so far, so I'm hopeful that we can take the lead on that."
10 things you need to know direct to your inbox every weekday. Sign up for the Daily Brief, Silicon Republic's digest of essential sci-tech news.

# data[].attributes.attachment_link.title
AI ambassador: Ireland could 'take the lead' in ethical AI

# data[].attributes.attachment_link.description
Ireland's first AI ambassador, Dr Patricia Scanlon, spoke to Jenny Darmody about where we are in the AI industry and what the future may hold. With the AI race heating up, I took the opportunity to speak to Dr Patricia Scanlon, Ireland's first AI ambassador.

# data[].attributes.attachment_link.readable_text
On a spring afternoon in 2014, Brisha Borden was running late to pick up her god-sister from school when she spotted an unlocked kid's blue Huffy bicycle and a silver Razor scooter. Borden and a friend grabbed the bike and scooter and tried to ride them down the street in the Fort Lauderdale suburb of Coral Springs.

Just as the 18-year-old girls were realizing they were too big for the tiny conveyances - which belonged to a 6-year-old boy - a woman came running after them saying, "That's my kid's stuff." Borden and her friend immediately dropped the bike and scooter and walked away.

But it was too late - a neighbor who witnessed the heist had already called the police. Borden and her friend were arrested and charged with burglary and petty theft for the items, which were valued at a total of $80.

Subscribe to the Series

Machine Bias: Investigating the algorithms that control our lives.

Compare their crime with a similar one: The previous summer, 41-year-old Vernon Prater was picked up for shoplifting $86.35 worth of tools from a nearby Home Depot store.

Prater was the more seasoned criminal. He had already been convicted of armed robbery and attempted armed robbery, for which he served five years in prison, in addition to another armed robbery charge. Borden had a record, too, but it was for misdemeanors committed when she was a juvenile.

Yet something odd happened when Borden and Prater were booked into jail: A computer program spat out a score predicting the likelihood of each committing a future crime. Borden - who is black - was rated a high risk. Prater - who is white - was rated a low risk.

Two years later, we know the computer algorithm got it exactly backward. Borden has not been charged with any new crimes. Prater is serving an eight-year prison term for subsequently breaking into a warehouse and stealing thousands of dollars' worth of electronics.

Scores like this - known as risk assessments - are increasingly common in courtrooms across the nation. They are used to inform decisions about who can be set free at every stage of the criminal justice system, from assigning bond amounts - as is the case in Fort Lauderdale - to even more fundamental decisions about defendants' freedom. In Arizona, Colorado, Delaware, Kentucky, Louisiana, Oklahoma, Virginia, Washington and Wisconsin, the results of such assessments are given to judges during criminal sentencing.

Rating a defendant's risk of future crime is often done in conjunction with an evaluation of a defendant's rehabilitation needs. The Justice Department's National Institute of Corrections now encourages the use of such combined assessments at every stage of the criminal justice process. And a landmark sentencing reform bill currently pending in Congress would mandate the use of such assessments in federal prisons.

In 2014, then U.S. Attorney General Eric Holder warned that the risk scores might be injecting bias into the courts. He called for the U.S. Sentencing Commission to study their use. "Although these measures were crafted with the best of intentions, I am concerned that they inadvertently undermine our efforts to ensure individualized and equal justice," he said, adding, "they may exacerbate unwarranted and unjust disparities that are already far too common in our criminal justice system and in our society."

The sentencing commission did not, however, launch a study of risk scores. So ProPublica did, as part of a larger examination of the powerful, largely hidden effect of algorithms in American life.

We obtained the risk scores assigned to more than 7,000 people arrested in Broward County, Florida, in 2013 and 2014 and checked to see how many were charged with new crimes over the next two years, the same benchmark used by the creators of the algorithm.

The score proved remarkably unreliable in forecasting violent crime: Only 20 percent of the people predicted to commit violent crimes actually went on to do so.

When a full range of crimes were taken into account - including misdemeanors such as driving with an expired license - the algorithm was somewhat more accurate than a coin flip. Of those deemed likely to re-offend, 61 percent were arrested for any subsequent crimes within two years.

We also turned up significant racial disparities, just as Holder feared. In forecasting who would re-offend, the algorithm made mistakes with black and white defendants at roughly the same rate but in very different ways.

The formula was particularly likely to falsely flag black defendants as future criminals, wrongly labeling them this way at almost twice the rate as white defendants.

White defendants were mislabeled as low risk more often than black defendants.

Could this disparity be explained by defendants' prior crimes or the type of crimes they were arrested for? No. We ran a statistical test that isolated the effect of race from criminal history and recidivism, as well as from defendants' age and gender. Black defendants were still 77 percent more likely to be pegged as at higher risk of committing a future violent crime and 45 percent more likely to be predicted to commit a future crime of any kind. ( Read our analysis.)

The algorithm used to create the Florida risk scores is a product of a for-profit company, Northpointe. The company disputes our analysis.

In a letter, it criticized ProPublica's methodology and defended the accuracy of its test: "Northpointe does not agree that the results of your analysis, or the claims being made based upon that analysis, are correct or that they accurately reflect the outcomes from the application of the model."

Northpointe's software is among the most widely used assessment tools in the country. The company does not publicly disclose the calculations used to arrive at defendants' risk scores, so it is not possible for either defendants or the public to see what might be driving the disparity. (On Sunday, Northpointe gave ProPublica the basics of its future-crime formula - which includes factors such as education levels, and whether a defendant has a job. It did not share the specific calculations, which it said are proprietary.)

Northpointe's core product is a set of scores derived from 137 questions that are either answered by defendants or pulled from criminal records. Race is not one of the questions. The survey asks defendants such things as: "Was one of your parents ever sent to jail or prison?" "How many of your friends/acquaintances are taking drugs illegally?" and "How often did you get in fights while at school?" The questionnaire also asks people to agree or disagree with statements such as "A hungry person has a right to steal" and "If people make me angry or lose my temper, I can be dangerous."

The appeal of risk scores is obvious: The United States locks up far more people than any other country, a disproportionate number of them black. For more than two centuries, the key decisions in the legal process, from pretrial release to sentencing to parole, have been in the hands of human beings guided by their instincts and personal biases.

If computers could accurately predict which defendants were likely to commit new crimes, the criminal justice system could be fairer and more selective about who is incarcerated and for how long. The trick, of course, is to make sure the computer gets it right. If it's wrong in one direction, a dangerous criminal could go free. If it's wrong in another direction, it could result in someone unfairly receiving a harsher sentence or waiting longer for parole than is appropriate.

The first time Paul Zilly heard of his score - and realized how much was riding on it - was during his sentencing hearing on Feb. 15, 2013, in court in Barron County, Wisconsin. Zilly had been convicted of stealing a push lawnmower and some tools. The prosecutor recommended a year in county jail and follow-up supervision that could help Zilly with "staying on the right path." His lawyer agreed to a plea deal.

But Judge James Babler had seen Zilly's scores. Northpointe's software had rated Zilly as a high risk for future violent crime and a medium risk for general recidivism. "When I look at the risk assessment," Babler said in court, "it is about as bad as it could be."

Then Babler overturned the plea deal that had been agreed on by the prosecution and defense and imposed two years in state prison and three years of supervision.

Criminologists have long tried to predict which criminals are more dangerous before deciding whether they should be released. Race, nationality and skin color were often used in making such predictions until about the 1970s, when it became politically unacceptable, according to a survey of risk assessment tools by Columbia University law professor Bernard Harcourt.

In the 1980s, as a crime wave engulfed the nation, lawmakers made it much harder for judges and parole boards to exercise discretion in making such decisions. States and the federal government began instituting mandatory sentences and, in some cases, abolished parole, making it less important to evaluate individual offenders.

But as states struggle to pay for swelling prison and jail populations, forecasting criminal risk has made a comeback.

Dozens of risk assessments are being used across the nation - some created by for-profit companies such as Northpointe and others by nonprofit organizations. (One tool being used in states including Kentucky and Arizona, called the Public Safety Assessment, was developed by the Laura and John Arnold Foundation, which also is a funder of ProPublica.)

There have been few independent studies of these criminal risk assessments. In 2013, researchers Sarah Desmarais and Jay Singh examined 19 different risk methodologies used in the United States and found that "in most cases, validity had only been examined in one or two studies" and that "frequently, those investigations were completed by the same people who developed the instrument."

Their analysis of the research through 2012 found that the tools "were moderate at best in terms of predictive validity," Desmarais said in an interview. And she could not find any substantial set of studies conducted in the United States that examined whether risk scores were racially biased. "The data do not exist," she said.

Since then, there have been some attempts to explore racial disparities in risk scores. One 2016 study examined the validity of a risk assessment tool, not Northpointe's, used to make probation decisions for about 35,000 federal convicts. The researchers, Jennifer Skeem at University of California, Berkeley, and Christopher T. Lowenkamp from the Administrative Office of the U.S. Courts, found that blacks did get a higher average score but concluded the differences were not attributable to bias.

The increasing use of risk scores is controversial and has garnered media coverage, including articles by the Associated Press, and the Marshall Project and FiveThirtyEight last year.

Most modern risk tools were originally designed to provide judges with insight into the types of treatment that an individual might need - from drug treatment to mental health counseling.

"What it tells the judge is that if I put you on probation, I'm going to need to give you a lot of services or you're probably going to fail," said Edward Latessa, a University of Cincinnati professor who is the author of a risk assessment tool that is used in Ohio and several other states.

But being judged ineligible for alternative treatment - particularly during a sentencing hearing - can translate into incarceration. Defendants rarely have an opportunity to challenge their assessments. The results are usually shared with the defendant's attorney, but the calculations that transformed the underlying data into a score are rarely revealed.

"Risk assessments should be impermissible unless both parties get to see all the data that go into them," said Christopher Slobogin, director of the criminal justice program at Vanderbilt Law School. "It should be an open, full-court adversarial proceeding."

Black Defendants' Risk Scores

White Defendants' Risk Scores

These charts show that scores for white defendants were skewed toward lower-risk categories. Scores for black defendants were not. (Source: ProPublica analysis of data from Broward County, Fla.)

Proponents of risk scores argue they can be used to reduce the rate of incarceration. In 2002, Virginia became one of the first states to begin using a risk assessment tool in the sentencing of nonviolent felony offenders statewide. In 2014, Virginia judges using the tool sent nearly half of those defendants to alternatives to prison, according to a state sentencing commission report. Since 2005, the state's prison population growth has slowed to 5 percent from a rate of 31 percent the previous decade.

In some jurisdictions, such as Napa County, California, the probation department uses risk assessments to suggest to the judge an appropriate probation or treatment plan for individuals being sentenced. Napa County Superior Court Judge Mark Boessenecker said he finds the recommendations helpful. "We have a dearth of good treatment programs, so filling a slot in a program with someone who doesn't need it is foolish," he said.

However, Boessenecker, who trains other judges around the state in evidence-based sentencing, cautions his colleagues that the score doesn't necessarily reveal whether a person is dangerous or if they should go to prison.

"A guy who has molested a small child every day for a year could still come out as a low risk because he probably has a job," Boessenecker said. "Meanwhile, a drunk guy will look high risk because he's homeless. These risk factors don't tell you whether the guy ought to go to prison or not; the risk factors tell you more about what the probation conditions ought to be."

"I'm surprised [my risk score] is so low. I spent five years in state prison in Massachusetts." (Josh Ritchie for ProPublica)

Sometimes, the scores make little sense even to defendants.

James Rivelli, a 54-year old Hollywood, Florida, man, was arrested two years ago for shoplifting seven boxes of Crest Whitestrips from a CVS drugstore. Despite a criminal record that included aggravated assault, multiple thefts and felony drug trafficking, the Northpointe algorithm classified him as being at a low risk of reoffending.

"I am surprised it is so low," Rivelli said when told by a reporter he had been rated a 3 out of a possible 10. "I spent five years in state prison in Massachusetts. But I guess they don't count that here in Broward County." In fact, criminal records from across the nation are supposed to be included in risk assessments.

Less than a year later, he was charged with two felony counts for shoplifting about $1,000 worth of tools from Home Depot. He said his crimes were fueled by drug addiction and that he is now sober.

Northpointe was founded in 1989 by Tim Brennan, then a professor of statistics at the University of Colorado, and Dave Wells, who was running a corrections program in Traverse City, Michigan.

Wells had built a prisoner classification system for his jail. "It was a beautiful piece of work," Brennan said in an interview conducted before ProPublica had completed its analysis. Brennan and Wells shared a love for what Brennan called "quantitative taxonomy" - the measurement of personality traits such as intelligence, extroversion and introversion. The two decided to build a risk assessment score for the corrections industry.

Brennan wanted to improve on a leading risk assessment score, the LSI, or Level of Service Inventory, which had been developed in Canada. "I found a fair amount of weakness in the LSI," Brennan said. He wanted a tool that addressed the major theories about the causes of crime.

Brennan and Wells named their product the Correctional Offender Management Profiling for Alternative Sanctions, or COMPAS. It assesses not just risk but also nearly two dozen so-called "criminogenic needs" that relate to the major theories of criminality, including "criminal personality," "social isolation," "substance abuse" and "residence/stability." Defendants are ranked low, medium or high risk in each category.

As often happens with risk assessment tools, many jurisdictions have adopted Northpointe's software before rigorously testing whether it works. New York State, for instance, started using the tool to assess people on probation in a pilot project in 2001 and rolled it out to the rest of the state's probation departments - except New York City - by 2010. The state didn't publish a comprehensive statistical evaluation of the tool until 2012. The study of more than 16,000 probationers found the tool was 71 percent accurate, but it did not evaluate racial differences.

A spokeswoman for the New York state division of criminal justice services said the study did not examine race because it only sought to test whether the tool had been properly calibrated to fit New York's probation population. She also said judges in nearly all New York counties are given defendants' Northpointe assessments during sentencing.

In 2009, Brennan and two colleagues published a validation study that found that Northpointe's risk of recidivism score had an accuracy rate of 68 percent in a sample of 2,328 people. Their study also found that the score was slightly less predictive for black men than white men - 67 percent versus 69 percent. It did not examine racial disparities beyond that, including whether some groups were more likely to be wrongly labeled higher risk.

Brennan said it is difficult to construct a score that doesn't include items that can be correlated with race - such as poverty, joblessness and social marginalization. "If those are omitted from your risk assessment, accuracy goes down," he said.

In 2011, Brennan and Wells sold Northpointe to Toronto-based conglomerate Constellation Software for an undisclosed sum.

Wisconsin has been among the most eager and expansive users of Northpointe's risk assessment tool in sentencing decisions. In 2012, the Wisconsin Department of Corrections launched the use of the software throughout the state. It is used at each step in the prison system, from sentencing to parole.

In a 2012 presentation, corrections official Jared Hoy described the system as a " giant correctional pinball machine " in which correctional officers could use the scores at every "decision point."

Wisconsin has not yet completed a statistical validation study of the tool and has not said when one might be released. State corrections officials declined repeated requests to comment for this article.

Some Wisconsin counties use other risk assessment tools at arrest to determine if a defendant is too risky for pretrial release. Once a defendant is convicted of a felony anywhere in the state, the Department of Corrections attaches Northpointe's assessment to the confidential presentence report given to judges, according to Hoy's presentation.

In theory, judges are not supposed to give longer sentences to defendants with higher risk scores. Rather, they are supposed to use the tests primarily to determine which defendants are eligible for probation or treatment programs.

Prediction Fails Differently for Black Defendants

Overall, Northpointe's assessment tool correctly predicts recidivism 61 percent of the time. But blacks are almost twice as likely as whites to be labeled a higher risk but not actually re-offend. It makes the opposite mistake among whites: They are much more likely than blacks to be labeled lower risk but go on to commit other crimes. (Source: ProPublica analysis of data from Broward County, Fla.)

But judges have cited scores in their sentencing decisions. In August 2013, Judge Scott Horne in La Crosse County, Wisconsin, declared that defendant Eric Loomis had been "identified, through the COMPAS assessment, as an individual who is at high risk to the community." The judge then imposed a sentence of eight years and six months in prison.

Loomis, who was charged with driving a stolen vehicle and fleeing from police, is challenging the use of the score at sentencing as a violation of his due process rights. The state has defended Horne's use of the score with the argument that judges can consider the score in addition to other factors. It has also stopped including scores in presentencing reports until the state Supreme Court decides the case.

"The risk score alone should not determine the sentence of an offender," Wisconsin Assistant Attorney General Christine Remington said last month during state Supreme Court arguments in the Loomis case. "We don't want courts to say, this person in front of me is a 10 on COMPAS as far as risk, and therefore I'm going to give him the maximum sentence."

That is almost exactly what happened to Zilly, the 48-year-old construction worker sent to prison for stealing a push lawnmower and some tools he intended to sell for parts. Zilly has long struggled with a meth habit. In 2012, he had been working toward recovery with the help of a Christian pastor when he relapsed and committed the thefts.

After Zilly was scored as a high risk for violent recidivism and sent to prison, a public defender appealed the sentence and called the score's creator, Brennan, as a witness.

Brennan testified that he didn't design his software to be used in sentencing. "I wanted to stay away from the courts," Brennan said, explaining that his focus was on reducing crime rather than punishment. "But as time went on I started realizing that so many decisions are made, you know, in the courts. So I gradually softened on whether this could be used in the courts or not."

"Not that I'm innocent, but I just believe people do change." (Stephen Maturen for ProPublica)

Still, Brennan testified, "I don't like the idea myself of COMPAS being the sole evidence that a decision would be based upon."

After Brennan's testimony, Judge Babler reduced Zilly's sentence, from two years in prison to 18 months. "Had I not had the COMPAS, I believe it would likely be that I would have given one year, six months," the judge said at an appeals hearing on Nov. 14, 2013.

Zilly said the score didn't take into account all the changes he was making in his life - his conversion to Christianity, his struggle to quit using drugs and his efforts to be more available for his son. "Not that I'm innocent, but I just believe people do change."

Florida's Broward County, where Brisha Borden stole the Huffy bike and was scored as high risk, does not use risk assessments in sentencing. "We don't think the [risk assessment] factors have any bearing on a sentence," said David Scharf, executive director of community programs for the Broward County Sheriff's Office in Fort Lauderdale.

Broward County has, however, adopted the score in pretrial hearings, in the hope of addressing jail overcrowding. A court-appointed monitor has overseen Broward County's jails since 1994 as a result of the settlement of a lawsuit brought by inmates in the 1970s. Even now, years later, the Broward County jail system is often more than 85 percent full, Scharf said.

In 2008, the sheriff's office decided that instead of building another jail, it would begin using Northpointe's risk scores to help identify which defendants were low risk enough to be released on bail pending trial. Since then, nearly everyone arrested in Broward has been scored soon after being booked. (People charged with murder and other capital crimes are not scored because they are not eligible for pretrial release.)

The scores are provided to the judges who decide which defendants can be released from jail. "My feeling is that if they don't need them to be in jail, let's get them out of there," Scharf said.

Scharf said the county chose Northpointe's software over other tools because it was easy to use and produced "simple yet effective charts and graphs for judicial review." He said the system costs about $22,000 a year.

In 2010, researchers at Florida State University examined the use of Northpointe's system in Broward County over a 12-month period and concluded that its predictive accuracy was "equivalent" in assessing defendants of different races. Like others, they did not examine whether different races were classified differently as low or high risk.

Scharf said the county would review ProPublica's findings. "We'll really look at them up close," he said.

Broward County Judge John Hurley, who oversees most of the pretrial release hearings, said the scores were helpful when he was a new judge, but now that he has experience he prefers to rely on his own judgment. "I haven't relied on COMPAS in a couple years," he said.

Hurley said he relies on factors including a person's prior criminal record, the type of crime committed, ties to the community, and their history of failing to appear at court proceedings.

ProPublica's analysis reveals that higher Northpointe scores are slightly correlated with longer pretrial incarceration in Broward County. But there are many reasons that could be true other than judges being swayed by the scores - people with higher risk scores may also be poorer and have difficulty paying bond, for example.

Most crimes are presented to the judge with a recommended bond amount, but he or she can adjust the amount. Hurley said he often releases first-time or low-level offenders without any bond at all.

However, in the case of Borden and her friend Sade Jones, the teenage girls who stole a kid's bike and scooter, Hurley raised the bond amount for each girl from the recommended $0 to $1,000 each.

Hurley said he has no recollection of the case and cannot recall if the scores influenced his decision.

Sade Jones, who had never been arrested before, was rated a medium risk. (Josh Ritchie for ProPublica)

The girls spent two nights in jail before being released on bond.

"We literally sat there and cried" the whole time they were in jail, Jones recalled. The girls were kept in the same cell. Otherwise, Jones said, "I would have gone crazy." Borden declined repeated requests to comment for this article.

Jones, who had never been arrested before, was rated a medium risk. She completed probation and got the felony burglary charge reduced to misdemeanor trespassing, but she has still struggled to find work.

"I went to McDonald's and a dollar store, and they all said no because of my background," she said. "It's all kind of difficult and unnecessary."

Julia Angwin is a senior reporter at ProPublica. From 2000 to 2013, she was a reporter at The Wall Street Journal, where she led a privacy investigative team that was a finalist for a Pulitzer Prize in Explanatory Reporting in 2011 and won a Gerald Loeb Award in 2010.

Lauren Kirchner is a senior reporting fellow at ProPublica. Surya Mattu is a contributing researcher. Design and production by Rob Weychert and David Sleight.

# data[].attributes.attachment_link.description
There's software used across the country to predict future criminals. And it's biased against blacks.

# data[].attributes.body
ChatGPT is a natural language processing tool driven by AI technology that allows you to have human-like conversations and much more with a chatbot. The language model can answer questions, and assist you with tasks such as composing emails, essays, and code. Usage is currently open to public free of charge because ChatGPT is in its research and feedback-collection phase.

# data[].attributes.attachment_link.readable_text
Since ChatGPT launched last year, the hype around the new AI chatbot has grown and grown.

t can write poems, screenplays, articles and even do children's homework.

Not only does it have teachers worried, but companies like Google have increased investment in generative AI, so they can keep up.

In this bonus episode of the Big Tech Show, Adrian takes a look at the chatbot taking the online world by storm.

The Big Tech Show is in association with Square.

# data[].attributes.attachment_link.title
The Big Tech Show BONUS: ChatGPT - What's all the hype about?

# data[].attributes.attachment_link.description
Since ChatGPT launched last year, the hype around the new AI chatbot has grown and grown. t can write poems, screenplays, articles and even do children's homework. Not only does it have teachers worried, but companies like Google have increased investment in generative AI, so they can keep up.