# data[].attributes.attachment_link.readable_text
This blog post is cross-published with OECD at the following link. A recording of the event is available on OECD.ai YouTube channel.
On 12 February 2025, the OECD and IEEE co-organised an event on the margin of the French AI Action Summit with a diverse group of experts to discuss AI's growing environmental challenges. The event had three main sessions about critical sustainability concerns: the environmental cost of inference, the impact of data centres on the electricity grid, and AI's water footprint.

Session 1: The environmental cost of inference

While AI has revolutionized countless industries, its environmental impact is a pressing issue. Energy-intensive training of AI models receive a lot of attention, but it is the inference stage when trained AI models are put into use whose significant long-term environmental cost is becoming clearer. Discussions focused on how inference, which could account for more energy consumption over time than training considering the level of scale, presents a growing challenge for sustainability.
Growing energy consumption from AI inference
Panellists recognised AI inference's substantial energy requirements, especially as models scale and complexify. One speaker pointed out that, for some models, inference's cost now outweighs the training cost after 50 million usages and that current incentives do not encourage companies to optimise AI's energy use. This leads to companies prioritising financial gain over sustainability, underscoring the need for stronger regulatory frameworks to balance technological progress with environmental responsibility. The moderator stressed that optimising the inference process could minimise energy consumption, by reducing, for instance, unnecessary computations at the model level.
The role of developers in optimizing AI inference to reduce the carbon footprint
Developers play a crucial role in minimising the environmental impact of AI. One speaker advocated for critical assessments of software development choices and prioritisation of smaller, more efficient AI models. One panellist said this ties into the growing demand for "green skills" in AI-related professions. Skills such as energy management and environmental, social, and governance (ESG) policy, are becoming increasingly valuable in reducing AI carbon footprint.
Evaluating energy efficiency claims
One speaker also mentioned that while some models like DeepSeek are more energy efficient during training, there are still uncertainties about how efficient they are during the inference phase, especially when they provide longer reasoning time for completing queries. There is still no research on the environmental impact of these recent developments and transparent evaluations are needed to realistically gauge AI's environmental cost.

Session 2: The impact of data centres on the electricity grid

As AI expands, data centres, the backbone of AI systems, are placing increasing pressure on the world's electricity grids. These massive facilities consume vast amounts of energy, which raises concerns about sustainability, grid stability and reliability.
Data centres as major energy consumers
The moderator highlighted the difficulty of obtaining reliable estimates on the energy consumption of data centres, and the extent to which energy consumption can be specifically accounted to AI, making it challenging to fully understand the magnitude of the problem. However, speakers expressed the view that while AI contributes to growing energy demand, it is not the largest driver of global electricity consumption. For example, electrification of the transportation sector outpaces AI in terms of electricity use. One panellist mentioned the importance of real-time cloud energy tracking to reduce uncertainty in calculations of emissions.
Renewable energy and carbon offsetting efforts
Some companies, like Google, have made a commitment to clean electricity in data centres by 2030. However, one speaker warned that even with these efforts, renewable energy supplies may not be sufficient to meet demand by 2025. Given AI's 24/7 energy needs, there is a risk that it may continue to rely on fossil fuels unless there are comprehensive policy changes. One participant noted that the role of nuclear energy, including SMRs (small modular reactors) could play in AI sustainability is often overlooked. Another shared initiative which focuses on reusing heat generated by data centres like AWS's projects in Ireland. However, scaling these efforts responsibly is a significant challenge.
Modulate energy demand
Other aspects were also mentioned, including location of data centres and use of mobile devices for inference which could help offload energy demands, considering the charging of mobile phones during the night.

Session 3: AI water usage

AI-driven data centres don't just consume massive amounts of energy-they also require vast quantities of water for cooling purposes in data centres, as well as other parts of the AI lifecycle, including production of AI-specific hardware or water-intensive electricity generation. In regions where water is scarce, this adds strain on local water supply and raises serious concerns. The moderator highlighted that water consumption hasn't received the same level of attention as energy consumption, that calculations are complex, without commonly accepted metrics, and are not included in regulatory compliance requirements, which may explain why it remains under-measured.
AI's hidden water footprint
One speaker explained that AI models, particularly large-scale ones, require significant amounts of water to cool servers. Data centres often use fresh water for cooling, which leads to substantial water loss due to evaporation. This not only affects the environment but also poses risks to local communities that depend on this water for drinking and agricultural use. To further exasperate the issue, AI also has an indirect water footprint. For example, semiconductors consumption uses water, so does generating electricity for AI operations.
Measuring AI's water footprint
A key challenge in addressing AI's water usage is the lack of standardised tools for measuring water consumption. One speaker explained that the use of water in data centres can be divided into embodied use and operational use. Embodied use, this is to say water used in manufacturing hardware, represents 30% of AI-related consumption. Operational use refers to water used to cool data centres and AI model processing. It equals to 70% of AI-related water consumption. However, one speaker pointed out that many companies use energy consumption as a proxy for water use, but these methods have an error margin of up to 500%, which often leaves companies unaware of their water consumption's true scale.
Geographical and ethical considerations
One speaker signalled that current global water allocation is already unsustainable: we are currently not on track to meet the Sustainable Development Goal 6 water targets, and that AI is an exacerbating factor. AI water consumption is a social justice issue. Many data centres are located in water-scarce regions, which can make local water scarcity worse and create competition for resources. All speakers emphasised that water is a local issue.
The Pricing Paradox: Water as an undervalued resource
One participant raised an important question: should water be priced differently to incentivize conservation? Speakers argued that water is often treated as a "free" resource, but this mindset fails to account for the risks associated with water scarcity. As such, there is a differentiation between the cost and the value attributed to water. By adjusting water pricing to reflect its true cost, companies and consumers could be incentivized to use it more efficiently.
Transparency and accountability
To address the issue of AI's water usage, one speaker suggested that companies should disclose their water consumption as part of their sustainability efforts. A representative from industry noted that new data centres use non-potable water sources, such as canal or seawater, to reduce their reliance on freshwater sources, while others have on-premises wastewater treatment to reduce the use of freshwater in data centres.

The path forward: Collaboration is key

Throughout the event, there was a recurring call for collaboration across all sectors. The OECD emphasised the need for multi-stakeholder cooperation, involving (local) governments, industries, academia, citizens including indigenous populations, and standard-setting bodies like IEEE, to ensure that AI development aligns with sustainability goals. This sentiment was echoed by IEEE, who stressed the importance of standardized metrics to measure and manage AI's environmental impact comprehensively and invited participants to join the IEEE P7100 standardization effort.

The discussion ended, participants recognised progress made and what is left to do. The complexity of AI's environmental, energy, and water usage challenges requires ongoing research, policy innovation, and technological advances. The good news is that as AI continues to evolve, as does our understanding of its environmental footprint. By working together, we can create a future where AI is both innovative and sustainable.

The environmental impact of AI is an urgent issue that requires collective efforts. As it continues to shape the future, we must prioritize sustainability at every stage of its lifecycle, from model development to deployment. By aligning our efforts and sharing knowledge, we can mitigate the environmental costs of AI and build a greener, more sustainable future for all.

Photo: OECD

# data[].attributes.attachment_link.title
The hidden cost of AI: Unpacking its energy and water footprint - IEEE Technology Climate Center (ITCC)

# data[].attributes.attachment_link.description
SummaryThis blog post, co-authored by Arti Garg (Chair, IEEE P7100 Working Group), Irene Kitsara (European Standardization Initiatives Director, IEEE), and Sarah Bérubé (Policy Analyst, AI Policy, OECD), is cross-published with... Continue Reading The hidden cost of AI: Unpacking its energy and water footprint

# data[].attributes.attachment_link.provider_name
MIT News | Massachusetts Institute of Technology

# data[].attributes.attachment_link.title
Explained: Generative AI’s environmental impact

# data[].attributes.attachment_link.description
MIT News explores the environmental and sustainability implications of generative AI technologies and applications.

# data[].attributes.body
[Australian tool]

# data[].attributes.attachment_link.readable_text
TikTok is one of the newest social media apps, but it's also unfortunately the one with the biggest carbon footprint, emitting 2.63 grams of carbon equivalent for each minute that you spend scrolling. That means that if you spent just five minutes a day on the app, you'd produce 13g a day and 4.8g a year.

# data[].attributes.attachment_link.description
TikTok is one of the newest social media apps, but it's also unfortunately the one with the biggest carbon footprint, emitting 2.63 grams of carbon equivalent for each minute that you spend scrolling. That means that if you spent just five minutes a day on the app, you'd produce 13g a day and 4.8g a year.

# data[].attributes.attachment_link.title
Research Guides: Artificial Intelligence: AI and the Research Process

# data[].attributes.attachment_link.description
Artifical Intelligence and the research process

# data[].attributes.body
Generative artificial intelligence (AI) describes algorithms (such as ChatGPT) that can be used to create new content, including audio, code, images, text, simulations, and videos.

# data[].attributes.body
Every AI-generated image is an infographic about the dataset. AI images are data patterns inscribed into pictures, and they tell us stories about that dataset and the human decisions behind it.

# data[].attributes.attachment_link.readable_text
Critical Topics: AI Images was an undergraduate class delivered for Bradley University in Spring 2023. It was an overview of the emerging contexts of AI art making tools that connected media studies and histories of new media art, with data ethics and critical data studies. Through this multidisciplinary lens, we examined current events and debates in AI and generative art, with students thinking critically about these tools as they learned to use them. They were encouraged to make work that reflected the context and longer history of these tools.

As a final project, students collected 500-1000 of their own images, cleaning them to create a unique, personalized dataset. Then, using RunwayML, they extended StyleGAN2's training data with their datasets to create a custom generative model. Along the way, we discussed the politics of image assembly and archives, the human labor of datasets and content moderation, and more.

The course included interviews with AI artists from a variety of perspectives. Students responded to each with short essays highlighting the diversity of thoughts and opinions about what AI art means, how it is made, and the ethics that surround it.

This website collects all of the asynchronous video lectures, alongside works referenced in the lectures. Guest lectures and artist talks are also archived, with permission.

I continue to write on these topics on my mailing list, Cybernetic Forests, which is free to read. You can sign up at the button below.

Are you using any of these materials in a class, paper, or other way?
Please let me know (I am happy for it, but documenting it helps me out!)

An introduction to the idea of AI generated images an infographics, or data visualizations. We compare AI images to John Snow's 1855 maps of the London Cholera epidemic to see how data moves into images, maps, and visualizations - and how reality is transformed as it is collected and represented.

This is a short introduction to the class.

Works Referenced:

The history of the idea of Artificial Intelligence is paired alongside trajectories of art history, starting with automated chessboards in 1914 to computer-generated drawings in 1968. This section tackles neural networks, Turing machines and cybernetics alongside the work of Jasia Reichardt, Vera Molnar, Ben Laposky, Lilllian Schwartz, Gordon Pask, John Cage, Nam June Paik, Andy Warhol and Harold Cohen.

To Read:

Works Referenced:

McCulloch, W.S., Pitts, W. A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics 5, 115-133 (1943). https://doi.org/10.1007/BF02478259

Eames, C., & Eames, R. (Directors). (1953). A Communications Primer [Film]. Eames Office, LLC.

Birhane, Abeba. (2020). Fair Warning: For as long as there has been AI research, there have been credible critiques about the risks of AI boosterism. Real Life. https://reallifemag.com/fair-warning/

Kubrick, Stanley (1968). 2001: A Space Odyssey.

Brecht, George (1965). Universal Machine II.

Reichardt, Jasia (1968). Cybernetic Serendipity (Interview). BBC Late Night Lineup, via YouTube.

Cohen, Harold (1994). The Further Exploits of AARON, Painter. Stanford Humanities Review. (PDF).

Cohen, Paul. (2017). Harold Cohen and AARON. AI Magazine, 37(4), 63-66. https://doi.org/10.1609/aimag.v37i4.2695

Cohen, Harold (1974) On Purpose: An Enquiry Into the Possible Roles of the Computer in Art. Studio International. (PDF).

A guest talk from Paul Pangaro of Carnegie Mellon University. President of the American Society for Cybernetics. Pangaro has restored one of Gordon Pask's artworks, Colloquy of Mobiles, shown at the Cybernetic Serendipities show in 1968. In this talk, Pangaro speaks from his own expertise on cybernetics as we walk through Pask's body of work connecting conversations and machines.

Read: Gordon Pask (1972) A Comment, a Case History, and a Plan
Watch: Interview with Lillian Schwartz, a pioneer of computer animation at Bell Labs. (9 Minutes).

Some of the same AI technologies and training data used to make AI art tools are also used in surveillance systems, both in the United States and abroad. In this class session, we'll explore some of the social issues around AI images. Particularly, AI image datasets of faces issues of race and gender in surveillance technologies. Adapted from my 2021 keynote talk, From Big Brother to Big Data: Nobody is Always Watching You.

To Read:

Works Referenced:

George Orwell (1949) 1984. Signet Classics.

Norbert Wiener (1948) Cybernetics: Command and Control in the Animal and the Machine. MIT Press.

Norbert Wiener (1950) The Human Use of Human Beings. MIT Press.

Tomas Smits and Melvin Wevers, Abe Lincoln Image Classification Series (Image).

For your first creative assignment, I'll ask you to create a piece of art, using an AI tool of your own choice, that reflects on a topic of interest from the previous readings or classes. I'll provide a list of tools and some examples, but you will be asked to define "AI" for yourself - and can include tools like George Brecht's box of cut-up images ("Universal Machine II") or contemporary AI tools.

We'll also hear from Anuradha Reddy on a particular way of making art with AI generated images that reflects some of the conversations we've had in our class: using AI to connect to the roots and origins of computing back to Ada Lovelace and the Jacquard loom.

In your readings, you'll encounter a number of artists working with AI today, from a variety of perspectives.

Anuradha Reddy ( website) is an interdisciplinary design researcher based in Sweden. Her research practice includes interaction design, user research, data technologies, creativity, and hacking. She has a practice-based PhD in Interaction Design from Malmö University, Sweden. By day, she works in the software industry as a design researcher. Outside her day job, she continues researching ways to bridge professional ML/AI expertise with informal communities of makers/hackers through craftivist and critical approaches to design. Her recent design work has been covered in conferences, journals, periodicals and zines. She is an open advocate of FOSS/libre design and collaborative technologies. She holds workshops, talks, and writes for magazines.

We look at Generative Adversarial Networks (GANs) and how they might reinforce - or disrupt - the logic of archives. How do the images generated by these systems tell stories about the underlying datasets? How can we, as artists, use this process to tell our own stories? We will also look at the history of GANs and their early reliance on datasets of European oil painting. How do the datasets available to us shape the work we make? How do these datasets reflect cultural contexts of museums and digitization?

We look to the work of Hannah Hoch, Lorna Simpson, Helena Sarin, Mario Klingemann, Gene Kogan, Obvious, and Robbie Barrat.

Works Referenced:

Tomas Smits and Melvin Wevers, Abe Lincoln Image Classification Series (Image).

Google, Machine Anatomy: Diagram of Generative Adversarial Network.

Tero Kerras, NVIDIA Labs (2019) FFHQ Dataset. (Link)

Jacques Derrida (1994) Archive Fever: A Freudian Impression. University of Chicago Press. p. 3, p. 17

Hannah Hoch (1971) On Collage. In The Ends of Collage, ed. Yuval Edgar (2017) Luxembourg & Dayan Press p. 143.

Lorna Simpson (2018) Collages. Chronicle Books LLC.

Obvious (2018) Edmond de Belamy. Painting.

Robbie Barrat (2018) Output from Neural Network. Twitter.

Mario Klingemann (2018) Memories of Passerby. Interactive Video work.

Gene Kogan (2019) Generative Antipodes. AI/Video work.

A deeper look at what drives Diffusion models. We'll explore the images inside the datasets used to train neural networks. Composite photography, statistical correlations and eugenics were all created by the same man - the British sociologist Francis Galton - in the 19th Century. We examine the legacy of this common root as it has reconvened in image synthesis - and the underlying principles of categorization and representation they enact. We look into the training data to see how today's image generators categorize and represent people. What lurks behind seemingly innocuous prompts such as "brave person" or "black girl?" We also offer an introduction to weights and biases in AI systems.
Works Referenced
Burbridge, Benedict (2013) "Agency and Objectification in Francis Galton's Family Composites." Photoworks, 28 Oct. 2013, https://photoworks.org.uk/agency-objectification-francis-galtons-family-composites/.

Galton, Francis (1878) "Step one in assembling a composite photograph" in Popular Science Monthly Volume 13, (August).

Hassani, B.K. Societal bias reinforcement through machine learning: a credit scoring perspective. AI Ethics 1, 239-247 (2021). https://doi.org/10.1007/s43681-020-00026-z

Jo, Eun Seo, and Timnit Gebru. "Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning." ArXiv [Cs.LG], 2019, http://arxiv.org/abs/1912.10389.

Lee-Morrison, Lila (2019). "Chapter 3: Francis Galton and the Composite Portrait". Portraits of Automated Facial Recognition: On Machinic Ways of Seeing the Face, Bielefeld: transcript Verlag, 2019, pp. 85-100. https://doi.org/10.1515/9783839448465-005

Noble, Safiya Umoja (2018). Algorithms of Oppression: How Search Engines Reinforce Racism. Ch. 3, Searching for Black Girls, p69. New York University Press.

Offert, Fabian, and Thao Phan (2022) "A Sign That Spells: DALL-E 2, Invisual Images and the Racial Politics of Feature Space." ArXiv [Cs.CY], 2022, http://arxiv.org/abs/2211.06323.

Treitler, V. (1998). Racial Categories Matter Because Racial Hierarchies Matter: A Commentary. Ethnic and Racial Studies, 21(5), 959-968

Turner, J. H. (2020) Positivism: Sociological via International Encyclopedia of Human Geography (Cambridge, Massachusetts: Elsevier) 11827. ()

Alamy Limited. "1940's Photograph Hitler Youth Medal Iron Cross." Accessed 18 Feb. 2023.

Caroline Sinders ( website) is a critical designer and artist. For the past few years, she has been examining the intersections of artificial intelligence, intersectional justice, systems design, harm, and politics in digital conversational spaces and technology platforms. She has worked with the United Nations, Amnesty International, IBM Watson, the Wikimedia Foundation, and others. Sinders has held fellowships with the Harvard Kennedy School, Google's PAIR (People and Artificial Intelligence Research group), Ars Electronica's AI Lab, the Weizenbaum Institute, the Mozilla Foundation, Pioneer Works, Eyebeam, Ars Electronica, the Yerba Buena Center for the Arts, the Sci Art Resonances program with the European Commission, and the International Center of Photography. Her work has been featured in the Tate Exchange in Tate Modern, Victoria and Albert Museum, MoMA PS1, LABoral, Wired, Slate, Quartz, the Channels Festival, and others. Sinders holds a Masters from New York University's Interactive Telecommunications Program.

Pre-Readings

Read: Caroline Sinders (2020) In Defense of Useful Art. Pioneer Works.
Read: Abigail Echo-Hawk, on "The Art and Science of Decolonizing Data."

What's in the training data of contemporary tools like Stable Diffusion and DALL-E 2? It's not always easy to find out. In this class we look at image datasets similar to those that have been integrated into GAN and Diffusion models. We'll learn how to look inside them, uncover where came from, and what the content means about the images these tools make.

This class introduces the Dataset Dissection assignment, reviewing a set of images from Kaggle image datasets and completing a "Datasheet for Datasets" evaluation.

Read: Kate Crawford & Trevor Paglen, Excavating AI. ( Online)

Read: Gebru, Timnit, et al. "Datasheets for Datasets." ArXiv [Cs.DB], 2018, doi:10.48550/ARXIV.1803.09010.

Works Referenced:

Abeba Birhane, et al. (2021). "Multimodal Datasets: Misogyny, Pornography, and Malignant Stereotypes." ArXiv [Cs.CY], doi:10.48550/ARXIV.2110.01963.

Christoph Schuhmann, et al. (2022) "LAION-5B: An Open Large-Scale Dataset for Training next Generation Image-Text Models." ArXiv [Cs.CV], doi:10.48550/ARXIV.2210.08402.

An informal technical walk-through of RunwayML and how to train GANs once you have a dataset. No theory, just pure walk-through, showing where to find the right tools, how to prep your dataset, and what you get from it when you do.
Additional Resources:

Dr Eleanor Dare is an academic and critical technologist who works with Game Engines and virtual spaces, who now works at Cambridge University, Faculty of Education, as well as UCL, institute of Education. Dr. Dare has a PhD and MSc in Arts and Computational Technologies from the department of Computing, Goldsmiths, and has exhibited work at galleries and festivals around the world. Dr. Dare was formerly Reader in Digital Media and Head of Programme for MA Digital Direction, at the Royal College of Art.

Working Day

This was an open discussion period to explore reflections on what's different about working with AI tools. We asked: What questions do you have about your results? How do you make sense of the relationship between the images in your dataset and the images the model creates? What stories can you tell with your dataset? What stories would you not tell with your dataset?

Read: Vilem Flusser (1983, English Translation 2000) Towards a Philosophy of Photography. Ch. 7, "The Reception of Photographs" & Ch. 9, "Why a Philosophy of Photography is Necessary." ( PDF)

Vladimir Alexeev (Merzmensch), born in 1979 in Moscow, living in Germany, speaking Japanese is an international writer, artist, and researcher, who explores areas of tension between Historical Avant-Garde (especially Dadaism) and AI art. In his art and essays, he investigates the horizons and possibilities of human-machine creative collaborations. Harper's Magazine describes him as Data Journalist. In his artworks, he uses mixed media, his own poetry, and photography, but also advanced AI-driven models and approaches.

The pseudonym Merzmensch is coming from MERZ-Art by Kurt Schwitters, who is the big inspiration and spiritual mentor for Vladimir Alexeev. In his works, Merzmensch is trying to follow Schwitters' path to create new realms and realities from mixed media, where every element has its significance and is equal to others.

Born in Caracas, Venezuela, Moises Sanabria is an artist interested in technology, internet culture and contemporary branding. He is one of the co-founders of the new media collective Art404 (Art Not Found), whose works often deal with legitimacy, value, and perception. Art404 often uses the online world as a medium for creating works, including projects involving Photoshop, the hacker group Anonymous, and the Sims.

Art404 has shown at Transmediale 2k+12 in Berlin, Gucci Vuitton in Miami, Ars Electronica Festival in Austria & Conflux Festival in NYC. Moises is currently attending the Cooper Union for the Advancement of Science and Art, where he is focusing on Interactive 3D graphics and physical computing. He lives and works in New York City.

Coca-Cola was one of the first corporations to make a foray into the world of generative art for a marketing campaign. We'll look at the history of media, marketing and advertisements to understand how and why they work, and how we can make informed decisions as consumers about the messages that pay to be there.

How does biased generated content normalize certain expectations in their content - and how does that connect to the dreams sold to you by ads? Are our imaginations the latest landscape for product placement? Or is that where they have been all along?

Assignment: Studio Reflections 2

Using an AI tool of your choice - including analog interpretations of "AI" - read and respond creatively to one of the following texts. The response should be visual, with a text reflecting your thinking, your use of the tool, and how you interpreted the text creatively.

Read: Aarathi Krishnan, Angie Abdilla, A Jung Moon, Carlos Affonso Souza, Chelle Adamson, Eileen M. Lach, Farah Ghazal, Jessica Fjeld, Jennyfer Taylor, John C. Havens, Malavika Jayaram, Monique Morrow, Nagla Rizk, Paola Ricaurte Quijano, R. Buse Çetin, Raja Chatila, Ravit Dotan, Sabelo Mhlambi, Sara Jordan & Sarita Rosenstock (2022) AI Decolonial Manyfesto. ( Link)

Read: Anja Kaspersen & Wendell Wallach (2023): Now is the Moment for a Systemic Reset of AI and Technology Governance. Carnegie Council. ( Link)

Read: Dan McQuillan (2023) We Come to Bury ChatGPT, Not to Praise It. ( Website).

Works Referenced
"Coca-Cola Invites Digital Artists to 'Create Real Magic' Using New AI Platform - News & Articles." The Coca-Cola Company, The Coca-Cola Organization, 20 Mar. 2023, https://www.coca-colacompany.com/news/coca-cola-invites-digital-artists-to-create-real-magic-using-new-ai-platform.

Shaw, Adrienne. "Encoding and Decoding Affordances: Stuart Hall and Interactive Media Technologies." Media, Culture, and Society, vol. 39, no. 4, 2017, pp. 592-602, doi:10.1177/0163443717692741.

Gurfinkle, Jenka. "AI and the American Smile." Medium, 26 Mar. 2023, https://medium.com/@socialcreature/ai-and-the-american-smile-76d23a0fbfaf.

Barrett, Lisa Feldman. How Emotions Are Made: The Secret Life of the Brain. Houghton Mifflin, 2017.

Charlotte Kent: "Art's Intelligence: AI and Human Systems." The Brooklyn Rail, 1 Apr. 2023, https://brooklynrail.org/2023/04/art-technology/Arts-Intelligence-AI-and-Human-Systems.

Tucker, Emily. "Artifice and Intelligence." Tech Policy Press, 17 Mar. 2022, https://techpolicy.press/artifice-and-intelligence/.

Williamson, Judith. Decoding Advertisements. Marion Boyars, 2014.

Utilizing cutting edge machine learning technology, Derrick Schultz's work explores multisensory perception, generative abstraction, and the future of ecology. In addition to creating his own work, Derrick also teaches machine learning to artists, designers, and image makers.

This course has pulled apart the threads of AI image synthesis. In this final review session, we reconnect them, systematically, through four lenses: 1) Data, 2) Interface, 3) Image, 4) Artistry. For each layer of this system, we rethink the ethical complications posed by these systems. Is AI art theft? Can images be representative? At which layer of these systems are the questions most, and least, compelling or plausible?

We conclude by revisiting the idea that machines are inspired, or see the world, or make art, in the same way people do. By now we might suspect that this isn't true - but if so, what can we make of the role of AI during this fast-paced period of change? Spoiler Alert: I don't have answers. But we'll try to find some footing for how to ask good questions.

Assigned Readings:

Recommended Readings:

# data[].attributes.attachment_link.title
Critical Topics: AI Images Course - Cybernetic Forests.

# data[].attributes.attachment_link.description
An archive of an undergraduate class on the context, history and ethical issues of AI images.

# data[].attributes.body
Artificial intelligence is a broad field, which refers to the use of technologies to build machines and computers that have the ability to mimic cognitive functions associated with human intelligence, such as being able to see, understand, and respond to spoken or written language, analyze data, make recommendations, and more.

# data[].attributes.attachment_link.title
Gemini variations on the Barge Haulers - Mirador 2

# data[].attributes.body
While artificial intelligence encompasses the idea of a machine that can mimic human intelligence, machine learning does not. Machine learning aims to teach a machine how to perform a specific task and provide accurate results by identifying patterns.

# data[].attributes.attachment_link.readable_text
Stay organized with collections Save and categorize content based on your preferences.

You might hear people use artificial intelligence (AI) and machine learning (ML) interchangeably, especially when discussing big data, predictive analytics, and other digital transformation topics. The confusion is understandable as artificial intelligence and machine learning are closely related. However, these trending technologies differ in several ways, including scope, applications, and more.

Increasingly AI and ML products have proliferated as businesses use them to process and analyze immense volumes of data, drive better decision-making, generate recommendations and insights in real time, and create accurate forecasts and predictions.

So, what exactly is the difference when it comes to ML vs. AI, how are ML and AI connected, and what do these terms mean in practice for organizations today?

We'll break down AI vs. ML and explore how these two innovative concepts are related and what makes them different from each other.

What is artificial intelligence?

Artificial intelligence is a broad field, which refers to the use of technologies to build machines and computers that have the ability to mimic cognitive functions associated with human intelligence, such as being able to see, understand, and respond to spoken or written language, analyze data, make recommendations, and more.

Although artificial intelligence is often thought of as a system in itself, it is a set of technologies implemented in a system to enable it to reason, learn, and act to solve a complex problem.

What is machine learning?

Machine learning is a subset of artificial intelligence that automatically enables a machine or system to learn and improve from experience. Instead of explicit programming, machine learning uses algorithms to analyze large amounts of data, learn from the insights, and then make informed decisions.

Machine learning algorithms improve performance over time as they are trained-exposed to more data. Machine learning models are the output, or what the program learns from running an algorithm on training data. The more data used, the better the model will get.

How are AI and ML connected?

While AI and ML are not quite the same thing, they are closely connected. The simplest way to understand how AI and ML relate to each other is:

AI is the broader concept of enabling a machine or system to sense, reason, act, or adapt like a human

ML is an application of AI that allows machines to extract knowledge from data and learn from it autonomously

One helpful way to remember the difference between machine learning and artificial intelligence is to imagine them as umbrella categories. Artificial intelligence is the overarching term that covers a wide variety of specific approaches and algorithms. Machine learning sits under that umbrella, but so do other major subfields, such as deep learning, robotics, expert systems, and natural language processing.

Differences between AI and ML

Now that you understand how they are connected, what is the main difference between AI and ML?

While artificial intelligence encompasses the idea of a machine that can mimic human intelligence, machine learning does not. Machine learning aims to teach a machine how to perform a specific task and provide accurate results by identifying patterns.

Let's say you ask your Google Nest device, "How long is my commute today?" In this case, you ask a machine a question and receive an answer about the estimated time it will take you to drive to your office. Here, the overall goal is for the device to perform a task successfully-a task that you would generally have to do yourself in a real-world environment (for example, research your commute time).

In the context of this example, the goal of using ML in the overall system is not to enable it to perform a task. For instance, you might train algorithms to analyze live transit and traffic data to forecast the volume and density of traffic flow. However, the scope is limited to identifying patterns, how accurate the prediction was, and learning from the data to maximize performance for that specific task.

Artificial intelligence

AI allows a machine to simulate human intelligence to solve problems

The goal is to develop an intelligent system that can perform complex tasks

We build systems that can solve complex tasks like a human

AI has a wide scope of applications

AI uses technologies in a system so that it mimics human decision-making

AI works with all types of data: structured, semi-structured, and unstructured

AI systems use logic and decision trees to learn, reason, and self-correct

Machine learning

ML allows a machine to learn autonomously from past data

The goal is to build machines that can learn from data to increase the accuracy of the output

We train machines with data to perform specific tasks and deliver accurate results

Machine learning has a limited scope of applications

ML uses self-learning algorithms to produce predictive models

ML can only use structured and semi-structured data

ML systems rely on statistical models to learn and can self-correct when provided with new data

Benefits of using AI and ML together

AI and ML bring powerful benefits to organizations of all shapes and sizes, with new possibilities constantly emerging. In particular, as the amount of data grows in size and complexity, automated and intelligent systems are becoming vital to helping companies automate tasks, unlock value, and generate actionable insights to achieve better outcomes.

Here are some of the business benefits of using artificial intelligence and machine learning:

Applications of AI and ML

Artificial intelligence and machine learning can be applied in many ways, allowing organizations to automate repetitive or manual processes that help drive informed decision-making.

Companies across industries are using AI and ML in various ways to transform how they work and do business. Incorporating AI and ML capabilities into their strategies and systems helps organizations rethink how they use their data and available resources, drive productivity and efficiency, enhance data-driven decision-making through predictive analytics, and improve customer and employee experiences.

Here are some of the most common applications of AI and ML:

Healthcare and life sciences

Patient health record analysis and insights, outcome forecasting and modeling, accelerated drug development, augmented diagnostics, patient monitoring, and information extraction from clinical notes.

Manufacturing

Production machine monitoring, predictive maintenance, IoT analytics, and operational efficiency.

Ecommerce and retail

I nventory and supply chain optimization, demand forecasting, visual search, personalized offers and experiences, and recommendation engines.

Financial services

Risk assessment and analysis, fraud detection, automated trading, and service processing optimization.

Telecommunications

Intelligent networks and network optimization, predictive maintenance, business process automation, upgrade planning, and capacity forecasting.

Related products and services

Google Cloud offers a wide range of AI and ML tools to help your teams focus on the valuable work that matters most. Designed and built with the best of Google's research and technology, our products and services are helping organizations transform and solve their most challenging real-world problems.

Take the next step

Start building on Google Cloud with $300 in free credits and 20+ always free products.

Take the next step

Start your next project, explore interactive tutorials, and manage your account.

You might hear people use artificial intelligence (AI) and machine learning (ML) interchangeably, especially when discussing big data, predictive analytics, and other digital transformation topics. The confusion is understandable as artificial intelligence and machine learning are closely related. However, these trending technologies differ in several ways, including scope, applications, and more.

Increasingly AI and ML products have proliferated as businesses use them to process and analyze immense volumes of data, drive better decision-making, generate recommendations and insights in real time, and create accurate forecasts and predictions.

So, what exactly is the difference when it comes to ML vs. AI, how are ML and AI connected, and what do these terms mean in practice for organizations today?

We'll break down AI vs. ML and explore how these two innovative concepts are related and what makes them different from each other.

What is artificial intelligence?

Artificial intelligence is a broad field, which refers to the use of technologies to build machines and computers that have the ability to mimic cognitive functions associated with human intelligence, such as being able to see, understand, and respond to spoken or written language, analyze data, make recommendations, and more.

Although artificial intelligence is often thought of as a system in itself, it is a set of technologies implemented in a system to enable it to reason, learn, and act to solve a complex problem.

What is machine learning?

Machine learning is a subset of artificial intelligence that automatically enables a machine or system to learn and improve from experience. Instead of explicit programming, machine learning uses algorithms to analyze large amounts of data, learn from the insights, and then make informed decisions.

Machine learning algorithms improve performance over time as they are trained-exposed to more data. Machine learning models are the output, or what the program learns from running an algorithm on training data. The more data used, the better the model will get.

How are AI and ML connected?

While AI and ML are not quite the same thing, they are closely connected. The simplest way to understand how AI and ML relate to each other is:

AI is the broader concept of enabling a machine or system to sense, reason, act, or adapt like a human

ML is an application of AI that allows machines to extract knowledge from data and learn from it autonomously

One helpful way to remember the difference between machine learning and artificial intelligence is to imagine them as umbrella categories. Artificial intelligence is the overarching term that covers a wide variety of specific approaches and algorithms. Machine learning sits under that umbrella, but so do other major subfields, such as deep learning, robotics, expert systems, and natural language processing.

Differences between AI and ML

Now that you understand how they are connected, what is the main difference between AI and ML?

While artificial intelligence encompasses the idea of a machine that can mimic human intelligence, machine learning does not. Machine learning aims to teach a machine how to perform a specific task and provide accurate results by identifying patterns.

Let's say you ask your Google Nest device, "How long is my commute today?" In this case, you ask a machine a question and receive an answer about the estimated time it will take you to drive to your office. Here, the overall goal is for the device to perform a task successfully-a task that you would generally have to do yourself in a real-world environment (for example, research your commute time).

In the context of this example, the goal of using ML in the overall system is not to enable it to perform a task. For instance, you might train algorithms to analyze live transit and traffic data to forecast the volume and density of traffic flow. However, the scope is limited to identifying patterns, how accurate the prediction was, and learning from the data to maximize performance for that specific task.

Artificial intelligence

AI allows a machine to simulate human intelligence to solve problems

The goal is to develop an intelligent system that can perform complex tasks

We build systems that can solve complex tasks like a human

AI has a wide scope of applications

AI uses technologies in a system so that it mimics human decision-making

AI works with all types of data: structured, semi-structured, and unstructured

AI systems use logic and decision trees to learn, reason, and self-correct

Machine learning

ML allows a machine to learn autonomously from past data

The goal is to build machines that can learn from data to increase the accuracy of the output

We train machines with data to perform specific tasks and deliver accurate results

Machine learning has a limited scope of applications

ML uses self-learning algorithms to produce predictive models

ML can only use structured and semi-structured data

ML systems rely on statistical models to learn and can self-correct when provided with new data

Benefits of using AI and ML together

AI and ML bring powerful benefits to organizations of all shapes and sizes, with new possibilities constantly emerging. In particular, as the amount of data grows in size and complexity, automated and intelligent systems are becoming vital to helping companies automate tasks, unlock value, and generate actionable insights to achieve better outcomes.

Here are some of the business benefits of using artificial intelligence and machine learning:

Applications of AI and ML

Artificial intelligence and machine learning can be applied in many ways, allowing organizations to automate repetitive or manual processes that help drive informed decision-making.

Companies across industries are using AI and ML in various ways to transform how they work and do business. Incorporating AI and ML capabilities into their strategies and systems helps organizations rethink how they use their data and available resources, drive productivity and efficiency, enhance data-driven decision-making through predictive analytics, and improve customer and employee experiences.

Here are some of the most common applications of AI and ML:

Healthcare and life sciences

Patient health record analysis and insights, outcome forecasting and modeling, accelerated drug development, augmented diagnostics, patient monitoring, and information extraction from clinical notes.

Manufacturing

Production machine monitoring, predictive maintenance, IoT analytics, and operational efficiency.

Ecommerce and retail

I nventory and supply chain optimization, demand forecasting, visual search, personalized offers and experiences, and recommendation engines.

Financial services

Risk assessment and analysis, fraud detection, automated trading, and service processing optimization.

Telecommunications

Intelligent networks and network optimization, predictive maintenance, business process automation, upgrade planning, and capacity forecasting.

Related products and services

Google Cloud offers a wide range of AI and ML tools to help your teams focus on the valuable work that matters most. Designed and built with the best of Google's research and technology, our products and services are helping organizations transform and solve their most challenging real-world problems.

Take the next step

Start building on Google Cloud with $300 in free credits and 20+ always free products.

Take the next step

Start your next project, explore interactive tutorials, and manage your account.

# data[].attributes.attachment_link.title
AI vs. Machine Learning: How Do They Differ? | Google Cloud

# data[].attributes.attachment_link.description
Artificial intelligence (AI) and machine learning (ML) are used interchangeably, but they differ with uses, data sets, and more.

# data[].attributes.body
Willing to get into it? Lee unpacks the way LLMs work.

# data[].attributes.attachment_link.readable_text
Hi, it's Tim Lee. I'm a journalist with a master's degree in computer science. This post is the result of two months of in-depth research. If you find it helpful, please subscribe to get future articles delivered straight to your inbox. Today's post is co-authored with Sean Trott, a cognitive scientist at the University of California, San Diego. If you are interested in the intersection of cognitive science and AI, I recommend that you subscribe to his excellent Substack.
When ChatGPT was introduced last fall, it sent shockwaves through the technology industry and the larger world. Machine learning researchers had been experimenting with large language models (LLMs) for a few years by that point, but the general public had not been paying close attention and didn't realize how powerful they had become.

Today almost everyone has heard about LLMs, and tens of millions of people have tried them out. But, still, not very many people understand how they work.

One reason for this is the unusual way these systems were developed. Conventional software is created by human programmers who give computers explicit, step-by-step instructions. In contrast, ChatGPT is built on a neural network that was trained using billions of words of ordinary language.

As a result, no one on Earth fully understands the inner workings of LLMs. Researchers are working to gain a better understanding, but this is a slow process that will take years-perhaps decades-to complete.

We'll start by explaining word vectors, the surprising way language models represent and reason about language. Then we'll dive deep into the transformer, the basic building block for systems like ChatGPT. Finally, we'll explain how these models are trained and explore why good performance requires such phenomenally large quantities of data.

To understand how language models work, you first need to understand how they represent words. Human beings represent English words with a sequence of letters, like C-A-T for cat. Language models use a long list of numbers called a word vector. For example, here's one way to represent cat as a vector:

[0.0074, 0.0030, -0.0105, 0.0742, 0.0765, -0.0011, 0.0265, 0.0106, 0.0191, 0.0038, -0.0468, -0.0212, 0.0091, 0.0030, -0.0563, -0.0396, -0.0998, -0.0796, ..., 0.0002]

Why use such a baroque notation? Here's an analogy. Washington DC is located at 38.9 degrees North and 77 degrees West. We can represent this using a vector notation:

This is useful for reasoning about spatial relationships. You can tell New York is close to Washington DC because 38.9 is close to 40.7 and 77 is close to 74. By the same token, Paris is close to London. But Paris is far from Washington DC.

Language models take a similar approach: each word vector

represents a point in an imaginary "word space," and words with more similar meanings are placed closer together. For example, the words closest to in vector space include dog, kitten, and pet. A key advantage of representing words with vectors of real numbers (as opposed to a string of letters, like "C-A-T") is that numbers enable operations that letters don't.

Words are too complex to represent in only two dimensions, so language models use vector spaces with hundreds or even thousands of dimensions. The human mind can't envision a space with that many dimensions, but computers are perfectly capable of reasoning about them and producing useful results.

Researchers have been experimenting with word vectors for decades, but the concept really took off when Google announced its word2vec project in 2013. Google analyzed millions of documents harvested from Google News to figure out which words tend to appear in similar sentences. Over time, a neural network trained to predict which words co-occur with which other words learned to place similar words (like dog and cat) close together in vector space.

Because these vectors are built from the way humans use words, they end up reflecting many of the biases that are present in human language. For example, in some word vector models, doctor minus man plus woman yields nurse. Mitigating biases like this is an area of active research.

Nevertheless, word vectors are a useful building block for language models because they encode subtle but important information about the relationships between words. If a language model learns something about a cat (for example: it sometimes goes to the vet), the same thing is likely to be true of a kitten or a dog. If a model learns something about the relationship between Paris and France (for example: they share a language) there's a good chance that the same will be true for Berlin and Germany and for Rome and Italy.

A simple word vector scheme like this doesn't capture an important fact about natural language: words often have multiple meanings.

For example, the word bank can refer to a financial institution or to the land next to a river. Or consider the following sentences:

The meanings of magazine in these sentences are related but subtly different. John picks up a physical magazine, while Susan works for an organization that publishes physical magazines.

LLMs like ChatGPT are able to represent the same word with different vectors depending on the context in which that word appears. There's a vector for bank (financial institution) and a different vector for bank (of a river). There's a vector for magazine (physical publication) and another for magazine (organization). As you might expect, LLMs use more similar vectors for polysemous meanings than for homonymous meanings.

Traditional software is designed to operate on data that's unambiguous. If you ask a computer to compute "2 + 3," there's no ambiguity about what 2,+, or 3 mean. But natural language is full of ambiguities that go beyond homonyms and polysemy:

In "the customer asked the mechanic to fix his car" does his refer to the customer or the mechanic?

In "the professor urged the student to do her homework" does her refer to the professor or the student?

In "fruit flies like a banana" is flies a verb (referring to fruit soaring across the sky) or a noun (referring to banana-loving insects)?

People resolve ambiguities like this based on context, but there are no simple or deterministic rules for doing this. Rather, it requires understanding facts about the world. You need to know that mechanics typically fix customers' cars, that students typically do their own homework, and that fruit typically doesn't fly.

Word vectors provide a flexible way for language models to represent each word's precise meaning in the context of a particular passage. Now let's look at how they do that.

GPT-3, the model behind the original version of ChatGPT

, is organized into dozens of layers. Each layer takes a sequence of vectors as inputs-one vector for each word in the input text-and adds information to help clarify the meaning of that word and better predict which word might come next.

Let's start by looking at a stylized example:

The model's input, shown at the bottom of the diagram, is the partial sentence "John wants his bank to cash the." These words, represented as word2vec-style vectors, are fed into the first transformer.

The second transformer adds two other bits of context: it clarifies that bank refers to a financial institution rather than a river bank, and that his is a pronoun that refers to John. The second transformer produces another set of hidden state vectors that reflect everything the model has learned up to that point.

The above diagram depicts a purely hypothetical LLM, so don't take the details too seriously. We'll take a look at research into real language models shortly. Real LLMs tend to have a lot more than two layers. The most powerful version of GPT-3, for example, has 96 layers.

Research suggests that the first few layers focus on understanding the syntax of the sentence and resolving ambiguities like we've shown above. Later layers (which we're not showing to keep the diagram a manageable size) work to develop a high-level understanding of the passage as a whole.

For example, as an LLM "reads through" a short story, it appears to keep track of a variety of information about the story's characters: sex and age, relationships with other characters, past and current location, personalities and goals, and so forth.

Researchers don't understand exactly how LLMs keep track of this information, but logically speaking the model must be doing it by modifying the hidden state vectors as they get passed from one layer to the next. It helps that in modern LLMs, these vectors are extremely large.

For example, the most powerful version of GPT-3 uses word vectors with 12,288 dimensions-that is, each word is represented by a list of 12,288 numbers. That's 20 times larger than Google's 2013 word2vec scheme. You can think of all those extra dimensions as a kind of "scratch space" that GPT-3 can use to write notes to itself about the context of each word. Notes made by earlier layers can be read and modified by later layers, allowing the model to gradually sharpen its understanding of the passage as a whole.

So suppose we changed our diagram above to depict a 96-layer language model interpreting a 1,000-word story. The 60th layer might include a vector for John with a parenthetical comment like "(main character, male, married to Cheryl, cousin of Donald, from Minnesota, currently in Boise, trying to find his missing wallet)." Again, all of these facts (and probably a lot more) would somehow be encoded as a list of 12,288 numbers corresponding to the word John. Or perhaps some of this information might be encoded in the 12,288-dimensional vectors for Cheryl, Donald, Boise, wallet, or other words in the story.

The goal is for the 96th and final layer of the network to output a hidden state for the final word that includes all of the information necessary to predict the next word.

Now let's talk about what happens inside each transformer. The transformer has a two-step process for updating the hidden state for each word of the input passage:

1. In the attention step, words "look around" for other words that have relevant context and share information with one another.

2. In the feed-forward step, each word "thinks about" information gathered in previous attention steps and tries to predict the next word.

For example, in the previous section we showed a hypothetical transformer figuring out that in the partial sentence "John wants his bank to cash the," his refers to John. Here's what that might look like under the hood. The query vector for his might effectively say "I'm seeking: a noun describing a male person." The key vector for John might effectively say "I am: a noun describing a male person." The network would detect that these two vectors match and move information about the vector for John into the vector for his.

Each attention layer has several "attention heads," which means that this information-swapping process happens several times (in parallel) at each layer. Each attention head focuses on a different task:

One attention head might match pronouns with nouns, as we discussed above.

Another attention head might work on resolving the meaning of homonyms like bank.

A third attention head might link together two-word phrases like "Joe Biden."

And so forth.

Attention heads frequently operate in sequence, with the results of an attention operation in one layer becoming an input for an attention head in a subsequent layer. Indeed, each of the tasks we just listed above could easily require several attention heads rather than just one.

The largest version of GPT-3 has 96 layers with 96 attention heads each, so GPT-3 performs 9,216 attention operations each time it predicts a new word.

GPT-2 predicted that the next word was Mary. The researchers found that three types of attention heads contributed to this prediction:

Three heads they called Name Mover Heads copied information from the Mary vector to the final input vector (for the word to). GPT-2 uses the information in this rightmost vector to predict the next word.

How did the network decide Mary was the right word to copy? Working backwards through GPT-2's computational process, the scientists found a group of four attention heads they called Subject Inhibition Heads that marked the second John vector in a way that blocked the Name Mover Heads from copying the name John.

How did the Subject Inhibition Heads know John shouldn't be copied? Working further backwards, the team found two attention heads they called Duplicate Token Heads. They marked the second John vector as a duplicate of the first John vector, which helped the Subject Inhibition Heads to decide that John shouldn't be copied.

In short, these nine attention heads enabled GPT-2 to figure out that "John gave a drink to John" doesn't make sense and choose "John gave a drink to Mary" instead.

We love this example because it illustrates just how difficult it will be to fully understand LLMs. The five-member Redwood team published a 25-page paper explaining how they identified and validated these attention heads. Yet even after they did all that work, we are still far from having a comprehensive explanation for why GPT-2 decided to predict Mary as the next word.

For example, how did the model know the next word should be someone's name and not some other kind of word? It's easy to think of similar sentences where Mary wouldn't be a good next-word prediction. For example, in the sentence "when Mary and John went to the restaurant, John gave his keys to," the logical next words would be "the valet."

Presumably, with enough research computer scientists could uncover and explain additional steps in GPT-2's reasoning process. Eventually, they might be able to develop a comprehensive understanding of how GPT-2 decided that Mary is the most likely next word for this sentence. But it could take months or even years of additional effort just to understand the prediction of a single word.

The language models underlying ChatGPT-GPT-3.5 and GPT-4-are significantly larger and more complex than GPT-2. They are capable of more complex reasoning than the simple sentence-completion task the Redwood team studied. So fully explaining how these systems work is going to be a huge project that humanity is unlikely to complete any time soon.

After the attention heads transfer information between word vectors, there's a feed-forward network

that "thinks about" each word vector and tries to predict the next word. No information is exchanged between words at this stage: the feed-forward layer analyzes each word in isolation. However, the feed-forward layer does have access to any information that was previously copied by an attention head. Here's the structure of the feed-forward layer in the largest version of GPT-3:

The green and purple circles are neurons: mathematical functions that compute a weighted sum of their inputs.

What makes the feed-forward layer powerful is its huge number of connections. We've drawn this network with three neurons in the output layer and six neurons in the hidden layer, but the feed-forward layers of GPT-3 are much larger: 12,288 neurons in the output layer (corresponding to the model's 12,288-dimensional word vectors) and 49,152 neurons in the hidden layer.

So in the largest version of GPT-3, there are 49,152 neurons in the hidden layer with 12,288 inputs (and hence 12,288 weight parameters) for each neuron. And there are 12,288 output neurons with 49,152 input values (and hence 49,152 weight parameters) for each neuron. This means that each feed-forward layer has 49,152 * 12,288 + 12,288 * 49,152 = 1.2 billion weight parameters. And there are 96 feed-forward layers, for a total of 1.2 billion * 96 = 116 billion parameters! This accounts for almost two-thirds of GPT-3's overall total of 175 billion parameters.

A neuron in layer 1 matched sequences of words ending with "substitutes."

A neuron in layer 6 matched sequences related to the military and ending with "base" or "bases."

A neuron in layer 13 matched sequences ending with a time range such as "between 3 pm and 7" or "from 7:00 pm Friday until."

A neuron in layer 16 matched sequences related to television shows such as "the original NBC daytime version, archived" or "time shifting viewing added 57 percent to the episode's."

As you can see, patterns got more abstract in the later layers. The early layers tended to match specific words, whereas later layers matched phrases that fell into broader semantic categories such as television shows or time intervals.

This is interesting because, as mentioned previously, the feed-forward layer examines only one word at a time. So when it classifies the sequence "the original NBC daytime version, archived" as related to television, it only has access to the vector for archived, not words like NBC or daytime. Presumably, the feed-forward layer can tell that archived is part of a television-related sequence because attention heads previously moved contextual information into the archived vector.

When a neuron matches one of these patterns, it adds information to the word vector. While this information isn't always easy to interpret, in many cases you can think of it as a tentative prediction about the next word.

Recent research from Brown University revealed an elegant example of how feed-forward layers help to predict the next word. Earlier we discussed Google's word2vec research showing it was possible to use vector arithmetic to reason by analogy. For example, Berlin - Germany + France = Paris.

The Brown researchers found that feed-forward layers sometimes use this exact method to predict the next word. For example, they examined how GPT-2 responded to the following prompt: "Q: What is the capital of France? A: Paris Q: What is the capital of Poland? A:"

Feed-forward layers in the same model used vector arithmetic to transform lower-case words into upper-case words and present-tense words into their past-tense equivalents.

This division of labor holds more generally: attention heads retrieve information from earlier words in a prompt, whereas feed-forward layers enable language models to "remember" information that's not in the prompt.

Indeed, one way to think about the feed-forward layers is as a database of information the model has learned from its training data. The earlier feed-forward layers are more likely to encode simple facts related to specific words, such as "Trump often comes after Donald." Later layers encode more complex relationships like "add this vector to convert a country to its capital."

Many early machine learning algorithms required training examples to be hand-labeled by human beings. For example, training data might have been photos of dogs or cats with a human-supplied label ("dog" or "cat") for each photo. The need for humans to label data made it difficult and expensive to create large enough data sets to train powerful models.

A key innovation of LLMs is that they don't need explicitly labeled data. Instead, they learn by trying to predict the next word in ordinary passages of text. Almost any written material-from Wikipedia pages to news articles to computer code-is suitable for training these models.

For example, an LLM might be given the input "I like my coffee with cream and" and be supposed to predict "sugar" as the next word. A newly-initialized language model will be really bad at this because each of its weight parameters-175 billion of them in the most powerful version of GPT-3-will start off as an essentially random number.

Here's an analogy to illustrate how this works. Suppose you're going to take a shower, and you want the temperature to be just right: not too hot, and not too cold. You've never used this faucet before, so you point the knob to a random direction and feel the temperature of the water. If it's too hot, you turn it one way; if it's too cold, you turn it the other way. The closer you get to the right temperature, the smaller the adjustments you make.

Now let's make a couple of changes to the analogy. First, imagine that there are 50,257 faucets instead of just one. Each faucet corresponds to a different word like the, cat, or bank. Your goal is to have water only come out of the faucet corresponding to the next word in a sequence.

This gets complicated because the same pipe often feeds into multiple faucets. So it takes careful thought to figure out which valves to tighten and which ones to loosen, and by how much.

Obviously, this example quickly gets silly if you take it too literally. It wouldn't be realistic or useful to build a network of pipes with 175 billion valves. But thanks to Moore's Law, computers can and do operate at this kind of scale.

All the parts of LLMs we've discussed in this article so far-the neurons in the feed-forward layers and the attention heads that move contextual information between words-are implemented as a chain of simple mathematical functions (mostly matrix multiplications) whose behavior is determined by adjustable weight parameters. Just as the squirrels in my story loosen and tighten the valves to control the flow of water, so the training algorithm increases or decreases the language model's weight parameters to control how information flows through the neural network.

The training process happens in two steps. First there's a "forward pass," where the water is turned on and you check if it comes out the right faucet. Then the water is turned off and there's a "backwards pass" where the squirrels race along each pipe tightening and loosening valves. In digital neural networks, the role of the squirrels is played by an algorithm called backpropagation, which "walks backwards" through the network, using calculus to estimate how much to change each weight parameter.

Completing this process-doing a forward pass with one example and then a backwards pass to improve the network's performance on that example-requires hundreds of billions of mathematical operations. And training a model as big as GPT-3 requires repeating the process billions of times-once for each word of training data.

OpenAI estimates that it took more than 300 billion trillion floating point calculations to train GPT-3-that's months of work for dozens of high-end computer chips.

You might find it surprising that the training process works as well as it does. ChatGPT can perform all sorts of complex tasks-composing essays, drawing analogies, and even writing computer code. So how does such a simple learning mechanism produce such a powerful model?

One reason is scale. It's hard to overstate the sheer number of examples that a model like GPT-3 sees. GPT-3 was trained on a corpus of approximately 500 billion words. For comparison a typical human child encounters roughly 100 million words by age 10.

The larger their models got, the better they were at tasks involving language. But this was only true if they increased the amount of training data by a similar factor. And to train larger models on more data, you need a lot more computing power.

OpenAI's first LLM, GPT-1, was released in 2018. It used 768-dimensional word vectors and had 12 layers for a total of 117 million parameters. A few months later, OpenAI released GPT-2. Its largest version had 1,600-dimensional word vectors, 48 layers, and a total of 1.5 billion parameters.

In 2020, OpenAI released GPT-3, which featured 12,288-dimensional word vectors and 96 layers for a total of 175 billion parameters.

Finally, this year OpenAI released GPT-4. The company has not published any architectural details, but GPT-4 is widely believed to be significantly larger than GPT-3.

Each model not only learned more facts than its smaller predecessors, it also performed better on tasks requiring some form of abstract reasoning:

For example, consider the following story:

Here is a bag filled with popcorn. There is no chocolate in the bag. Yet, the label on the bag says "chocolate" and not "popcorn." Sam finds the bag. She had never seen the bag before. She cannot see what is inside the bag. She reads the label.

GPT-1 and GPT-2 flunked this test. But the first version of GPT-3, released in 2020, got it right almost 40 percent of the time-a level of performance Kosinski compares to a three-year-old. The latest version of GPT-3, released last November, improved this to around 90 percent-on par with a seven-year-old. GPT-4 answered about 95 percent of theory-of-mind questions correctly.

"Given that there is neither an indication that ToM-like ability was deliberately engineered into these models, nor research demonstrating that scientists know how to achieve that, ToM-like ability likely emerged spontaneously and autonomously, as a byproduct of models' increasing language ability," Kosinski wrote.

It's worth noting that researchers don't all agree that these results indicate evidence of Theory of Mind: for example, small changes to the false-belief task led to much worse performance by GPT-3; and GPT-3 exhibits more variable performance across other tasks measuring theory of mind. As one of us (Sean) has written, it could be that successful performance is attributable to confounds in the task-a kind of "clever Hans" effect, only in language models rather than horses.

Nonetheless, the near-human performance of GPT-3 on several tasks designed to measure theory of mind would have been unthinkable just a few years ago-and is consistent with the idea that bigger models are generally better at tasks requiring high-level reasoning.

For example, one researcher asked GPT-4 to draw a unicorn using an obscure graphics programming language called TiKZ. GPT-4 responded with a few lines of code that the researcher then fed into the TiKZ software. The resulting images were crude, but they showed clear signs that GPT-4 had some understanding of what unicorns look like.

The researchers thought GPT-4 might have somehow memorized code for drawing a unicorn from its training data, so they gave it a follow-up challenge: they altered the unicorn code to remove the horn and move some of the other body parts. Then they asked GPT-4 to put the horn back on. GPT-4 responded by putting the horn in the right spot:

GPT-4 was able to do this even though the training data for the version tested by the authors was entirely text-based. That is, there were no images in its training set. But GPT-4 apparently learned to reason about the shape of a unicorn's body after training on a huge amount of written text.

Another possible reason that training with next-token prediction works so well is that language itself is predictable. Regularities in language are often (though not always) connected to regularities in the physical world. So when a language model learns about relationships among words, it's often implicitly learning about relationships in the world too.

Further, prediction may be foundational to biological intelligence as well as artificial intelligence. In the view of philosophers like Andy Clark, the human brain can be thought of as a "prediction machine", whose primary job is to make predictions about our environment that can then be used to navigate that environment successfully. Intuitively, making good predictions benefits from good representations -you're more likely to navigate successfully with an accurate map than an inaccurate one. The world is big and complex, and making predictions helps organisms efficiently orient and adapt to that complexity.

Traditionally, a major challenge for building language models was figuring out the most useful way of representing different words-especially because the meanings of many words depend heavily on context. The next-word prediction approach allows researchers to sidestep this thorny theoretical puzzle by turning it into an empirical problem. It turns out that if we provide enough data and computing power, language models end up learning a lot about how human language works simply by figuring out how to best predict the next word. The downside is that we wind up with systems whose inner workings we don't fully understand.
Tim Lee was on staff at Ars from 2017 to 2021. He recently launched a new newsletter, Understanding AI. It explores how AI works and how it's changing our world. You can subscribe to his newsletter here. Sean Trott is an Assistant Professor at University of California, San Diego, where he conducts research on language understanding in humans and large language models. He writes about these topics, and others, in his newsletter The Counterfactual.

# data[].attributes.attachment_link.title
Large language models, explained with a minimum of math and jargon

# data[].attributes.attachment_link.description
Want to really understand how large language models work? Here's a gentle primer.

# data[].attributes.headline
Gemini variations on Repin's Barge Haulers

# data[].attributes.subject
Gemini variations on Repin's Barge Haulers

# data[].attributes.body
https://imperiia.omeka.fas.harvard.edu/collections/mirador2/127

# data[].attributes.attachment_link.title
Barge Haulers on the Volga - Ilya Repin - Google Arts & Culture

# data[].attributes.attachment_link.description
In 1868, when sailing on the River Neva in St Petersburg, Ilya Repin caught his first glimpse of men hauling barges. He was stunned at the sight of these u…

# data[].attributes.body
Here is the AI-powered resource we built to supplement a podcast episode.

# data[].attributes.attachment_link.provider_name
The Imperiia Project: building the spatial history of the Russian Empire

# data[].attributes.attachment_link.readable_text
This site requires Javascript to be turned on. Please enable Javascript and reload the page.

Do you believe what you see? Do you believe what you hear?
How can you even begin to make sense of history if you can't keep track of the cast of characters?

If you have landed on this interactive guide you already know that the events described in episode 6 of the MapMaker Podcast feature a huge cast. Although a few "big" names put in an appearance, most of the men involved were low-ranking, ordinary people without the means to commission a portrait let alone maintain a family archive.

Historians are not in the business of inventing the past. Relying on written evidence places constraints on what we can know and what we can say about how and why things happened. Even when we have piles of documents at our disposal we never have the full picture. Not really.

Would we have a better sense of how gossip spread if we could assess the width of the streets, the height of the buildings, and the way the wind blew in from the water? Would we understand this piece of history better if we could watch Admiral Mordvinov's facial expressions as he collected testimonies? What if we could see which men took meticulous care of their uniforms and which had crumbs in their beards?

What if there were a way to conjure those lost faces?

In a way, there is.

We threw off the historian's restraints and experimented with using DALL·E 3 to produce portraits of our cast of characters. Our AI-generated gallery embraces the gaps in the historical record and provides a way to relate to the individuals who inhabit this story. We sorted them for dramatic effect. All you need to do is grab a bag of popcorn, get comfortable, and rest your cursor on the image titles to learn more about the role each character played in the watermelon rebellion. (We added lots of juicy details.)

Best of all, each image - including the image at the top of this page - has a hidden note. Look closely and rest your cursor on the "jackpot" area to learn more about what AI gets right - but more often wrong - about the past. *If the pop-up text does not display in full, click on the image to expand. Close the image to return to this page.
Misinformation Spreaders

The Accused

How are they all connected?
You might have noticed that some characters occur in multiple categories. This visualization tool gives you another way to explore the overlaps and connections among the players in our story. Click a red dot (the green dots aren't relevant) to expand its content and click a second time to fold it back together. Drag the dots around as much as you like. We recommend using the full screen view button at lower right.

Where in the world does this story take place? (The short answer: Ukraine)
The Imperiia Project : building the spatial history of the Russian Empire

# data[].attributes.attachment_link.title
The Imperiia Project: Who's Who in the Infamous Watermelon Rebellion

# data[].attributes.attachment_link.description
It's the interactive guide you've been waiting for!

# data[].attributes.body
An experiment from Northwestern's Kellogg School of Management

# data[].attributes.attachment_link.description
Can you spot the DeepFake? Detect Fakes challenges you to discern AI-manipulated videos from real videos. Can you do better than an algorithm?

# data[].attributes.headline
A great read on how AI "reads" human emotion (from a family photograph)

# data[].attributes.subject
A great read on how AI "reads" human emotion (from a family photograph)

# data[].attributes.attachment_link.title
Artificial Intelligence, Totalitarianism, and the Future of Cognitive Liberty

# data[].attributes.attachment_link.description
If AI can read us like an open book based on our faces, our body language, and our microexpressions, what can it do to the trajectory of our lives?

# data[].attributes.attachment_link.title
chatgpt analysis of barge haulers_initial

# data[].attributes.attachment_link.title
chatgpt analysis of barge haulers_individual figures