# data[].attributes.attachment_link.readable_text
The Technology and Social Change Project was active from 2019-2023. The Media Manipulation Casebook was a major publication of the TaSC project, and will remain fully accessible at this URL going forward, but will no longer be actively updated.

# data[].attributes.attachment_link.description
A digital research platform linking together theory, methods, and practice for mapping media manipulation and disinformation cam

# data[].attributes.attachment_link.title
Play GO VIRAL! | Stop Covid-19 misinformation spreading

# data[].attributes.attachment_link.description
New online infodemic game shows how to spot coronavirus misinformation in 5 minutes. Based on Cambridge University pre-bunking research. Play Go Viral!

# data[].attributes.attachment_link.readable_text
As they start their careers, doctors swear to uphold the Hippocratic Oath. If people tackling misinformation were to establish an equivalent oath, we should make sure to borrow one of the original's phrases: "Prevention is preferable to cure."

As with medicine, so with misinformation: It is better to prevent misinformation from spreading at all than to try to debunk it once it's spread.

Here's why. Debunks don't reach as many people as misinformation, and they don't spread nearly as quickly. If they do reach us, they generally struggle to erase the misinformation from our debates or our brains. Even when we've been told that the misinformation is false, research suggests it continues to influence our thinking.

So it helps to take a page from medicine: Prevention, not cure, may be a more effective way to combat misinformation.

Understanding how prebunks work (and how they don't) is essential for reporters, fact checkers, policy makers and platforms.

The basics

The idea of inoculating people against false or misleading information is simple. If you show people examples of misinformation, they will be better equipped to spot it and question it. Much like vaccines train your immune response against a virus, knowing more about misinformation can help you dismiss it when you see it.

Inoculating people can be achieved through a number of tools or approaches, such as First Draft's text message course that texts you one lesson per day, games such as " Go Viral!" that put you in the shoes of a bad actor, or simply the ability to track down and share an article that prebunks key claims and techniques.

And research shows that these methods work. For example, five minutes of playing the game " Go Viral!" can reduce susceptibility to misinformation for up to three months.

The most accessible inoculation technique is prebunking - the process of debunking lies, tactics or sources before they strike. They can be quick and cheap for reporters, fact checkers, governments and others to make.

A good prebunk addresses people's concerns, speaks to their lived experience and compels them to share that knowledge. Prebunks are empowering: The whole point is about building trust with your audience instead of simply correcting facts.

There are three main types of prebunks:

1. fact-based: correcting a specific false claim or narrative

2. logic-based: explaining tactics used to manipulate

3. source-based: pointing out bad sources of information

Research has shown that the logic-based approach has far-reaching benefits. If you teach people to recognize tactics, they can spot them more often than individual claims.

John Cook, a research fellow at the Climate Change Communication Research Hub at Monash University in Melbourne, Australia, has been exploring inoculation for years. He told First Draft that the ideal prebunk will combine "fact and logic so people can understand the facts but also be able to spot attempts to distort the facts."

What to prebunk

1. Figure out what information people need

Anticipate your audience's questions. Don't assume that your questions are the same as your audience's. Use tools such as Google Trends to figure out trending questions or issues, check in with community figures and think about creating a space where people can submit their questions.

What is it that people find confusing? What pre-existing narratives might bad actors exploit? Are there any events coming up, including elections or health campaigns, that people might need more information about? How can you help people identify these tactics and narratives so that they are less likely to fall for them?

2. Choose your example carefully

Some rumors crop up again and again, and will reliably resurface. Focus on them.

Focus on a claim when there are likely to be key vulnerabilities, such as right before an election. A false claim might be that the date of an election has been changed.

Focus on tactics if you want to build more generalized resilience to disinformation. A tactic might be the use of fake "experts" to bolster a lie, or the use of emotional language to manipulate you.

Cook has found this while researching inoculation against climate change misinformation. He showed participants examples of tactics used in tobacco misinformation. Afterward, they were all able to spot climate change misinformation regardless of their political beliefs.

"This tells us that regardless of where you sit on the political spectrum, nobody likes being misled," says Cook.

How to design it

3. Wrap it in truth

Prioritize the truth. That means either leading with the facts or a very clear warning of how information is being manipulated. The "truth sandwich" - a factual wrapper - works for both prebunks and debunks.

4. Warn your audience

Before you repeat the myth, warn your audience. Remind your audience that people are trying to manipulate us and why. This puts people's guard up, increasing their mental resistance to misinformation. This could simply be a warning that some people will try to mislead others about the scientific consensus on climate change for political reasons, or a reminder that some individuals profit from selling fake cures.

"It is important to provide cues or warnings before you mention the myth, because you want to put people cognitively on guard when they are reading the myth," explains Cook.

5. Add some detail

You don't want to overwhelm your audience, but try to pack in a few reasons why something is false. This helps to increase belief, and arms people with counterarguments they can use to debunk the claim when they encounter it.

6. Draw out the tactic at play

If you're correcting a false claim, remind your audience that this tactic is not exclusive to this example. Use false claims as teaching examples to help your audience recognize the tactics or strategies being used for manipulation. Sometimes an example from an issue that is less political with your audience can be just as effective as a teaching tool.

7. Explain how you know what you know (and what we don't know yet)

Explaining how you know what you know helps to build trust. This helps people resolve the conflict between fact and myth and gives your audience the tools to reject the claim in the future. Explaining what we don't know yet helps warn your audience that the facts might change as things develop.

One way to do this is to focus on consensus: Remind people of what experts agree on and why. This is common when talking about climate change, where there is almost unanimous agreement among scientists, and can help to target the disinformation tactic of false debate.

How and where to share

8. Keep it simple

Break it down into the simplest version of the idea so it's easier to remember and spot afterward. Infographics, especially those that build on ideas that are already widely shared, are a great way to prebunk. Five debunks that each contain an idea are better than one debunk that contains five ideas but is confusing. Don't get lost in the weeds.

9. Make them shareable

Whether it's online or offline, in the end we're talking about people. People spread misinformation because they're searching for answers, and they share what they find. If you want your prebunk to go far, design it so it's shareable. Think about the strategies that bad actors use to spread disinformation: The idea is to use those same time-proven techniques, but for good. Design your prebunks with mobile phones in mind. For many people, mobile phones are their sole access to the internet and much of their lives happen on closed messaging apps, including WhatsApp or Facebook Messenger, which don't use up as much data.

10. Find where your audience is and publish there

Successful prebunks will join and be integrated in online spaces and platforms where your audience is already spending time. Think of it like a party: If you're in a room alone providing good information while everybody else is outside in the garden having a drink, your prebunk won't go far. Use social listening and monitoring tools to figure out the digital spaces where "the party" is happening and join in.

As with parties, you've got to read the room before you jump into the conversation. Think about the culture of the specific online space or platform you have identified. What does engaging content look like? What are the trends or styles that people use to communicate? How can you use those in a way that effectively communicates the information you want people to have?

Want to dive deeper?

The theory and evidence around inoculation have been pioneered by researchers such as Cook, Dr. Sander van der Linden and many others. New research is being published all the time, and it is worth keeping track of if you're implementing an inoculation-based campaign, game or project.

Here are some readings to get you started:

If all this sounds a bit boring, try out one of these games so you can inoculate yourself:

# data[].attributes.attachment_link.title
A guide to prebunking: a promising way to inoculate against misinformation

# data[].attributes.attachment_link.description
As they start their careers, doctors swear to uphold the Hippocratic Oath. If people tackling misinformation were to establish an equivalent oath, we should make sure to borrow one of the original's phrases: "Prevention is preferable to cure."

# data[].attributes.attachment_link.title
Timing matters when correcting fake news

# data[].attributes.attachment_link.description
Countering misinformation can reduce belief in the moment, but corrective messages quickly fade from memory. We tested whether the longer-term impact of fact-checks depends on when people receive them. In two experiments (total N = 2,683), participants read true and false headlines taken from social…

# data[].attributes.headline
How to Find Better Information Online: Click Restraint

# data[].attributes.subject
How to Find Better Information Online: Click Restraint

# data[].attributes.body
How to Find Better Information Online: Click Restraint

# data[].attributes.attachment_link.title
How to Find Better Information Online: Click Restraint

# data[].attributes.attachment_link.title
Technology and the Global Struggle for Democracy | by Manuel Muñiz - Project Syndicate

# data[].attributes.attachment_link.description
Manuel Muñiz shows why the January 6, 2021, assault on the US Capitol should not be viewed as a uniquely American event.

# data[].attributes.headline
Cómo enfocar la mente. Cómo afectan las redes sociales la atención.

# data[].attributes.subject
Cómo enfocar la mente. Cómo afectan las redes sociales la atención.

# data[].attributes.attachment_link.title
Watch The Mind, Explained | Netflix Official Site

# data[].attributes.attachment_link.description
Ever wonder what's happening inside your head? From dreaming to anxiety disorders, discover how your brain works with this illuminating series. Watch trailers & learn more.

# data[].attributes.headline
Importancia de la claridad y la rapidez en el alfabetismo mediático

# data[].attributes.subject
Importancia de la claridad y la rapidez en el alfabetismo mediático

# data[].attributes.attachment_link.readable_text
It is a profoundly erroneous truism, repeated by all copy-books and by eminent people when they are making speeches, that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case. Civilization advances by extending the number of important operations which we can perform without thinking about them.Operations of thought are like cavalry charges in a battle - they are strictly limited in number, they require fresh horses, and must only be made at decisive moments. - Alfred North Whitehead, An Introduction to Mathematics (1911)

My younger daughter is learning to drive right now, and it's a nerve-wracking process. You realize pretty quickly that what separates you from next week's car wreck is not really anyone's amazing driving skills. It's habits. It's scanning the crosswalk before making the turn. It's the mirror and head check before changing lanes.

As Whitehead noted, habits are underrated because when they work they disappear. They reduce cognitive overload to manageable levels, and allow us to focus on more complex tasks. Over time, we identify those complex tasks as the activity we're engaged in: most people who think they are good at driving cite their fluency or precision, not their automatic and unthinking use of turn signals. But take away the habits, the overload returns, and bad decisions ensue.

The debate around whether media literacy can work to reduce the impact of misinformation has often focused on whether teaching citizens to think more deeply about media can help, or if people are pursuing truth-neutral strategies for reasons having little to do with knowledge. At the Digital Polarization Initiative we're interested in something more concrete: how do we reduce the cognitive overload associated with truth-seeking behavior? How can we develop methods that requiring less thinking when initially encountering information? And will students - once that overload is reduced - fall back less on simplifying frameworks of confirmation bias, tribalism, cynicism, and conspiratorial thinking?

We've worked relentlessly to streamline these techniques over the past two years. What follows are some of the techniques we teach our students as they learn to navigate the web.

Here's an example: recent news revealed that Russian-connected entities were trying to spoof conservative sites for possible spear-phishing campaigns. So how do we know if the Hudson Institute site we are on is really the real site? Here's our two second scan of the crosswalk:
The steps:

Go up to the "omnibar"

Strip off everything after the domain name, type wikipedia and press enter

This generates a Google search for that URL with the Wikipedia page at the top

Click that link, then check in the sidebar that the URL matches.

Forty-nine out of fifty times it will. The fiftieth time you may have some work to do.

In this case, the URL does match. What does this look like if the site is fake? Here's an example. A while back a site at bloomberg.ma impersonated the Bloomberg News site. If you arrived at the site via a link and found an important story, you'd want to check the site before you shared it. Here's what that would look like:

In this case you scroll down and click the link. Checking the associated URL in Wikipedia you find it is different. It's not the real site. If you're lazy (which I am) you might click that Wikipedia link to get to the real site.

Let's stick with the Wikipedia technique for a moment, because it's useful for a few other questions. As an example, let's take one that got past both a Washington Post reporter and the WaPo fact-checkers a month or so ago. Question: Is this article really by the lead singer of Green Day?

Let's check:

Again, same process. Now, does this mean that you are 100% sure that it's not Billie Joe that wrote that article? No - there's a slight slight chance that maybe somehow the lead singer of Green Day wrote a -

Nah, you know what? It's not him. Or if it is, the chances are so infinitesimal it's not worth spending any more time on it. Find another source.

How about this site, and its searing commentary on Antifa and journalists?

Maybe you agree with this article. I don't, but maybe you do. And that's okay. But do you want to share from this particular site to your friends and family and co-workers? Let's take a look!

You can dig into this if you want, and look through the numerous links in that Wikipedia page that support this description. Maybe have a little mini-forum in your head about the differences between white nationalism and white supremacy.

Or maybe - here's a thought - find a similar article from some other site that hasn't been called a white supremacist organization by half a dozen mainstream groups. Because no matter what you think of the article, funneling friends and family to a site that has published such sentences as "When blacks are left entirely to their own devices, Western civilization - any kind of civilization - disappears" is not ethical - or likely to put you in the best light.

Here's some breaking news.

More people than you would think believe that the blue checkmark = trustworthy. But all the blue checkmark really does is say that the person is who they say they are, that they are the person of that name and not an imposter.

Your two-second "mirror and head-check" here is going to be to always, always hover, and see what they are verified for. In this case the verification means something: this person works for CNBC.com, a legitimate news site, and she covers a relevant beat here (the White House):

But maybe you don't know CNBC, or maybe you see this news from someone not verified, or verified but not as a reporter. How will you know whether to share this? Because you know you're DYING to share it and you can't wait much longer

Use our "check for other coverage" technique:

When a story is truly breaking, this is what it looks like. Our technique here is simple.

Select some relevant text.

Right-click or Cmd-click to search Google

When you get to Google don't stop, click the "News" tab to get a more curated feed

Read and scan. Investigate more as necessary.

Scan the stories. If you want to be hypervigilant, scan for sources you recognize, and consider sharing one of the stories featuring original reporting instead clickbait rehash or unsourced rumor.

Do these techniques fail sometimes? Of course they do! Do they sort all issues into neat true and false columns? Thankfully, no.

But as with Whitehead's operations, they conserve student effort for the larger problems. They break the relentless cycle of reaction by getting you to read less and learn more before pushing you to make judgments. Given that belief in false news may be more about cognitive laziness than tribalism, reducing the amount of effort required to make such judgments has to be central to any digital literacy effort.

# data[].attributes.attachment_link.title
For online media literacy that works, speed and ease matters

# data[].attributes.attachment_link.description
It is a profoundly erroneous truism, repeated by all copy-books and by eminent people when they are making speeches, that we should cultivate the habit of thinking of what we are doing. The precise opposite is the case.

# data[].attributes.headline
Modelo de creencias en salud (HBM) + Teoría de la motivación a la protección (PMT)

# data[].attributes.subject
Modelo de creencias en salud (HBM) + Teoría de la motivación a la protección (PMT)

# data[].attributes.body
La teoría de motivación a la protección fue planteada en 1975 por R. W. Rogers, y reformulada en 1987 por Rippetoe y Rogers. La teoría propone la variable motivación a la protección para explicar las conductas de salud.

De esta forma, la motivación es la que dirige el proceso de afrontamiento conductual y la que finalmente desencadena la conducta (Umeh, 2004; Milne et al., 2002).

# data[].attributes.wish_content.attachment_props.text
'Son ilusiones', nuestro spot final de campaña.

El 4 de mayo, #QueHableLaMayoría

# data[].attributes.wish_content.attachment_props.original_text
'Son ilusiones', nuestro spot final de campaña.

El 4 de mayo, #QueHableLaMayoría https://t.co/oebs7ZZzWO

# data[].attributes.attachment_link.embed_code
'Son ilusiones', nuestro spot final de campaña.

El 4 de mayo, #QueHableLaMayoría pic.twitter.com/oebs7ZZzWO
— PODEMOS (@PODEMOS) May 2, 2021

# data[].attributes.attachment_link.metadata.attachment_props_source.data.text
'Son ilusiones', nuestro spot final de campaña.

El 4 de mayo, #QueHableLaMayoría https://t.co/oebs7ZZzWO

# data[].attributes.attachment_link.metadata.attachment_props.original_text
'Son ilusiones', nuestro spot final de campaña.

El 4 de mayo, #QueHableLaMayoría https://t.co/oebs7ZZzWO

# data[].attributes.attachment_link.metadata.attachment_props.text
'Son ilusiones', nuestro spot final de campaña.

El 4 de mayo, #QueHableLaMayoría

# data[].attributes.headline
Sort Fact from Fiction Online with Lateral Reading.

# data[].attributes.subject
Sort Fact from Fiction Online with Lateral Reading.

# data[].attributes.attachment_link.title
Sort Fact from Fiction Online with Lateral Reading

# data[].attributes.attachment_link.title
Hertwig & Yanoff (2017) Nudging and boosting.

# data[].attributes.attachment_link.readable_text
What is "boosting"?

Boosting is a policy approach that targets people's competences and thereby helps people make good, informed decisions by themselves.

A boost is an intervention that enlists human cognition, the environment, or both to foster people's existing competences or develop new ones.

See here for more information about boosts.

What types of boosts are there?

See here for the different types of boosts.

How is boosting different from education?

Boosting differs from school education in several ways. This list appears in Hertwig & Grüne-Yanoff (2017, p. 982).

1. Knowledge vs. competence: The primary goal of boosts is not to offer accurate declarative knowledge and cultural skills such as reading, writing, grammar, and algebra. Instead, boosts offer competences in domains that are not typically addressed in school curricula, such as good financial decision making, accurate risk assessment, healthy food choices, informed medical decisions, and effective self-regulation.

2. Behavioral science: Boosts, like nudges, should be informed by behavioral science evidence. This is not necessarily the case for what is taught in schools.

3. Time contraints: Boosts aim to foster or develop new competences under conditions of limited time and resources (on the part of the target audience and the policy makers) and typically in an adult citizenry that cannot be subjected to years of additional schooling.

4. Doing vs. knowing: The focus of boosts is typically on actionable motivational and decisional competences (e.g., procedural routines, heuristics, goal implementation skills) and not on information per se.

5. The right moment: Boosts are often "just-in-time" interventions, whereas school education is scheduled. In all likelihood, people are most motivated to develop a new competence when they experience a specific need for it.

6. Personal agency: Boosts preserve and enable individuals' personal agency and autonomy. (If boosts were included in a mandatory school curriculum, the autonomy of the student would be curtailed.)

How is boosting different from nudging?

Nudging involves making small changes to a specific environment to steer people towards a specific behavior. For example, making a salad the default side dish in company canteens nudges people towards a healthier lunch at work, but has presumably little effect on what they decide to eat when they go out for dinner.

Boosting targets people's competences, which can be used in many other contexts. The focus lies on teaching skills that promote lasting and generalisable changes in judgment and decision-making competences.

Both nudging and boosting can be effective interventions, but which approach or combination works best depends on the application (see the example of a comparative tests of a boost and a nudge for improving hand hygiene). See this checklist for policy makers, which helps assessing whether nudging, boosting, or a combination is likely to work.

When do boosting interventions work?

Boosts can be used to promote better decision-making in many different contexts. A key requirement is that the targeted population of the boost intervention is motivated and able to improve their decision-making by acquiring a new skill or deepening skills they already have.

Boosting has successfully been applied in areas such as medicine, finance, and health to help people invest wisely, establish healthier eating behaviors, and determine the risk of side effects associated with their treatment options (see also the boosting domains).

However, boosting requires the cooperation of those whose decision-making competences are to be boosted-not only during the intervention, but also when the new competences will need to be applied in practice. Boosting is not an appropriate approach for individuals who cannot or do not want to change their behavior.

Why should I care about boosting? Can't we just use nudges?

Boosting and nudging are not interchangeable, although there are circumstances where either type of intervention (or both) could be appropriate. Boosting holds several key advantages over nudging, such as autonomy and transparency. Boosting aims to preserve autonomy and human agency by targeting competences rather than immediate behavior and, because a boost only works if people are willing to engage with it, boosting interventions are necessarily transparent.

Hertwig (2017) suggests the following guidelines to assess whether boosting, nudging, or their combination might be more promising:

1. If individuals lack the cognitive ability or motivation to acquire new skills or competences, then nudging is likely to be the more efficient intervention.

2. If policy-makers are uncertain about people's goals, if there is marked heterogeneity of goals across the population or if an individual has conflicting goals, then boosting is the less error-prone intervention.

3. If the working of a nudge requires it to be non-transparent or even invisible to the person being nudged, then it fails the easy-reversibility test and is paternalistic.

4. If governments do not (always) act benevolently, or if they permit the private sector to create "toxic" choice architectures, then boosting will provide better protection for individuals.

5. If the policy-maker aims to foster generalisable and lasting behaviours, boosting seems, ceteris paribus, to be more expedient.

6. If there is substantial danger of unanticipated (unpredictable) and undesired consequences of a nudging or boosting intervention, then consider the respective alternative.

© honzahruby, licensed at Adobe Stock

Where can I learn more about boosting?

What is still not known about boosting?

The full potential of boosting in behavioral public policy has not yet been realized, and some questions remain open.

Long-term effects: Unlike other interventions, boosting offers the opportunity to create lasting behavior change. We need to know more about the long-term effects of boosting (but also about other behavioral approaches more generally).

Public acceptance: The degree to which the general public and policy makers believe boosting to be an effective, cost-effective, and acceptable means of promoting behavior change should be investigated.

Inequity: People who are unwilling to change will not be reached by boosting, and the motivation to foster or establish competences could be distributed differently across society. We need to know more about whether widespread implementation of boosting could increase inequity by only reaching people who are already able and willing to bear the costs of taking part in boosting interventions. It is not yet known whether boosting would reliably reach those who would benefit the most from a boosting intervention.

# data[].attributes.attachment_link.description
Boosting is a policy approach that targets people's competences and thereby helps people make good, informed decisions by themselves. A boost is an intervention that enlists human cognition, the environment, or both to foster people's existing competences or develop new ones. See here for more information about boosts.

# data[].attributes.headline
Risk Poll- 2021 Lloyd's Register Foundation

# data[].attributes.subject
Risk Poll- 2021 Lloyd's Register Foundation

# data[].attributes.attachment_link.provider_name
The Lloyd's Register Foundation World Risk Poll

# data[].attributes.attachment_link.readable_text
We are now looking for ideas for the 2023 and 2025 polls and are especially interested to hear views on:

Which questions should be retained to show trends over time?

Which questions could be discontinued?

Which questions could be dropped for one year, due, for example, to being slow-moving variables and then brought back in the third wave?

Which questions should be used as a focus to drill down into a particular area in more detail?

Which new areas of focus should be included in the 2021 poll?

We cannot guarantee all suggestions will be included due to limits on length of the questionnaire, but we value all suggestions and comments on making the next wave of research ever more effective and targeted.

# data[].attributes.attachment_link.title
2021 poll - The Lloyd's Register Foundation World Risk Poll

# data[].attributes.attachment_link.description
We are now looking for ideas for the 2023 and 2025 polls and are especially interested to hear views on: Which questions should be retained to show trends over time? Which questions could be discontinued? Which questions could be dropped for one year, due, for example, to being slow-moving variables and then brought back in the third wave?

# data[].attributes.attachment_link.title
Industrialized Disinformation: 2020 Global Inventory of Organized Social Media Manipulation

# data[].attributes.attachment_link.description
Technical Advisory Group Graphika) and Dr. Scott Hale ( / Oxford Internet Institute International Fact Checking Network/Poynter Reporters without Borders/Journalism Trust Inititiave), and Cristina Tardáguila ( ),

# data[].attributes.attachment_link.readable_text
Con el fin de promover el empoderamiento de la ciudadanía en la era de la desinformación, este
informe trata de explicar qué es la desinformación, por qué sus efectos son tan perjudiciales y
cómo pueden los responsables de formulación de políticas hacer frente a sus consecuencias
negativas. Partiendo de la premisa de que se puede y se debe empoderar a la ciudadanía si se
mejora la producción, el consumo y la circulación de la información, este informe analiza en
profundidad dos cuestiones:

* ¿Qué instrumentos, iniciativas y políticas actuales ayudan a la ciudadanía a formarse un
mejor criterio a la hora de consumir información?

* ¿Cómo transmitir mejor a los responsables de formulación de políticas y los
profesionales enfoques sistémicos que les permitan hacer frente a largo plazo a la
desinformación?

Para responder a estas preguntas, la investigación se centra en el análisis de cinco casos de
iniciativas puestas en marcha por diversos actores que tratan de hacer frente al problema de la
desinformación desde una perspectiva multidimensional. Este informe trata de profundizar en
un tema que ha pasado a ser esencial para nuestras democracias.

Cuando mil personas creen en una historia inventada durante un mes, es "fake news". Cuando mil millones de personas lo creen durante mil años, eso es una religión, y se nos aconseja no llamarlo "fake news" para no herir los sentimientos de los fieles (o incurrir en su ira).

# data[].attributes.attachment_link.title
Cómo combatir la desinformación | Digital Future Society

# data[].attributes.attachment_link.description
Con el fin de promover el empoderamiento de la ciudadanía en la era de la desinformación, este informe trata de explicar qué es la desinformación, por qué sus efectos son tan perjudiciales y cómo pueden los responsables de formulación de políticas hacer frente a sus consecuencias negativas.

# data[].attributes.body
Respondents were randomly assigned to exposure to the media literacy intervention or to a con-
trol condition in which no information was shown (simple random assignment with 50% assignment
probability via the YouGov platform). The text shown to respondents is provided in Figure A1. All
respondents in Wave 1 were then shown 8 news articles for evaluation out of a possible 16: 4 pro-
Democratic mainstream news articles (2 from low-prominence sources and 2 from high-prominence
sources), 4 pro-Republican mainstream news articles (2 from low-prominence sources and 2 from
high-prominence sources), 2 pro-Democratic false news articles, 2 pro-Republican false news arti-
cles, 2 pro-Democratic hyperpartisan sources, and 2 pro-Republican hyperpartisan sources. The 8
displayed were chosen by randomly selecting 1 out of each group of 2 listed above for each respon-
dent. In Wave 2, all participants were exposed to all 16 possible articles, thus providing perceptions
of accuracy for repeated and non-repeated articles. Headline stimuli are provided in Figures H1 and
H2. (See the replication archive for this study for the full survey instrument, along with replication
data and code: https://doi.org/10.7910/DVN/Q5QINN.)
We find no evidence of imbalance on observable measures across conditions (Table C3). Sim-
ilarly, attrition between waves was not significantly related to treatment status (Table C4), though
participants who took part in wave 2 had higher levels of political knowledge, were older, and rated
false news as less accurate compared to those who did not take part (Table C5).

# data[].attributes.attachment_link.readable_text
See all Hide authors and affiliations

Edited by David G. Rand, Massachusetts Institute of Technology, Cambridge, MA, and accepted by Editorial Board Member Margaret Levi April 28, 2020 (received for review November 20, 2019)

Significance

Few people are prepared to effectively navigate the online information environment. This global deficit in digital media literacy has been identified as a critical factor explaining widespread belief in online misinformation, leading to changes in education policy and the design of technology platforms. However, little rigorous evidence exists documenting the relationship between digital media literacy and people's ability to distinguish between low- and high-quality news online. This large-scale study evaluates the effectiveness of a real-world digital media literacy intervention in both the United States and India. Our largely encouraging results indicate that relatively short, scalable interventions could be effective in fighting misinformation around the world.

Abstract

Widespread belief in misinformation circulating online is a critical challenge for modern societies. While research to date has focused on psychological and political antecedents to this phenomenon, few studies have explored the role of digital media literacy shortfalls. Using data from preregistered survey experiments conducted around recent elections in the United States and India, we assess the effectiveness of an intervention modeled closely on the world's largest media literacy campaign, which provided "tips" on how to spot false news to people in 14 countries. Our results indicate that exposure to this intervention reduced the perceived accuracy of both mainstream and false news headlines, but effects on the latter were significantly larger. As a result, the intervention improved discernment between mainstream and false news headlines among both a nationally representative sample in the United States (by 26.5%) and a highly educated online sample in India (by 17.5%). This increase in discernment remained measurable several weeks later in the United States (but not in India). However, we find no effects among a representative sample of respondents in a largely rural area of northern India, where rates of social media use are far lower.

Social media platforms have proved to be fertile ground for inflammatory political misinformation. People around the world increasingly worry that so-called "fake news" and other forms of dubious or false information are misleading voters-a fear that has inspired government actions to address the problem in a number of countries ( 1, 2).

Research into online misinformation has thus far focused on political, economic, and psychological factors ( 3⇓ - 5). In this article, we focus on another human vulnerability to online political misinformation: shortfalls in digital media literacy.

While largely overlooked in the emerging empirical literature on digital disinformation and fake news, the concept of digital media literacy usefully captures the skills and competencies needed to successfully navigate a fragmented and complex information ecosystem ( 6). Even under ideal conditions, most people struggle to reliably evaluate the quality of information they encounter online because they lack the skills and contextual knowledge required to effectively distinguish between high- and low-quality news content.

The connection between digital media literacy and misinformation was identified early by theorists. "Misinformation-and disinformation-breeds as easily as creativity in the fever-swamp of personal publishing," according to an influential 1997 introduction to the subject. "It will take all of the critical skills users can muster to separate truth from fiction" (ref. 7, p. xii).

More than 20 y later, these warnings seem prescient. Survey research shows that few people are prepared to effectively navigate the digital world. For example, the Pew Research Center found as recently as 2017 that only 17% of US adults have the skills and confidence to learn new information effectively online ( 8). Nonetheless, people worldwide increasingly obtain news and information from social media platforms that lack traditional editorial controls ( 9, 10), allowing politicians and other actors to widely disseminate misinformation via algorithmic news feeds. Without the necessary digital media literacy skills, people frequently fall victim to dubious claims they encounter in this context.

These concerns have become especially salient in the United States and India in recent years. In the United States, low-quality online articles were distributed widely on social media in the months before the 2016 US presidential election ( 11). This phenomenon created widespread fears that fake news was misleading people at a massive scale ( 12). Smartphone use has also made India, the world's largest democracy, a fertile environment for online rumors and misinformation. Viral misinformation spread via WhatsApp in India has reportedly provoked hatred and ethnic violence ( 13). Moreover, online political misinformation became a significant concern during the 2019 Indian general election as political parties engaged in aggressive digital campaign efforts via short message service (SMS) and messaging applications like WhatsApp ( 14, 15). For instance, one analysis found that over 25% of the news shared on Facebook during the election by the governing Bharatiya Janata Party (BJP) came from dubious outlets ( 16).

Many nonprofits and governments are seeking to counter these trends (and the related threat of foreign manipulation campaigns) by improving the digital media literacy of news consumers ( 17⇓ ⇓ - 20). For instance, American universities increasingly teach media literacy to undergraduate students ( 21) and similar efforts are also being proposed at the kindergarten to grade 12 ( 22). Similarly, WhatsApp and the National Association of Software and Service Companies announced plans to train nearly 100,000 people in India through in-person events and posts on social media to spot misinformation ( 23).

Despite the attention and resources these initiatives have received, however, little large-scale evidence exists on the effectiveness of promoting digital media literacy as a response to online misinformation. Existing scholarly work related to digital and media literacy is frequently qualitative in nature or focused on specific subpopulations and/or issues. Observational findings are mixed ( 24, 25) and randomized controlled trials remain rare ( 26).

Two related but more specific approaches have been shown to be somewhat effective in countering misinformation and are important to note, however. First, inoculation interventions have been employed to protect audiences against misleading content by warning of misinformation and either correcting specific false claims or identifying tactics used to promote it. This approach has been shown to reduce the persuasiveness of misinformation in specific domains ( 27⇓ ⇓ ⇓ ⇓ - 32). In addition, other studies evaluate the effectiveness of providing warnings about specific misinformation ( 33, 34).

We therefore seek to determine whether efforts to promote digital media literacy can improve respondents' ability to correctly evaluate the accuracy of online content across issues. Such a finding would suggest that digital media literacy shortfalls are a key factor in why people fall victim to misinformation. In particular, we consider the effects of exposure to Facebook's "Tips to Spot False News," which were developed in collaboration with the nonprofit First Draft and subsequently promoted at the top of users' news feeds in 14 countries in April 2017 and printed in full-page newspaper advertisements in the United States, the United Kingdom, France, Germany, Mexico, and India ( 35⇓ ⇓ ⇓ ⇓ - 40). A variant of these tips was later distributed by WhatsApp (a Facebook subsidiary) in advertisements published in Indian and Pakistani newspapers in 2018 ( 41, 42). These tips are therefore almost surely the most widely disseminated digital media literacy intervention conducted to date. (The full treatments are provided in SI Appendix, section A.) The US treatment, which was adapted verbatim from Facebook's campaign, consists of 10 strategies that readers can use to identify false or misleading stories that appear on their news feeds, whereas the India treatment, which uses adapted versions of messages shown in India by Facebook and WhatsApp, presents 6.

These interventions provide simple rules that can help individuals to evaluate the credibility of sources and identify indicators of problematic content without expending significant time or attention. For instance, one sample tip recommends that respondents "[b]e skeptical of headlines," warning that "If shocking claims in the headline sound unbelievable, they probably are." Such an approach should reduce reliance on low-effort processes that frequently lead people astray (e.g., perceptions of cognitive fluency) by teaching people more effective heuristics (e.g., skepticism toward catchy headlines). Importantly, the success of this approach does not require readers to take burdensome steps like conducting research or thinking deeply about each piece of news they encounter (which is typically impossible in practice given the volume of stories that social media users encounter). Instead, this intervention aims to provide simple decision rules that help people distinguish between mainstream and false news, which we call "discernment" following ref. 4.

There are important reasons to be skeptical about the effectiveness of this approach. Prior research has found that media literacy interventions like this can help people think critically about the media content they receive ( 43). However, prior studies focus mostly on offline health behavior; the extent to which these interventions are effective for controversial political claims or online (mis)information is largely unknown. Moreover, such interventions may struggle to overcome people's reliance on heuristics such as familiarity and congeniality that news consumers use to evaluate the credibility of online stories ( 44, 45). Finally, attempting to identify false news through close scrutiny of a headline differs from the typical approach of professional fact checkers, who usually use "lateral reading" of alternative sources to corroborate claims ( 46).

We therefore conducted preregistered survey experiments in both the United States and India examining the effectiveness of presenting people with "tips" to help spot false news stories. [The US and India studies were each preregistered with Evidence in Governance and Politics; see Materials and Methods. All preregistered analyses are reported in this article or in the replication archive for the study ( 47).] Strikingly, our results indicate that exposure to variants of the Facebook media literacy intervention reduces people's belief in false headlines. These effects are not only an artifact of greater skepticism toward all information-although the perceived accuracy of mainstream news headlines slightly decreased, exposure to the intervention widened the gap in perceived accuracy between mainstream and false news headlines overall. In the United States, the effects of the treatment were particularly strong and remained statistically measurable after a delay of approximately 3 wk. These findings suggest that efforts to promote digital media literacy can improve people's ability to distinguish between false and mainstream news content, a result with important implications for both scientific research into why people believe misinformation online and policies designed to address the problem.

Our main research hypotheses evaluate whether the media literacy intervention reduces belief in false news stories (hypothesis 1 [H1]), increases belief in mainstream news content (H2), and improves respondents' ability to distinguish between them (H3). We also consider three research questions (RQs) for which our a priori expectations were less clear. First, past research shows that the effects of many experimental treatments (e.g., in persuasion and framing studies) decay quickly over time ( 48), although providing participants with novel information may have more long-lasting effects ( 49). We therefore test the durability of our treatment effect by leveraging a two-wave panel design to tests its effects several weeks after the initial intervention (RQ1). Second, it is also possible that interventions may work only to make individuals more skeptical of noncongenial content they are already inclined to dismiss, leaving their vulnerability to ideologically consistent misinformation unchanged. We therefore test for the heterogeneity of the treatment effects based on the partisan congeniality of the content (RQ2). Finally, we test whether the intervention changed self-reported intentions to share false stories or subsequent online news consumption behavior in the US sample where these measures were available (RQ3). Additional analyses exploring heterogenous treatment effects and alternate outcomes are discussed below, but full models appear in SI Appendix, section C. These analyses include whether intuitive cognitive style or prior headline exposure moderates the treatment effect, as well as whether the treatment affects the perceived credibility of "hyperpartisan" headlines.

Results

US Survey Experiment.

Consistent with our first hypothesis (H1), randomized exposure to the media literacy intervention causes a decrease in the perceived accuracy of false news articles. Results from wave 1 of the US study in Table 1 show a decrease of nearly 0.2 points on a 4-point scale (intent to treat [ITT]: , ; ). We observe similar effects of the media literacy intervention on the perceived accuracy of hyperpartisan headlines (ITT: , ; ) (SI Appendix, section C, Table C2).

One concern is that the intent-to-treat effects described above understate the true effect of the intervention, which may have been neglected by some respondents. While we can offer the opportunity to read the digital literacy "fake news tips" intervention to a random subset of respondents, we cannot force every respondent to read these tips carefully.

We therefore also estimate the effect of the treatment on those who actually received it, which is known as the average treatment effect on the treated (ATT), using an instrumental variables approach. In this model, our indicator for receipt of treatment is the ability to correctly answer a series of follow-up questions about the content of the news tips (approximately two-thirds of respondents in the treatment condition [66%] were successfully treated) and our instrument is the original random assignment. Table 1 reports the ATT, which we compute using two-stage least-squares regression. With this approach, we estimate that the perceived accuracy of false headlines decreased by nearly 0.3 points on a 4-point scale (ATT: , ; ).†

We compare the characteristics of respondents who would successfully take the treatment only if assigned to it ("compliers") to those who would not even if assigned to treatment ("never takers") (SI Appendix, section B) ( 50). Compliers were more likely to be older, college graduates, interested in politics, politically knowledgeable, Republican identifiers, and more polarized in their feelings toward the two political parties than never takers. Compliers also scored lower in conspiracy predispositions and their feelings toward Donald Trump. However, the substantive magnitudes of most of these differences are modest (SI Appendix, section B, Fig. B1). Crucially, we find no statistically significant evidence that respondents who take the treatment differ in their baseline propensity to visit untrustworthy websites compared to those who do not (analysis conducted among participants for whom presurvey behavioral data are available; see SI Appendix, section A for details). The average number of prior visits to false news websites is actually greater among compliers than among never takers but this difference does not reach conventional thresholds of statistical significance (0.35 compared to 0.18; ).

Our next hypotheses predicted that the media literacy intervention would increase the perceived accuracy of mainstream news (H2) and increase people's ability to successfully distinguish between mainstream and false news articles (H3). These results are shown in the second and third columns in Table 1. We find that exposure to the media literacy intervention had a small negative effect on belief in mainstream news in wave 1 (ITT, [], ; ATT, [], ). However, the negative effects of the intervention on the perceived accuracy of false news described above are larger. As a result, the media literacy intervention increased discernment between mainstream and false stories (ITT, [], ; ATT, [], ), demonstrating that it helped respondents to better distinguish between these two types of content. In relative terms, this effect represents a 26.5% improvement in respondents' ability to distinguish between mainstream and false news stories compared to the control condition.

In addition, we test the durability of these treatment effects in wave 2 per RQ1. After a delay between waves that averaged several weeks, the effect of the media literacy intervention on the perceived accuracy of false headlines remains statistically distinguishable from zero (SI Appendix, section C, Table C1). The median interval between waves was 20 d; the 5 95 t h percentile range was 16 to 29 d. While the effect is still present weeks later, its magnitude attenuates by more than half relative to wave 1 (ITT, [], ; ATT, [], ). In addition, the negative effect of the media literacy treatment on the perceived accuracy of mainstream news content was no longer statistically measurable by wave 2. As a result, the perceived accuracy difference between mainstream and false headlines remained statistically distinguishable from zero in the second wave, although its magnitude decayed (; ; ). t h to

Fig. 1 illustrates the substantive magnitude of the intent to treat effects of the media literacy intervention in the United States using a binary indicator of perceived headline accuracy. The proportion of respondents rating a false headline as "very accurate" or "somewhat accurate" decreased from 32% in the control condition to 24% among respondents who were assigned to the media literacy intervention in wave 1, a decrease of 7 percentage points. This effect represents a relative decrease of approximately one-fourth in the percentage of people wrongly endorsing misinformation. Treatment effects continue to persist with this alternate measure-in wave 2, the intervention reduced the proportion of people endorsing false headlines as accurate from 33 to 29%, a 4-percentage-point effect. By contrast, the proportion of respondents who classified mainstream news as not very accurate or not at all accurate rather than somewhat or very accurate decreased only from 57 to 55% in wave 1 and 59 to 57% in wave 2.

Finally, RQ2 explores whether the effects of the media literacy intervention are moderated by the partisan congeniality of the headlines people rated. We find no consistent evidence that the effects of media literacy intervention are moderated by congeniality (SI Appendix, section C). In other words, the effects of the intervention were not differentially concentrated among headlines that were uncongenial to respondents-an encouraging null result that echoes findings in recent studies ( 34, 51, 52).

Additional results reported in SI Appendix, section C indicate that we have insufficient evidence to conclude that the intervention had an effect on self-reported intention to share false news or behavioral measures of news consumption (SI Appendix, section C, Table C13). However, the intervention did significantly increase sharing intentions for mainstream news and decrease sharing intentions for hyperpartisan news. This is consistent with previous studies that have reported mixed effects of warning labels on sharing intent ( 33, 34). The mixed results we observe for sharing intent may be attributable to the fact that belief accuracy questions appeared immediately before the sharing intent questions in the survey, which may prime accuracy concerns among respondents and thereby alter both real and self-reported sharing behavior. In addition, we find no measurable effect of the intervention on posttreatment visits to false news, mainstream news, or fact-checking sites, although these effects can be estimated only among the subset of respondents for whom we have behavioral data (SI Appendix, section C, Tables C14-15).

India Survey Experiments.

As detailed in Materials and Methods, we conducted separate online and face-to-face surveys in India of different populations. For the online sample, we again find support for H1. The media literacy treatment significantly reduced beliefs in false news stories (ITT: , SE = 0.026; ) in the first wave of a two-wave survey ( Table 2). As with the US analysis, the ATT estimate was substantially larger ( Table 2), although the difference between the ITT and the ATT is larger for the Indian analysis because fewer respondents answered all comprehension checks correctly in the Indian sample (28% in the online sample versus 66% in the United States).‡ Respondents to the online survey who received the treatment were nearly half of a response category more skeptical of false news stories (ATT: , SE = 0.097; ).

As in the United States, we also find no support for H2, which predicted that exposure to the treatment would increase the perceived accuracy of mainstream news. Instead, the perceived accuracy of mainstream news decreased, although by less than the effect observed for false news (ITT, [SE = 0.025], ; ATT, [SE = 0.095], ). Results again mirror the US study for H3-respondents better distinguished between mainstream and false articles (ITT, [SE = 0.025], ; ATT, [SE = 0.088], ).§ While the magnitude of this effect is lower than for the US sample, it translates to a 17.3% improvement in discernment between mainstream and false news relative to the difference observed in the control condition. As we discuss in more detail below, treatment effects cannot be distinguished from zero in the second wave (RQ1).

While the analyses of the online samples from the United States and India show substantially similar results, results from the face-to-face survey in India differ in important ways. As shown in Table 2, we find no evidence that the treatment increased the perceived accuracy of mainstream news articles as predicted by H2. However, it did not reduce the perceived accuracy of these headlines either as we found in the United States and online studies. In addition, unlike the other studies, we find no evidence that the media literacy treatment systematically affected beliefs in false news stories (H1) or discrimination between false and mainstream news (H3) among India face-to-face respondents.

We directly assess the difference between the two samples using a pooled model. These results, presented in SI Appendix, section D, Table D13, indicate that we can reject the null of no difference in media literacy intervention effects between the face-to-face and online samples for ratings of false news (ITT estimate of the face-to-face − online difference, [SE = 0.035], ; ATT, [SE = 0.149], ) and for the ITT estimate for ratings of mainstream news stories (ITT, β β=0.266 [SE=0.148] , ), but not for the average difference in perceived accuracy between mainstream and false news stories (ITT, [SE = 0.039], ; ATT, [SE = 0.163], ). Potentially, the differences in our results between studies may reflect the different survey modes or demographic compositions of the samples (we consider this issue further in =0.072 [SE=0.035] , ; ATT, Discussion).

Fig. 2 illustrates the substantive magnitudes of the ITT effect for respondents to the two India surveys. For the online survey, exposure to the media literacy intervention reduced the percentage of respondents rating false headlines as somewhat accurate or very accurate from 49% in the control group to 44% in the treatment group, a decrease of approximately 10% in relative terms. As noted above, the effect of the intervention was much greater for those who received the treatment successfully-the ATT estimate indicates a decline of approximately 19 percentage points in endorsement of false headlines among this group (SI Appendix, section D, Table D7). By contrast, although mainstream stories were also viewed more skeptically by online survey respondents who received the media literacy intervention (from 63% for controls to 60% in the treatment group), the relative decrease in perceived accuracy was only half of what was observed for false headlines. Finally, as noted above, there was no significant difference on average between face-to-face survey respondents who received the media literacy treatment and those who did not for either false or mainstream headlines (belief levels were higher overall among face-to-face respondents-62% for false headlines and 72% for mainstream headlines).

A key research question was whether any treatment effects would persist over time (RQ1). We found no statistically reliable evidence that the treatment affected headline accuracy ratings among wave 2 respondents in either India sample (SI Appendix, section D, Table D2).¶ Finally, we did not find statistically reliable evidence in either India survey that the media literacy intervention's effects were moderated by partisan congeniality (RQ2; SI Appendix, section D, Table D9). We cannot conclude that the effects we observe depend on whether the headlines were congenial to respondent partisanship.

Discussion

Comparing our results across studies reveals a relatively consistent pattern. As Fig. 3 indicates, both the US study and the India online study find negative effects on the perceived accuracy of false headlines that are diminished for mainstream news headlines. As a result, respondents' ability to discern between mainstream and false news increased. Although these findings are not observed in the India face-to-face study, a combined estimate pooling data from all three studies replicates the overall pattern of reduced false news accuracy perceptions and increased discernment between mainstream and false news. The treatment effect estimates for each study as well as the pooled results are shown in Fig. 3 (see SI Appendix, section F for full model results).

These effects are also substantively meaningful. Although our study does not instruct respondents to apply the lessons from the intervention to the headline ratings task, the effect sizes are comparable to the estimated effect of exposure to the labels Facebook initially used to indicate that articles were disputed by fact checkers (ITT estimate: β SE=β SE=0.034 ; ). A comparison of effect sizes with other randomized media literacy interventions in =−0.079 , 0.036 ; ) (=−0.236 , 34) and greatly exceed the effects of a general warning about false news in the same study (ITT estimate: SI Appendix, section E, Table E1 shows that our US study has the largest measured effect size to date on ratings of false headlines () and that the India online study () falls in the upper middle of the distribution. Moreover, effect sizes are substantially larger for respondents who were successfully treated with the media literacy intervention.

Despite the strength of the combined estimates, the effectiveness of the intervention varied across samples. First, the intervention may have been more unfamiliar or difficult to understand for Indian respondents, who successfully received the treatment at a much lower rate than those in the United States. Additional evidence suggests that respondents may have applied the intervention differently. Specifically, the US data show that the negative effects of the media literacy intervention on perceived headline accuracy were greater for headlines from untrustworthy, hyperpartisan, and unfamiliar mainstream sources that respondents in the control group found less plausible to begin with (; SI Appendix, section C, Table C11 and Fig. C2). This finding helps explain the observed negative effect of the media literacy intervention on the perceived accuracy of mainstream news overall.# By contrast, no such relationship between baseline headline accuracy and media literacy intervention effects is observed in the Indian online data ( r=−0.09 ; SI Appendix, section D8), suggesting respondents became more skeptical across the board.

Problems applying the intervention may have been particularly acute for respondents in the face-to-face sample. This group is quite dissimilar from both the highly educated online sample in India and the US sample on a number of important indicators (SI Appendix, section B). In particular, participants in the face-to-face study had much less experience with the task of evaluating news headlines online-only 11% reported using WhatsApp compared with 90% for the online sample in India. Correspondingly, an exploratory analysis shows the effects of the intervention were similar among WhatsApp users across samples; however, these effects are imprecisely estimated among face-to-face participants because WhatsApp use was so rare (SI Appendix, section D9).

Finally, we consider the potential trade-off between increased skepticism toward false news headlines and decreased belief in mainstream news headlines. Our results do indicate that increased skepticism of false news headlines may come at the expense of decreased belief in mainstream news headlines-the media literacy intervention reduced the perceived accuracy of these headlines in both the US and India online surveys. However, the magnitude of the decrease in the perceived accuracy of mainstream news headlines ranges from under 25% (United States) to just over half (India online sample) of the estimated size of the decrease in the perceived accuracy of untrustworthy news headlines in wave 1 of our surveys. As a result, respondents' overall ability to distinguish between mainstream and untrustworthy news increases by more than 26% in the US sample and 17% in the highly educated online Indian sample. Moreover, we observe no measurable decrease in the perceived accuracy of mainstream news headlines in wave 2 of any of our surveys.

A related concern is that the intervention could reduce the overall accuracy of people's beliefs given that they typically consume much more information from mainstream sources than from untrustworthy ones ( 53). To address this concern, we use US Pulse web metering data to estimate the overall change that the intervention would hypothetically induce in people's ability to accurately discern credible news given exposure rates for US participants to different types of news sources (see SI Appendix, section G for details). Because Americans' news consumption is concentrated among the high-prominence mainstream outlets for which the intervention may have had a small positive effect#, these calculations indicate that individuals would reach valid accuracy beliefs for 64.6% of stories in the treatment group compared to 62.9% in the control group. Moreover, the percentage of "false positives"-stories they encounter from dubious sources and believe to be true-would decrease from 6.1% of all stories consumed in the control group to 4.9% in the treatment group.

Conclusion

The findings we present provide important evidence that shortfalls in digital media literacy are an important factor in why people believe misinformation that they encounter online. We find that a simple, scalable media literacy intervention can decrease the perceived accuracy of false news content and help people to better distinguish it from factual mainstream news in both the United States and India. Moreover, the improvement in headline accuracy rating performance we observe does not depend on whether the claims in question align with respondents' political predispositions.

Our results further suggest that media literacy campaigns could be an effective strategy to help counter false or misleading news, a finding with important real-world implications. Some explanations for belief in misinformation identify factors that resist intervention ( 4), while others propose policy approaches that are effective in practice but difficult to scale ( 34, 44). Inoculation, while effective in preemptively refuting misinformation in specific domains ( 27⇓ ⇓ - 30), may not improve discernment when evaluating a diverse array of real-world news content. By contrast, these results show that a brief intervention which could be inexpensively disseminated at scale can be effective at reducing the perceived accuracy of false news stories, helping users more accurately gauge the credibility of news content they encounter on different topics or issues.

Although these results suggest that digital literacy interventions could be a valuable tool in the fight against misinformation, several caveats should be offered. First, the effect sizes were modest; this simple intervention did not eliminate belief in false news headlines. Second, the effects decayed over time (diminishing in magnitude in the United States and no longer remaining statistically measurable in the India online study), suggesting the need for social media companies, journalists, and educators to reinforce these lessons on a recurring basis. Third, although the intervention improved overall discernment between mainstream and false news headlines, it did have a small but measurable negative effect on the perceived accuracy of mainstream news stories. Fourth, all treated participants were exposed to the intervention; many real-world Facebook users are likely to have ignored the Tips to Spot False News link when the company offered it in their feeds. (The difference between our ITT and ATT estimates illustrates how lack of attention to the treatment reduces its effectiveness.) Finally, we have insufficient evidence to conclude that the intervention changed real-world consumption of false news, perhaps because information habits are ingrained and difficult to alter. However, we do find evidence that the treatment increased respondents' intent to share mainstream news and decreased intent to share hyperpartisan news, suggesting the possibility of changes to social media behavior.

Our findings also suggest a number of directions for future research. One advantage of the study is that we used an actual intervention deployed globally by a technology company that has served as an important access point to false news ( 53). However, scholars should conduct comparative evaluations of the effects of different interventions rather than relying on this model as a default and test the effectiveness of these approaches in different samples, countries, and electoral contexts. These evaluations should include tests of more intensive digital literacy training models (such as the "lateral reading" approach used by professional fact checkers), which could potentially have larger and/or more durable effects. In addition, scholars should seek to better understand the mechanism through which such interventions operate, identifying whether the effects they observe are due to increased accuracy concerns versus helping people learn more effective heuristics for evaluating news content. Finally, researchers should further examine whether and how media literacy interventions can increase the frequency or effectiveness of accuracy-promoting behavior in social contexts. Even if these interventions do not reach everyone, improving the media literacy of a subset of the population could yield wider positive effects if, for instance, those who are treated help to correct the misinformation they see on social media ( 54).

To our knowledge, though, these findings are the most compelling demonstration to date that a real-world digital literacy intervention can have significant and potentially lasting effects. While efforts to improve online digital literacy are not a panacea, they may prove to be a vital and cost-effective way to reduce people's vulnerability to false news and thereby improve the information health of democracies.

Materials and Methods

Data Collection.

We conducted two-wave panel surveys of respondents that included an embedded media literacy intervention. One survey took place in the United States and two were conducted in India. All took place during periods of high political interest during and immediately after national electoral campaigns.

In the United States, we conducted a two-wave online panel survey fielded by the survey company YouGov shortly after the 2018 US midterm elections (wave 1, November 20 to December 27, 2018, ; wave 2, December 14, 2018 to January 3, 2019, ).‖ Respondents were selected by YouGov's matching and weighting algorithm to approximate the demographic and political attributes of the US population (32% college graduates, 45% male, median age 50 y; 46% identify as Democrats and 36% as Republicans). A subset of these respondents were members of the YouGov Pulse panel and voluntarily provided behavioral data on their online information consumption as well (see SI Appendix, section A for more details).

US data collection was approved by the Institutional Review Boards (IRBs) at the University of Michigan (HUM00153414), Washington University in St. Louis (201806142), and Princeton University (10875). University of Exeter accepted the University of Michigan IRB approval. All subjects gave consent to participate in our study. The US study preanalysis plan is available at https://osf.io/u3sgc.

For India, we conducted two separate two-wave panel studies, one online and the other face to face. Both surveys were conducted in Hindi. Respondents were excluded if indicated they mostly or always give humorous or insincere answers to survey questions (which amounted to 7-8% of responses in the online survey by wave compared to less than 1% in the face-to-face survey; this exclusion represents a deviation from our preregistration, but the results in Table 2 are robust to including these respondents). In the online survey, we collected survey data from a national convenience sample of Hindi-speaking Indians recruited via Mechanical Turk and the Internet Research Bureau's Online Bureau survey panels (wave 1, April 17 to May 1, 2019, ; wave 2, May 13 to 19, 2019, ). The India face-to-face survey was conducted by the polling firm Morsel in Barabanki, Bahraich, Domariyaganj, and Shrawasti, four parliamentary constituencies in the state of Uttar Pradesh where Hindi is the dominant language (wave 1, April 13 to May 2, 2019, ; wave 2, May 7 to 19, 2019, ). These locations were chosen, in part, due to their higher levels of religious polarization, which we anticipated might increase demand for and belief in online misinformation and rumors. The representative random sample for the India face-to-face survey was drawn from the public voter registration list for these constituencies and was administered orally by trained enumerators to account for low literacy rates. Relative to the face-to-face survey, online respondents were more likely to be male (72% versus 64%), younger (median age 30 y versus 37 y), more educated (76% college graduates versus 6%), higher caste (42% low caste versus 74% in the face-to-face sample), more active on social media (90% use WhatsApp versus 11%), more interested in politics (3.7 versus 2.9 on a 4-point scale), more knowledgeable about politics (providing correct responses to an average of 2.8 vs. 2.1 of four true-false questions about Indian politics), and slightly less likely to support the BJP (42% versus 46%) (SI Appendix, section D, Table D1).

India data collection was approved by the IRBs at the University of Michigan (HUM00160358), Ashoka University, and Morsel Research and Development (HIRB000007598). University of Exeter accepted the University of Michigan IRB approval. All subjects gave consent to participate in our study. The India study preanalysis plan is available at https://osf.io/97rnz.

Our study contexts can be viewed as a most-different case comparison among democracies ( 55). India and the United States are broadly considered the poorest and richest countries, respectively, in terms of income per capita among longstanding large democracies (ref. 56, p. 42). As a result, India is likely to have lower levels of education and media literacy than the United States, which raises questions about the efficacy of any media intervention. The two studies we conduct within India further refine this comparison, allowing us to evaluate the effects of the media literacy intervention among both an online sample that has demographics that are more similar to the United States and a face-to-face sample in one of the poorest regions in India. Our study can thus provide evidence about the efficacy of a media literacy intervention across democratic contexts that differ in levels of income, education, and digital media use.

News Headline Rating Task.

The main outcome of interest in all three surveys was the perceived accuracy of mainstream and false news headlines. To construct this measure, we asked respondents to evaluate the accuracy of a number of headlines on a 4-point scale ranging from very accurate (4) to not at all accurate (1). All of the headlines were published by actual news sources or circulated on Facebook or WhatsApp within 6 mo of the respective survey, and a portion of the headlines were rated as false by at least one third-party fact-checking organization. The order of the headlines was randomized within wave for each respondent. All headlines are shown in SI Appendix, section H1.

In the US survey, respondents evaluated 16 different headlines that varied across multiple dimensions: news type (mainstream versus hyperpartisan versus false), valence (pro-Democrat versus pro-Republican), and prominence among mainstream sources (high versus low). We define high-prominence mainstream sources as those that more than 4 in 10 Americans recognize in Pew polling ( 57). Hyperpartisan stories are those that are technically factual but present slanted facts in a deceptive manner. We selected these stories from hyperpartisan sources identified in prior work ( 58) (SI Appendix, section H). This process resulted in 8 mainstream, 4 false, and 4 hyperpartisan headlines. In wave 1, respondents were shown 8 headlines (a randomly selected headline from the two available for each possible combination of news type, valence, and source prominence), while respondents in wave 2 were shown all 16 headlines. Headlines were presented as they would appear on the Facebook news feed to replicate a typical decision environment. Specifically, respondents were shown the article previews that are automatically generated by Facebook when a link is entered into the news feed that featured a headline, a photo, the news source's web domain, and in certain cases a byline or story text snippet. Respondents were asked to rate the accuracy of each headline.

In the India surveys, we adopted the same approach in asking respondents to evaluate the accuracy of headlines that varied across several dimensions: valence (congenial to BJP supporters versus congenial to BJP opponents) and accuracy (true articles from mainstream sources†† versus false articles as identified by fact checkers). Nationalism is also commonly linked to misinformation in India ( 59). The issue was particularly salient when the India surveys were conducted (a time of escalating tensions between India and Pakistan), so we also asked respondents to rate the accuracy of true and false headlines relevant to nationalist concerns in the country (either India-Pakistan or Hindu-Muslim relations). Unlike the US study (where the same headlines were used in both waves 1 and 2 to test for prior exposure effects), we used different sets of headlines in each wave. Finally, 4 additional false headlines were included in the second wave based on fact checks conducted between the two waves.‡‡ In total, respondents rated 12 headlines in wave 1 (6 false and 6 true) and 16 in wave 2 (10 false and 6 true). Respondents were presented with the headline in text format in the online survey, while enumerators read the headlines to respondents in the face-to-face survey. In both cases, participants were asked to evaluate the accuracy of all headlines they were presented in each wave.

Analysis.

Our primary analyses are pooled OLS models predicting perceptions of headline accuracy on a four-point scale that ranges from not at all accurate to very accurate. These models were estimated at the headline level with fixed effects for each headline. Although we attempted to choose stories that were balanced in their face validity, the headlines differed in plausibility because the actual stories were not constructed by researchers. We therefore use the fixed effects to account for baseline differences in perceived accuracy between headlines. Because respondents rated multiple headlines, we also compute cluster-robust standard errors. In addition to the pooled OLS models, we also examine the difference in accuracy beliefs between mainstream and false headlines at the respondent level by calculating a respondent-level measure of the difference in mean levels of perceived accuracy between mainstream and false headlines. Higher scores on this scale indicate better ability to discern between stories of different types.

Congeniality is a binary variable that is coded at the headline level for partisans to indicate whether a story is consistent with a respondent's partisan leanings (e.g., a Democrat evaluating a story that is favorable to a Democrat would take the value of 1). Uncongenial is coded as the opposite. The baseline category is reserved for headline ratings by pure nonpartisans. To determine the partisanship of respondents in the US survey, we used the standard two-question party identification battery (which includes leaners) to classify respondents as Democrats or Republicans. Because India has a multiparty system, we classified respondents there as BJP supporters if they reported liking the BJP more than any other party (on a four-point scale) and as a BJP opponent if they liked any other party more than the BJP.

The key explanatory variable of interest is exposure to the media literacy intervention, which was adapted from an intervention deployed by Facebook and WhatsApp around the world, including in Hindi-language newspapers in India (see SI Appendix, section A for details and the exact text). We randomly assigned respondents in wave 1 of the US and India studies with probability 0.5 to be exposed to a set of tips for distinguishing false news stories from mainstream stories. In the US survey experiment, 10 tips published by Facebook were presented verbatim in 2 groups of 3 and one group of 4. In the India surveys, 6 tips from those published by Facebook and WhatsApp were presented in 2 groups of 3 after being adapted for the face-to-face format (omitting cues such as URLs that would not be present and reducing their length when possible). The treatment was administered before the headline rating task and respondents were asked comprehension questions after each group of tips to determine receipt of treatment. We calculate both ITT estimates using the full sample and the ATT below.

For calculating the ATT, receipt of treatment is defined as answering all comprehension questions correctly on the first try (online participants had up to three chances to answer correctly; face-to-face respondents had one chance). Receipt of treatment was substantially higher in the United States (66% in the US online survey and 24% and 28%, respectively, in the India face-to-face and online surveys). Our ATT estimates likely understate effects for two reasons. A small fraction of respondents may be misclassified as compliers because they guessed correctly on all of the comprehension checks, which will diminish our ATT estimates relative to the true effect (although the likelihood of such an outcome under random guessing is low relative to the estimated compliance rates). Additionally, some respondents we classify as receiving treatment are effectively "always takers"-people who either saw the tips in real life or have already internalized them through frequent experience. The intervention should have no effect on them. As such, our ATT estimates will understate the true effect.

Data Availability.

Data files and scripts necessary to replicate the results in this article are available at the Dataverse repository: https://doi.org/10.7910/DVN/Q5QINN.

Acknowledgments

We thank Democracy Fund, the European Research Council under the European Union's Horizon 2020 research and innovation program (Grant 682758), and the Facebook Integrity Foundational Research Awards for funding support and Sumit Parashar for outstanding research assistance. We are also grateful to Calvin Lai, Ben Kaiser, Ro'ee Levy, Jonathan Mayer, Kevin Munger, Yusuf Neggers, Christine Stedtnitz, Emily Thorson, Tim Vercellotti, Shun Yamaya, Dannagal Young, and Thomas Zeitzoff; to Henrik Oscarsson, Jesper Stromback, and the "Knowledge Resistance" group at the University of Gothenburg; and to seminar participants at Brigham Young University, George Washington University, Harvard Kennedy School, Katholieke Universiteit Leuven, and Stanford University for helpful comments. We thank YouGov, Morsel Research and Development, and the Internet Research Bureau for assistance administering the surveys. The India study was supported by funding from Facebook, but the company and its employees played no role in how the study was designed, conducted, or analyzed. All conclusions and any errors are our own.

Footnotes

Author contributions: A.M.G., M.L., B.L., J.M.M., B.N., J.R., and N.S. designed research, performed research, analyzed data, and wrote the paper.

The authors declare no competing interest.

This article is a PNAS Direct Submission. D.G.R. is a guest editor invited by the Editorial Board.

Data deposition: Data files and scripts necessary to replicate the results in this article are available at the Dataverse repository at https://doi.org/10.7910/DVN/Q5QINN. The US study preanalysis plan is available at https://osf.io/u3sgc. The India study preanalysis plan is available at https://osf.io/97rnz.

↵‡The analysis of compliers is presented in SI Appendix, section B. As SI Appendix, Fig. B2 shows, compliers in the India online sample (those who would take the treatment if assigned) were more likely to be young, male, Hindu, and high caste; to have graduated from college; to use WhatsApp; and to have more political knowledge and interest than never takers (respondents who would not take the treatment if assigned to receive it). We find no significant differences between these groups in the face-to-face sample (SI Appendix, section B, Fig. B3).

↵¶The median intervals between waves were 20 d for the face-to-face survey and 19 d in the online survey; the 5 95 t h percentile ranges were 14 to 29 d and 15 to 26 d, respectively. The wave 2 results are substantively unchanged if we include the four additional false headlines from the fact-check message experiment described in t h to SI Appendix, section A (SI Appendix, section D, Table D10).

↵‖ Respondents took the surveys in order with a delay between them averaging approximately 3 wk. The dates above reflect the ranges in which all wave 1 and wave 2 responses were collected.

↵††Mainstream news sources included ZeeNews, Washington Post, National Herald India, IndiaToday, Nikkei Asian Review, Reuters, and Bloomberg.

This article contains supporting information online at https://www.pnas.org/lookup/suppl/doi:10.1073/pnas.1920498117/-/DCSupplemental.

# data[].attributes.attachment_link.title
A digital media literacy intervention increases discernment between mainstream and false news in the United States and India

# data[].attributes.attachment_link.description
Few people are prepared to effectively navigate the online information environment. This global deficit in digital media literacy has been identified as a critical factor explaining widespread belief in online misinformation, leading to changes in education policy and the design of technology platforms. However, little rigorous evidence exists documenting the relationship between digital media literacy and people's ability to distinguish between low- and high-quality news online.

# data[].attributes.headline
A digital media literacy intervention increases discernment between mainstream and false news in the United States and India- A.1.3 U.S. news tip intervention

# data[].attributes.subject
A digital media literacy intervention increases discernment between mainstream and false news in the United States and India- A.1.3 U.S. news tip intervention

# data[].attributes.body
Sugerencias para detectar las noticias falsas

Queremos impedir la difusión de noticias falsas en Facebook. Obtén más información sobre las medidas que estamos tomando en este sentido. Mientras trabajamos para limitar la difusión de este tipo de noticias, te ofrecemos unas cuantas formas de identificarlas:

Desconfía de los titulares. A menudo las noticias falsas tienen titulares llamativos en mayúsculas con signos de exclamación. Si el titular contiene afirmaciones impactantes que resultan inverosímiles, lo más probable es que sean falsas.
Examina la URL. Una URL falsa o que copia una real puede indicar que se trata de una noticia falsa. Muchos de los sitios web de noticias falsas imitan fuentes de noticias auténticas haciendo pequeños cambios en la URL. Puedes ir al sitio web para comparar la URL con las fuentes oficiales.
Investiga la fuente de la noticia. Asegúrate de que la historia provenga de una fuente de confianza, que cuente con una buena reputación por su veracidad. Si la historia procede de una organización desconocida, consulta la sección “Información” de su sitio web para conocer más detalles.
Presta atención al formato. Muchos sitios de noticias falsas tienen faltas de ortografía o un diseño extraño. Si detectas cualquiera de estos indicios, lee con atención.
Presta atención a las fotos. Las noticias falsas suelen contener imágenes o vídeos manipulados. En ocasiones, la foto puede ser auténtica, pero haber sido sacada de contexto. Puedes hacer una búsqueda de la foto o imagen para verificar su procedencia.
Revisa las fechas. Las noticias falsas pueden tener una cronología sin sentido o incluir fechas que han sido alteradas.
Verifica los hechos. Verifica las fuentes del autor para confirmar que son exactas. Si no se aportan pruebas o se confía en expertos cuya identidad no se menciona, es posible que la noticia sea falsa.
Consulta otras noticias. Si ninguna otra fuente de noticias informa de la misma historia, es posible que sea falsa. Si, en cambio, varias de las fuentes en las que confías informan de ella, es más probable que sea cierta.
¿La historia es una broma? A veces es difícil diferenciar las noticias falsas del humor o la sátira. Comprueba si la fuente de la noticia es conocida por sus parodias, y si los detalles y el tono de la historia sugieren que esta se ha escrito en clave de humor.
Algunas historias son falsas de forma intencionada. Mantén una actitud crítica cuando leas una historia y comparte solo las noticias que pienses que son creíbles.

# data[].attributes.attachment_link.readable_text
Online behavioral data, such as digital traces from social media, have the potential to allow researchers an unprecedented new window into human behavior in ecologically valid everyday contexts. However, research using such data is often purely observational, which limits its usefulness for identifying causal relationships. Here we review recent innovations in experimental approaches to studying online behavior, with a particular focus on research related to misinformation and political psychology. In hybrid lab-field studies, exposure to social-media content can be randomized, and the impact on attitudes and beliefs can be measured using surveys, or exposure to treatments can be randomized within survey experiments, and their impact on subsequent online behavior can be observed. In field experiments conducted on social media, randomized treatments can be administered directly to users in the online environment (e.g., via social-tie invitations, private messages, or public posts) without revealing that they are part of an experiment, and the effects on subsequent online behavior can then be observed. The strengths and weaknesses of each approach are discussed, along with practical advice and central ethical constraints on such studies.

References

Bakshy, E., Eckles, D., Bernstein, M. S. (2014). Designing and deploying online field experiments. In WWW '14: Proceedings of the 23rd International Conference on World Wide Web (pp. 283-292). Association for Computing Machinery.
Google Scholar

Desposato, S. (2015). Ethics and experiments: Problems and solutions for social scientists and policy professionals. Routledge.
Google Scholar

Guess, A. M. (2021). Experiments using social media data. In Druckman, J. N., Green, D. P. (Eds.), Advances in experimental political science (Vol. 184, pp. 184-198). Cambridge University Press. https://doi.org/10.1017/9781108777919.013
Google Scholar

Mosleh, M., Martel, C., Eckles, D., Rand, D. (2021a). Perverse downstream consequences of debunking: Being corrected by another user for posting false political news increases subsequent sharing of low quality, partisan, and toxic content in a Twitter field experiment. In CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Paper 182). Association for Computing Machinery. https://doi.org/10.1145/3411764.3445642
Google Scholar

Rosenzweig, L. R., Bergquist, P., Pham, K. H., Rampazzo, F., Mildenberger, M. (2020). Survey sampling in the Global South using Facebook advertisements. SocArXiv. https://doi.org/10.31235/osf.io/dka8f
Google Scholar

Taylor, S. J., Eckles, D. (2018). Randomized experiments to detect and estimate social influence in networks. In Lehmann, S., Ahn, Y.-Y. (Eds.), Complex spreading phenomena in social systems: Influence and contagion in real-world social networks (pp. 289-322). Springer.
Google Scholar

# data[].attributes.attachment_link.title
Field Experiments on Social Media - Mohsen Mosleh, Gordon Pennycook, David G. Rand, 2021

# data[].attributes.attachment_link.description
Online behavioral data, such as digital traces from social media, have the potential to allow researchers an unprecedented new window into human behavior in ecologically valid everyday contexts. However, research using such data is often purely observational, which limits its usefulness for identifying causal relationships.

# data[].attributes.body
1. Enganchar la atención de las personas participantes (sistema 1) en la comprensión de su riesgo a la manipulación digital, específicamente el riesgo a la desinformación.

2. Poner a prueba la Teoría de la Motivación a la Protección (TMP) en materia de manipulación digital: a mayor percepción de susceptibilidad, mayores posibilidades de tomar acción para disminuir los riesgos a los que se expone el individuo.

3. Poder detonar un comportamiento de protección en dónde las personas conozcan herramientas para disminuir su riesgo y efectivamente las apliquen en sus interacciones digitales.

# data[].attributes.body
1. Una de las tendencias que se encontraron es la de la prevención frente a la desinformación a través de la teoría de la inoculación de McGuire (1961) (Lewandowsky & Linden, 2021; Roozenbeek & van der Linden, 2019). Esta propone que los usuarios pueden eludir la influencia de mensajes persuasivos. Como criterio general, cuanto mayor es el conocimiento que el receptor tiene del tema y más articulado está el conocimiento (cuanta más fuerza tiene la “actitud”), más difícil es persuadirle o engañarle. En el caso de la desinformación, el “texto inoculante” necesita de dos elementos. Primero, incluir una advertencia explícita sobre el peligro de ser engañado por la desinformación. En segundo lugar, se necesita proporcionar contraargumentos que explican los fallos en esa desinformación.

2.     Creación de capacidades informacionales que le permitan a las personas en diferentes contextos identificar desinformación.

3.     Posibilidad de sostener en el tiempo los efectos de la intervención.