# data[].attributes.headline
Introducing the CRAAP Test - Information Literacy

# data[].attributes.subject
Introducing the CRAAP Test - Information Literacy

# data[].attributes.body
The CRAAP Test is one of the mostly widely used methods to evaluate especially internet resources for credibility and accuracy.  Is the CRAAP Test still a viable evaluative tool?  Explain your reasoning based on the resources provided.

# data[].attributes.headline
American Library Association Updates Its CRAAP Test for Spotting Fake News

# data[].attributes.subject
American Library Association Updates Its CRAAP Test for Spotting Fake News

# data[].attributes.attachment_link.readable_text
In an age when dubious "news" stories spread on social media like viruses, the importance of spotting bogus sources is more relevant than ever. But the American Library Association (ALA) has been using its own BS-detecting system for years, and its name is hard to forget.

The CRAAP test consists of five criteria: currency, relevance, authority, accuracy, and purpose [ PDF]. Sarah Blakeslee of the University of California at Chico's Meriam Library developed the standard as a way for students and teachers to evaluate sources with a skeptical eye. The tool is especially helpful when conducting research on the internet, where unsubstantiated blog posts show up in a Google search beside peer-reviewed studies.

@AbrahamsonLucy Is this a reliable source? Does it pass the CRAAP Test? pic.twitter.com/KasqAOGnfo

- Bob Seidel Jr. (@RSeidel_NBCT) February 9, 2017

But there's one portion of the test that's recently required a second look. According to The Huffington Post, the ALA has updated its guidelines so that the "authority" checkmark now calls for greater scrutiny. "We talk differently about authority [now]," ALA president Julie Todaro told The Huffington Post. "And we talk about credentials in a different way. We talk about going beyond a title that someone has."

That means not blindly trusting every story posted on social media by someone from a verified account. Look at the author of the article that's been shared, and the authors of the sources that article cites, if any. And remember that just because an author is educated and prolific that doesn't necessarily mean they're trustworthy. "Authority is contextual," a CRAAP guide put out by the Gumberg Library at Duquesne University states. "Having a Ph.D. in Astronomy would not give someone authority to write about the impact of music therapy on children who have autism. The expertise or experience needs to be relevant to the topic."

After gauging the validity of your source, do the same for the other four markers. If the information has authority but lacks currency, relevance, accuracy, or purpose, it's probably not worth citing in an academic essay (or tweeting out to your followers).

Subscribe to our Newsletter!

# data[].attributes.attachment_link.title
American Library Association Updates Its CRAAP Test for Spotting Fake News

# data[].attributes.attachment_link.description
In an age when dubious "news" stories spread on social media like viruses, the importance of spotting bogus sources is more relevant than ever. But the American Library Association (ALA) has been using its own BS-detecting system for years, and its name is hard to forget.

# data[].attributes.headline
Information Literacy Toolkit: Evaluating Resources: The CRAAP Test

# data[].attributes.subject
Information Literacy Toolkit: Evaluating Resources: The CRAAP Test

# data[].attributes.attachment_link.readable_text
How do you know if you have found "good" information? The CRAAP Test is a list of questions that you can use to evaluate the information that you find.

When was the information published or posted?

Has the information been revised or updated?

Is the information current or out-of-date for your topic?

Are the links functional?

elevance: the importance of the information for your needs

Does the information relate to your topic or answer your question?

Who is the intended audience?

Is the information at an appropriate level (i.e. not too elementary or advanced for your needs)?

Have you looked at a variety of sources before determining this is one you will use?

Would you be comfortable using this source for a research paper?

Who is the author/publisher/source/sponsor?

Are the author's credentials or organizational affiliations given?

What are the author's credentials or organizational affiliations?

What are the author's qualifications to write on the topic?

Is there contact information, such as a publisher or e-mail address?

Does the URL reveal anything about the author or source?
examples: .com (commercial), .edu (educational), .gov (U.S. government), .org (nonprofit organization), or .net (network)

ccuracy: the reliability, truthfulness, and correctness of the content

Where does the information come from?

Is the information supported by evidence?

Has the information been reviewed or refereed?

Can you verify any of the information in another source or from personal knowledge?

Does the language or tone seem biased and free of emotion?

Are there spelling, grammar, or other typographical errors?

What is the purpose of the information? to inform? teach? sell? entertain? persuade?

Do the authors/sponsors make their intentions or purpose clear?

Is the information fact? opinion? propaganda?

Does the point of view appear objective and impartial?

Are there political, ideological, cultural, religious, institutional, or personal biases?

# data[].attributes.attachment_link.title
LibGuides: Information Literacy Toolkit: Evaluating Resources: The CRAAP Test

# data[].attributes.attachment_link.description
How do you know if you have found "good" information? The CRAAP Test is a list of questions that you can use to evaluate the information that you find. When was the information published or posted? Has the information been revised or updated? Is the information current or out-of-date for your topic?

# data[].attributes.headline
Criteria for Evaluating Information: What to Think About

# data[].attributes.subject
Criteria for Evaluating Information: What to Think About

# data[].attributes.attachment_link.readable_text
Skip to main content

Questions to Ask

Authority / Credentials

Who is the author or creator?

What are the author's qualifications and credentials for writing about this subject?

Are they an expert in the field? How do you know?

How reputable is the publisher?

Are there organization affiliations? And are they reputable?

Does the information provide references or sources for data or quotations?

Goals and aims of the people or groups presenting material is often unclear

Web often functions as a "virtual soapbox"

Distinction between advertising and information is blurred on Web.

It is easier now more than ever for anyone to publish on the web. Scholars, too. So it becomes difficult to judge authors. See Using Blogs for more information.

Type of Source / Audience
Types of Information:
How to Determine Audience

Is the information academic, Trade or Professional, Substantive News, Popular, or Sensational?

Is it an encyclopedia or published through a museum?

Is the purpose to educate?

How do you know?

Is there a peer-review process, fact-checkers, or editors in place who review the material before publication.

More scholars are self publishing and it's harder to tell.

Just because someone has credentials doesn't mean they are experts on all subjects.

Relevancy

Is the information related to the topic you are researching?

You may find something to be interesting, but make sure it has a place in your main argument.

Point of View / Objectivity / Bias

Is the information presented with a minimum of bias?

Is it fact or opinion? Does the information reflect an author's bias?

Does the point of view appear objective and impartial?

Are there political, ideological, cultural, religious, institutional or personal biases?

To what extent is the information trying to sway the opinion of the audience?

Who is responsible for its dissemination?

What is the purpose of the information? Is it to inform, teach, sell, entertain or persuade?

Is the creator/author trying to sell you something?

Do the authors/sponsors make their intentions or purpose clear?

If the author has a position, is it well reasoned and argued and supported by empirical evidence?

If you are going to use a source that is biased, you want to make sure that the position reflected is supported by ample evidence.

You also want to acknowledge any bias in your paper or article.

Goals and aims of the people or groups presenting material is often unclear

Web often functions as a "virtual soapbox"

Distinction between advertising and information is blurred on Web

Currency

How recent is the information?

Is it current enough for your topic?

Has it been updated or revised?

Is the publication/copyright date clearly labeled?

Is the information up-to-date?

Dates not always included on Web pages

If included, a date may have various meanings:

1. Date information first written

2. Date information placed on Web

3. Date information last revised

Remember that just because the webpage has been updated recently, that doesn't mean that all of the information on the webpage has necessarily been updated.

Accuracy

Where does the information come from?

How reliable and free from error is the information? How do you know?

Were there editors and fact checkers?

Is the information supported by evidence?

Has the information been reviewed or refereed?

Can you verify the information in another source or from personal knowledge?

Is the content primarily opinion? Or is it balanced with multiple points of view?

Does the language or tone seem unbiased and free of emotion?

Anyone can publish on the Web

Web resources may not be verified by editors or checked for accuracy

No standards yet developed.

Web pages move. If you quote this source, will it be available later?

Web pages are susceptible to accidental and deliberate alteration.

Current information is more likely to be accurate, but not always. Even if it is current, be sure to cross check the information with other sources.

Content
Coverage/Scope

What topics are included in the work?

Is the scope broad or narrow? Are the topics included explored in depth?

Is the coverage adequate?

Are references to additional sources provided?

Are many sides of the topic covered?

Is there evidence to back up the arguments?

Be sure the source has the kind of coverage of the topic that you want before you decide to use it.

Web version may differ from print version of same title

Often hard to determine extent of Web coverage

Relevance

Does the information relate to your topic or answer your question?

Is it easy to navigate and read?

Who is the intended audience?

Is the information at an appropriate level (i.e. not too elementary or advanced for your needs)?

Have you looked at a variety of sources before determining this is one you will use?

Would you be comfortable using this source for a research paper?

Are the topics included explored in depth?

Is there a contents page, site map, navigation bar?

Is it easy to navigate and read?

Are special plug-ins required?

Is there a way to return to the "home page" to determine the source of the information?

Often hard to determine extent of Web coverage.

Usability

Is there a Table of Contents, Index, page numbers to guide you?

Is it easy to navigate and read?

Is there a contents page, site map, navigation bar?

Is it easy to navigate and read?

Are special plug-ins required?

Is there a way to return to the "home page" to determine the source of the information?

# data[].attributes.attachment_link.title
Otis College LibGuides: Criteria for Evaluating Information: What to Think About

# data[].attributes.attachment_link.description
Skip to main content Questions to Ask Authority / Credentials Who is the author or creator? What are the author's qualifications and credentials for writing about this subject? Are they an expert in the field? How do you know? How reputable is the publisher? Are there organization affiliations? And are they reputable?

# data[].attributes.headline
CITY BRIGHTS: HOWARD RHEINGOLDCrap Detection 101

# data[].attributes.subject
CITY BRIGHTS: HOWARD RHEINGOLDCrap Detection 101

# data[].attributes.attachment_link.readable_text
"Every man should have a built-in automatic crap detector operating inside him."
Ernest Hemingway, 1954
The answer to almost any question is available within seconds, courtesy of the invention that has altered how we discover knowledge - the search engine. Materializing answers from the air turns out to be the easy part - the part a machine can do. The real difficulty kicks in when you click down into your search results. At that point, it's up to you to sort the accurate bits from the misinfo, disinfo, spam, scams, urban legends, and hoaxes. "Crap detection," as Hemingway called it half a century ago, is more important than ever before, now that the automation of crapcasting has generated its own word: "spamming."

Unless a great many people learn the basics of online crap detection and begin applying their critical faculties en masse and very soon, I fear for the future of the Internet as a useful source of credible news, medical advice, financial information, educational resources, scholarly and scientific research. Some critics argue that a tsunami of hogwash has already rendered the Web useless. I disagree. We are indeed inundated by online noise pollution, but the problem is soluble. The good stuff is out there if you know how to find and verify it. Basic information literacy, widely distributed, is the best protection for the knowledge commons: A sufficient portion of critical consumers among the online population can become a strong defense against the noise-death of the Internet.

The first thing we all need to know about information online is how to detect crap, a technical term I use for information tainted by ignorance, inept communication, or deliberate deception. Learning to be a critical consumer of Webinfo is not rocket science. It's not even algebra. Becoming acquainted with the fundamentals of web credibility testing is easier than learning the multiplication tables. The hard part, as always, is the exercise of flabby think-for-yourself muscles.

The issue of info pollution has been on my mind since at least 1994, when I wrote "The Tragedy of the Electronic Commons" about the infamous Canter and Siegel - the first Internet spammers. A few years later, I personally confronted the importance of teaching information literacy to 14 year olds when I watched my daughter come of age at the same time online search engines became available. I sat down in front of the circa-1999 computer with my daughter and explained that most of the books she could get from the library could be counted on to be factually accurate. But when you enter words into a search engine, there is no guarantee that your search will lead you to accurate information. "You have to do some investigation before you accept anything you find online," I warned her.

"Ask a few questions and use available tools to see if you can find answers," is what I told her when she asked me how to go about investigating.

Today, just as it was back then, "Who is the author?" is the root question. If you don't find one, turn your skepticism meter to the top of the dial. And use easywhois.com to find out who owns the site if there is no author listed. If the author provides a way to ask questions, communicate, or add comments, turn up the credibility meter and dial back the skepticism. When you identify an author, search on the author's name in order to evaluate what others think of the author - and don't turn off your critical stance when you assess reputation. Who are these other people whose opinions you are trusting? Is the site a .gov or .edu? If so, turn up the credibility a notch. If it helps, envision actual meters and dials in your mind's eye - or a thermometer or speedometer. Take the website's design into account - professional design should not be seen as a certain indicator of accurate content, but visibly amateurish design is sometimes an indicator that the "Institute of Such-and-Such" might be an obsessive loner.

More good questions to use as credibility probes: Does the author provide sources for factual claims, and what happens when you search on the names of the authors of those sources? Have others linked to this page, and if so, who are they (use the search term "link: http://..." and Google shows you every link to a specified page). See if the source has been bookmarked on a social bookmarking service like Delicious or Diigo; although it shouldn't be treated as a completely trustworthy measurement, the number of people who bookmark a source can furnish clues to its credibility. All the mechanics of doing this kind of checking take only a few seconds of clicking, copying and pasting, searching, and judging for yourself. Again, the part that requires the most work is learning to do your own judging.

You aren't paranoid if you suspect that some sites might even deliberately try to deceive you. Some sites insidiously cloak their real bias, for example. I use martinlutherking.org as an example with my students today - it's not owned by admirers of the late civil rights leader, but you wouldn't know that at first glance. Another, less sinister but equally sobering teaching story: "The parody site Gatt.org once duped the Center for International Legal Studies into believing it was the Web site of the World Trade Organization. Accordingly, a few years ago, the association arranged for someone from the parody site to speak at its annual conference. The speaker - an imposter from an activist group known as the Yes Men - offended several attendees with racist remarks. A staged pie-throwing incident followed the presentation, and the fiasco later culminated in the faked death of the speaker/imposter." A good question to ask yourself, particularly when a website asks you to download something to your own computer, is "might somebody be trying to put one over on me?"

When I began teaching my daughter how to evaluate the credibility of web pages, I started collecting rules of thumb, strategies, tools - especially free and easy to use ones - for sorting the goodinfo from the badinfo. Fortunately, tools are far more powerful today than they were a decade ago; the bad news is that too many people don't know about them. In recent years, as so many more people have started to rely on the web for such vitally important forms of information as news, medical information, scholarly research, investment advice, the lack of general education in critical consumption of information found online is turning into a public danger. No, Bill Gates won't send you $5 for forwarding this chain e-mail, the medical advice you get in a chat room isn't necessarily better than what your doctor tells you, and the widow of the deceased African dictator is definitely not going to transfer millions of dollars to your bank account. That scurrilous rumor about the political candidate that never makes the mainstream media but circulates as email and blog posts probably isn't true. The data you are pasting into your memo or term paper may well be totally fabricated.

Use the following methods and tools to protect yourself from toxic badinfo. Use them, pass them along to others. Promote the notion that more info-literacy is a practical answer to growing info-pollution.

Although the Web undermines authority, the usefulness of authority as another clue to credibility hasn't entirely disappeared. I would add credibility points if a source is a verified professor at a known institution of higher learning, an authentic M.D. or Ph.D., but I wouldn't subtract points from uncredentialed people whose expertise seems authentic. Nor would I stop at simply verifying that the claim to be a professor is valid. The next step: use the scholarly productivity index that derives a score from the scholar's publications, citations by other scholars, grants, honors, and awards. If you want to get even more serious, download a free copy of Publish or Perish software, which analyzes scientific citations from Google Scholar according to multiple criteria. Again, don't trust just one source. Triangulate.

I got good strategy advice from John McManus, author of "Detecting Bull: How to Identify Bias and Junk Journalism in Print, Broadcast and on the Wild Web", who told me "you have think like a detective." Think of tools like search engines, the productivity index, hoax debunking sites like Snopes.com, and others I will mention later as forensic instruments, like Sherlock Holmes' magnifying glass or the crime scene investigator's fingerprint kit. The tools are only useful as the means to sleuthing out a mystery. In the case of people who stake their health on online medical information from a virtual community, their economic well-being on online financial information, their political liberty on the news they get from Twitter, blogs, or YouTube, the stakes in this detective game are high. Triangulation is what detectives do - try to find three different ways to test a source's credibility. For example, you could Google the author's name, enter the author's name in the scholarly productivity index, and use the literacy resources at factchecked.org to triangulate a source. (Factchecked.org's sister site, Factcheck.org researches claims from all political factions.)

Know how to use online filters. As more people get their news online, and more people at the site of newsworthy events have Web-linked cameras and video cameras, we'll see more situations like the 250,000 "tweets" per hour that passed through Twitter during the Iranian political demonstrations of June, 2009. Before Twitter came on the scene, online services like Flickr and YouTube enabled users to "tag" photographs and video with key words, making it possible to search for images tagged with those key words, revealing all the still images and videos coming in from amateur chroniclers during an event. The San Diego Union-Tribune called publicly for citizen reporters to use the same tags for their images of the 2007 wildfire. Using the search facility on Flickr or YouTube enables you to see a stream of images or videos, and automatically subscribing to that search through "RSS" means you can continue to see visual reports stream in as others upload them - in real time. At the height of the Iran demonstrations, CNN was displaying videos posted to YouTube, alerted via Twitter. Quite properly, CNN introduced the images with the disclaimer that they were as-yet unverified. As Clay Shirky has noted, we're in the age of "publish, then filter."

Again, it's up to the consumer of the information to decide which images, videos, tweets are authentic. As always happens when there is a high demand for separating signal from noise, people began to put together filters for doing that - and human tools for sorting the more trustworthy information. After the terrorist attacks in Mumbai provided both noteworthy on-the-scene reports and outright rumors, some experts started talking about "crowdsourcing the filter" by growing populations of trusted editors who would collectively identify the good stuff. Although they did not start as a filter for fast-breaking news, American Public Media's Public Insight Network is moving in the direction of a crowdsourced filter.

When the Iran demonstrations happened, people in Iran and around the world used the Twitter equivalent of a Flickr or YouTube tag, a now-famous "hashtag" - #iranelection - and for a few days flooded the world with riveting images that will probably win Pulitzers, along with shocking and politically inflammatory videos, torrents of contradictory reports, bogus rumors, apparent disinformation, both informed and ignorant political arguments and, as always, spam, porn, and porn spam. As the Iran events unfolded, Marc Ambinder wrote an astute article in The Atlantic.com, "Follow the developments in Iran like a CIA analyst." Just as thinking like a detective is a strategy for trying to determine the credibility of webinfo, thinking like an intelligence analyst is a strategy for trying to gauge the credibility of online reports about breaking news events. Ambinder recommends watching for disinformation, looking for patterns in the geographic location of sources (but warns against assuming that everything that resembles a pattern really is one), examining your assumptions and looking for sources that contradict them.

Twitter Journalism ("Where News and Tweets Converge"), published a series of steps to take to verify a tweet, including, among many other tips, checking the history of past tweets by a person to see what context you might find before the claim about a news event was tweeted, checking the bio of twitterer who makes a claim, being wary of news tweets from someone with very few previous tweets or who joined very recently, use Twitter's reply feature to engage the twitterer directly. I've been collecting links for journalism students who want to understand the journalistic requirements of good Twitter practice from the production side.

The biases of trusted sources like newspapers and television need to be examined critically, as well as those that come in from what are increasingly called "social media." Questioning Video is all about understanding the vocabulary of visual deception that can be used to distort television news. Newstrust is trying to crowdsource the filter for mainstream as well as alternative news sources by growing a bipartisan virtual community of critical news filterers who use the same set of criteria for evaluating whether a news story exhibits bias, makes factual claims that can or cannot be verified, presents more than one interpretation of events. Fairspin 's community votes on stories in order for the community's aggregate judgements to identify opinion disguised as fact and reflect the degree of political bias detected in stories from both the left and right. And on the cutting edge of community-based filtering tools, Intel labs' Dispute Finder Firefox Extension "highlights disputed claims on web pages you browse and shows you evidence for alternative points of view."

The good news about the pace of medical research is also the bad news - few medical specialists can keep up with the rate of new discoveries. That means that it's possible for the collective intelligence of a committed community - and there is nothing as committed as people who are suffering from a disease - to stay ahead of all but those most dedicated individual specialists. However, along with the latest word on cutting-edge drug trials are unsubstantiated claims, rumors, outright quackery. When it comes to medical information, just as when it comes to information that affects political liberty, believing or forwarding badinfo can be unhealthy or fatal. Again, the critical consumer of online medical advice has a number of triangulation tools at hand. For scientific articles, Science Direct has guest access. The Health on the Net Foundation has been a steady source of finding reliable/credible health information online. They even have a browser plug-in that enables you to check health information on any website against HON's database. An astute medical student wrote a guide to how to check quality of medical information online. How much work is it to check three links before believing or passing along health information you find online? Simply Googling the name of the Dr. who tried to sell do-it-yourself eye surgery kits, for example, immediately raises questions for those who are considering aiming lasers at their own retinas.

To me, the issue of information literacy could be even more important than the health or education of some individuals. Fundamental aspects of democracy, economic production, the discovery and use of knowledge might be at stake. Some of the biggest problems facing the world today seem to be far beyond the ability of any individual or community, or even the whole human race, to tackle. But the noise death of the Internet is something we can take on and win. Although large forces are at work, when it comes to the shape of online media, I believe that what people know - and how many people know - matters. Digital media and networked publics are only the infrastructure for participation - the cables and chips do no good unless people know how to use them. The collision of newly participative populations with authoritarian control is taking place every day in Teheran and Beijing, Berlin and Washington, D.C. Nobody knows how this clash will play out, but the one cost-effective measure that the participative have in their contest with central control is know-how. And the lack of know-how among the population is an asset to those who seek to put the lid back on their - our - power of expression.

Resources:

(Please suggest resources in comments, and I will add them below as I receive, find, and vet them)
Teachable moment: Crap Detecting Assessing the Credibility of Online Sources The CRAP Test Urban Legends Check quality of backlinks (links TO a site)
Newstrust guide to finding good journalism online
Museum of Hoaxes Urban Legends (About.com) Carl Sagan's "Baloney Detection Kit"

# data[].attributes.attachment_link.description
"Every man should have a built-in automatic crap detector operating inside him." Ernest Hemingway, 1954 The answer to almost any question is available within seconds, courtesy of the invention that has altered how we discover knowledge - the search engine. Materializing answers from the air turns out to be the easy part - the part a machine can do.

# data[].attributes.headline
Tips from the Experts What Does Bad Information Look Like? Using the CRAAP Test for Evaluating Substandard Resources

# data[].attributes.subject
Tips from the Experts What Does Bad Information Look Like? Using the CRAAP Test for Evaluating Substandard Resources

# data[].attributes.attachment_link.readable_text
What Does Bad Information Look Like? Using the CRAAP Test for Evaluating Substandard Resources

Abbey B. Lewis
STEM Learning and Collections Librarian
Norlin Library
University of Colorado Boulder
Boulder, Colorado
Abbey.B.Lewis@colorado.edu

Abstract

The CRAAP Test (Currency, Relevance, Authority, Accuracy, Purpose), is a popular tool among academic librarians for instructing students in resource evaluation. Its applicability to web-based information makes it particularly useful for students as they think about information they encounter both inside and outside of scholastic settings. However, while students may be capable of explaining concepts like bias, authority, and scholarly publications, when it comes to questions of reliability they may have limited experience. For instance, students may have no trouble recognizing high quality resources, but still have difficulty identifying those characteristics that signify an unreliable or substandard resource. In an undergraduate science writing course, the CRAAP Test was introduced to students, who then used the test in a group activity to evaluate a resource that failed key aspects of the test. This activity engages students in how to critically evaluate resources and provides useful feedback for instructors on those evaluative concepts that remain unclear.

Introduction

Resource evaluation is a key component of library instruction at many higher education institutions that seek to enhance students' information literacy skills. The CRAAP Test ( Blakeslee 2004), RADAR ( Mandalios 2013), and similar strategies provide a framework for teaching concepts that can be applied to both scholarly and popular resources to gauge their appropriateness and credibility. The Association of College & Research Libraries (ACRL) Board further underscored the importance of critical evaluation of resources in its "Framework for Information Literacy for Higher Education" with its in-depth discussion of concepts, such as authority, that information literate learners can recognize and articulate ( ACRL 2016).

However, successful identification of quality information can feel a bit like checking off boxes when an academically suitable resource is used for teaching purposes: here are the author's credentials, here is the title of a reputable publication, and here are the research claims along with data and citations from other reputable publications to support those claims, and so on. Are students really learning to think critically about resources when the resources don't have any identifiable flaws?

Class Activity

The CRAAP Test was introduced to students in an undergraduate science writing course during day two of a two-day library instruction session. Students were somewhat familiar with resource evaluation concepts like bias, credibility, and peer review from previous discussions in the class. At the beginning of the activity, the CRAAP Test was presented as a tool for students to assess the quality of information they were considering for use in an academic assignment. Students were then placed in five groups with each group assigned to answer the questions from either the Currency, Relevance, Authority, Accuracy, or Purpose sections of the CRAAP Test and asked to share their observations via Padlet (Figure 1).

Figure 1

The resource that students were asked to evaluate was an article from a blog that dealt with the topic of climate change from a viewpoint of heavy skepticism ( CO2 is Life 2017). While the article could pass some aspects of the CRAAP Test (it was posted fairly recently and the topic of the given article could potentially be of interest to a student researching the scientific causes of climate change), it failed the test in others, often spectacularly so. The article showed evidence of political and ideological bias, no authors or credentials were given (although the blog itself claimed to be "the definitive source for exposing the global warming hoax"), and graphs were presented without any indication of where the data came from and with only tangential ties to the accompanying text (Figure 2 and Figure 3). Students were not informed of the article's shortcomings, but simply instructed to use the article to answer the questions in the CRAAP Test.

Figure 2

Figure 3

Observations

While students were overall successful in applying the CRAAP Test, some points of confusion emerged for each group, which in turn led to productive teaching points.

The group assigned to the Currency questions was confident in their answers overall, but had some difficulty deciding how recent information needed to be.

The Relevance group initially assumed that the article was meant for a scholarly audience because it contained graphs.

The Authority group was unsettled because no information on the authors or their affiliations was provided. They believed they were missing something obvious.

The Accuracy group expressed difficulty in determining if the article had undergone peer review, although they were able to explain the peer review process.

The group answering the Purpose questions felt that the article had a heavy bias, but needed some assistance in identifying evidence that was indicative of this condition.

Best Practices for This Activity

Prior to conducting this activity, students should be introduced to concepts like peer review, scholarly vs. popular literature, supporting research claims with evidence, forms of bias, and the ways in which authority is constructed and displayed in different types of literature. With this foundation, the activity then allows students to apply these concepts and serves as a type of assessment tool through which instructors can gauge students' understanding of resource evaluation. Some questions in the CRAAP test have ambiguous answers (Would you be comfortable citing this source in your research paper? Is the author qualified to write on the topic?) and students benefit from having one or more instructors circulating during the activity and helping them form more complete responses. Students should also be provided with a hypothetical topic to ensure that all questions within the test can be answered. Finally, students should be given enough time to read the resource in its entirety and examine peripheral information, such as links, URLs, and the larger context of where, how, and why the information was produced.

Conclusion

While it's unlikely that students will encounter information that fails so many aspects of the CRAAP test in a library database, it's also inevitable that they'll come across questionable resources in their personal, academic, or professional lives. Even within the realm of peer-reviewed literature, students still need to be able to think critically about the quality of information they find. The CRAAP Test offers an excellent framework for students to evaluate resources, but applying that framework to a flawed resource can provide students with the ability to recognize problematic information when they see it again. Future directions for this activity could include a structured assessment of student learning, conducting the activity with a different resource evaluation tool, or using a resource with less obvious flaws for upper division or graduate courses.

References

ACRL (Association of College and Research Libraries). 2016. Framework for information literacy for higher education. [Internet]. [Cited 2017 Dec 22]. Available from: http://www.ala.org/acrl/standards/ilframework

Blakeslee, S. 2004. The CRAAP test. LOEX Quarterly. [Internet]. [Cited 2017 Dec 22];31(3):6-7. Available from: http://commons.emich.edu/cgi/viewcontent.cgi?article=1009&context=loexquarterly

2017. Climate "science" on trial; the forensic files: exhibit J [Internet]. [Cited 2017 Dec 22]. Available from: https://co2islife.wordpress.com/2017/01/17/climate-science-on-trial-the-forensic-files-exhibit-j/

Mandalios, J. 2013. RADAR: An approach for helping students evaluate internet sources. Journal of Information Science 39(4):470-478. DOI: 10.1177/0165551513478889

This work is licensed under a Creative Commons Attribution 4.0 International License.

# data[].attributes.attachment_link.title
What Does Bad Information Look Like? Using the CRAAP Test for Evaluating Substandard Resources

# data[].attributes.attachment_link.description
Abbey B. Lewis STEM Learning and Collections Librarian Norlin Library University of Colorado Boulder Boulder, Colorado Abbey.B.Lewis@colorado.edu The CRAAP Test (Currency, Relevance, Authority, Accuracy, Purpose), is a popular tool among academic librarians for instructing students in resource evaluation.

# data[].attributes.headline
Show Me the Learning:Navigating Information Literacy through              Multiple Life Perspectives

# data[].attributes.subject
Show Me the Learning:Navigating Information Literacy through              Multiple Life Perspectives

# data[].attributes.headline
Fake News and Alternative Facts: A Guide to News Literacy: Evaluate Sources

# data[].attributes.subject
Fake News and Alternative Facts: A Guide to News Literacy: Evaluate Sources

# data[].attributes.attachment_link.readable_text
News sources need to be evaluated for purpose. A blog or letter to the editor is, by intent, an opinion and should not be given the same weight of authority as a news report. Similarly, parody and humor are intended as entertainment, not sources of factual analysis. Opinion, parody, and humor, even when presented by a respected news source, are not intended to be "real news" and should not be taken as such.

# data[].attributes.attachment_link.title
LibGuides: Fake News and Alternative Facts: A Guide to News Literacy: Evaluate Sources

# data[].attributes.attachment_link.description
News sources need to be evaluated for purpose. A blog or letter to the editor is, by intent, an opinion and should not be given the same weight of authority as a news report. Similarly, parody and humor are intended as entertainment, not sources of factual analysis.

# data[].attributes.attachment_link.readable_text
07_Fielding

Getting students thinking like fact-checkers in evaluating web sources

Jennifer A. Fielding is coordinator of library services at Northern Essex Community College-Lawrence Campus, email: jfielding@necc.mass.edu

For over two decades, librarians have been at the forefront of helping their patrons and students discern what online information is reliable, and what may be biased or outright false. Particularly as more formal information literacy programs developed at the college and university level (and the attendant inclusion of information literacy in many general education programs), academic librarians have developed curricula and taught students how to evaluate web sources for credibility. In many institutions, this has frequently been achieved via a "one-shot" session with a checklist of sorts, often some variation of the CRAAP Method (Currency, Reliability, Authority, Accuracy, and Purpose) developed nearly 15 years ago at California State University-Chico.1

The CRAAP method encourages the user to perform an in-depth analysis of the website to determine its credibility, often by finding and analyzing the "About Us" section, and thoroughly exploring the site to determine if there are named authors, what their credentials may be, and the stated purpose or mission of the publishing organization. Additional CRAAP assessments include how recently the site has been updated, whether page links are working and lead to other reliable information, and whether the site is a commercial, nonprofit, or educational site. Students focus on the site itself, performing a "deep dive" into what they find at a particular URL. As currently employed, the CRAAP method does not explicitly encourage leaving the site to place any content found there in a wider context.

However, in recent years, the dissemination of mis- and disinformation online has become increasingly sophisticated and prolific, so restricting analysis to a single website's content without understanding how that site relates to a wider scope now has the potential to facilitate the acceptance of misinformation as fact. Once a site is deemed "credible," all information on it is frequently trusted and taken at face value. This is clearly problematic, since studies show that once information has been accepted as valid and assimilated by the user, it becomes far more difficult to counter, even with accurate facts.2,3,4

In addition, considering the disciplinary shift away from static Information Competency Standards for Higher Education to the more organic and contextual approach of the Framework for Information Literacy for Higher Education, it would seem apparent that the pedagogy surrounding the critical assessment of web sources must also evolve.

The Stanford Study

A 2017 Stanford working paper by Sam Wineburg and Sara McGrew highlights this evolution in stark relief, assessing the critical evaluation skills for web content between students, faculty, and professional fact-checkers. They found that faculty (arguably information-savvy critical thinkers) performed barely better than undergraduates in assessing the credibility of web content, primarily because they used the deep-dive type of assessment endorsed by methods like CRAAP-thoroughly examining the site itself.

Fact-checkers, on the other hand, almost immediately began an independent verification process, a strategy the researchers dubbed "lateral reading"-opening multiple tabs, and searching for independent information on the publishing organization, funding sources, and other factors that might indicate the reliability and perspective of the site and its authors or sponsors.

This lateral reading approach produced significantly better results for the fact-checkers-both in critical assessment and in the speed of their conclusions-than the "vertical reading" deep dive did for both the student group and the faculty group.5

Informal trials with lateral searching

In an effort to trial and potentially incorporate this new research finding in teaching methods, a group of community college librarians revised their basic one-shot content to include a "lateral reading" assessment with their first-year students.

Northern Essex Community College (NECC) is a two-year associates degree- granting institution 30 miles north of Boston, Massachusetts. Almost all students in every program take an English 101 and/or English 102 course, during which research and information literacy skills are frequently addressed in both the content and/or with a "one-shot" library visit. In addition, as part of the college's Core Academic Skills requirements, all students graduating with an associate's degree must also complete a course designated as "intensive" in information literacy within their discipline.

In the Fall 2018 semester, several NECC librarians adapted the portion of some one-shot sessions where the CRAAP method would have been used to assess the credibility of a website. In one example of this alternative "lateral reading" activity, students analyzed two results from a Google search on "asthma" and compared sites for two of the top ten results: asthma.com versus medlineplus.gov/asthma.html.

While the instructor did note the difference in the domain names as a likely indicator of the purpose of each site (part of the CRAAP method), students were then encouraged to open new tabs and search for information on both GSK (which students had identified as the publisher of asthma.com) and Medline. Students quickly discovered that GSK is the pharmaceutical company GlaxoSmithKline, while the publisher of Medline is the National Library of Medicine.

Students then skimmed information on both of these organizations on sites like Wikipedia and news sites, and were asked for their assessment on which would deliver more trustworthy information regarding asthma diagnosis, prognosis, and treatments. Frequently at this point, students would spontaneously start discussing the information they found regarding GSK's legal battles, and the ethical implications of drug companies giving health advice, which then often led to animated discussions regarding the responsibilities of information creators.

The simple shift to a lateral reading method not only visibly engaged students more thoroughly in the process, but also directly applied several of the ACRL Framework for Information Literacy for Higher Education principles: Searching as Strategic Exploration, Information Has Value, and Authority is Constructed and Contextual.

In-class discussions around these concepts were often robust and student-driven, and frequently led to additional explorations of various websites. One student remarked they saw the activity like "detective work" and enjoyed that aspect of it. While this feedback is certainly anecdotal, it demonstrates the difference in framing that helped engage students more actively in the evaluation process.

Future classroom exercises are anticipated to focus on specific outcomes and behaviors articulated in the Framework, particularly exhibiting "mental flexibility and creativity" and developing an "awareness of the importance of assessing content with a skeptical stance and with a self-awareness of their own biases and worldview."6

Since these early, unstructured efforts have been so promising, discussions by the librarians around next steps in applying these pedagogical changes have been focused on adapting the approach to differing disciplinary perspectives across the curriculum (i.e., incorporating student activities evaluating sites specific to history, psychology, criminal justice, etc.), and development of an assessment method to verify lateral searching's efficacy versus traditional methods for NECC's students. Should the shift be incorporated into ongoing teaching, communication with faculty will also be important, as (ironically due to the diligent efforts of librarians) many faculty use CRAAP as the website assessment "standard" in their courses (librarians frequently receive requests from faculty to "teach the session on CRAAP").

It is important to note here, however, that the Stanford study attributed the success of the fact-checkers to both the lateral reading strategy described above, but also their "robust knowledge of sources to inform their decisions,"7 for instance, understanding that a nonprofit site does not necessarily connote altruism, and that purported news sites can lean heavily left or right in their reporting. "Fact checkers also possessed knowledge of online structures, particularly how search results are organized and presented. They knew that the first result was not necessarily the most authoritative."8

These findings indicate that any strategies taught regarding information evaluation must also be paired with content on search engine ranking, personalization, and Eli Pariser's now well-known filter bubble effect.9

Conclusions

It is widely acknowledged that the current information landscape places an increasing burden on the information consumer. The lack of editorial control in a web environment, coupled with personalized search engine results and filter bubbles of mis/disformation on social media and other platforms10,11,12 makes obvious the need for librarians to evolve our pedagogy to teach and encourage lateral, fact-checking behaviors and dispositions.

As such, while very useful for many early manifestations of web content, I would argue that the CRAAP "deep-dive" examination of a specific web source is no longer wholly adequate in light of the increasing sophistication of the web, nonexistent barriers to content creation, and the muddling effect of social media on information consumption and sharing. While it is clear that each of CRAAP's individual assessments have ongoing value, it has become vitally important to place information into a wider context to adequately evaluate its credibility, as well as teach how information is ranked and presented on search engines and social media. As consumers are able to both curate their information to suit their interests and propagate information nearly instantaneously, evolving information literacy instruction across the curriculum has broad implications not just for the research process, but for issues such as citizenship, democracy, and social responsibility.

Notes

1. Sarah Blakeslee, "The CRAAP Test," LOEX Quarterly 31, no. 3 (2004).

2. John Cook and Stephan Lewandowsky, The Debunking Handbook (St. Lucia, Australia: University of Queensland, 2011), http://sks.to/debunk.

3. Elizabeth Kolbert, "Why Facts Don't Change Our Minds," The New Yorker (February 27, 2017).

4. Alice Marwick and Rebecca Lewis, "Media Manipulation and Disinformation Online," Data & Society (May 15, 2017), https://datasociety.net/output/media-manipulation-and-disinfo-online/.

5. Sam Wineburg and Sarah McGrew, "Lateral Reading: Reading Less and Learning More When Evaluating Digital Information," Stanford History Education Group-Working Paper (September 2017), https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3048994.

6. ACRL "Framework for Information Literacy for Higher Education," 2015, www.ala.org/acrl/standards/ilframework.

7. Wineburg and McGrew, "Lateral Reading."

8. Ibid.

9. Eli Pariser, The Filter Bubble: What the Internet is Hiding From You (New York: Penguin Press, 2011).

10. Ibid.

11. Amy Mitchell, Elisa Shearer, Jeffrey Gottfried, and Michael Barthel, "Pathways to News," Pew Research Center, July 7, 2016, www.journalism.org/2016/07/07/pathways-to-news/.

12. Chris Meserole, "How Misinformation Spreads on Social Media-And What To Do About It," The Brookings Institution (May 9, 2018), https://www.brookings.edu/blog/order-from-chaos/2018/05/09/how-misinformation-spreads-on-social-media-and-what-to-do-about-it/.

# data[].attributes.attachment_link.description
07_Fielding Getting students thinking like fact-checkers in evaluating web sources Jennifer A. Fielding is coordinator of library services at Northern Essex Community College-Lawrence Campus, email: jfielding@necc.mass.edu For over two decades, librarians have been at the forefront of helping their patrons and students discern what online information is reliable, and what may be biased or outright false.

# data[].attributes.headline
Can you recognize misleading information?

# data[].attributes.subject
Can you recognize misleading information?

# data[].attributes.attachment_link.readable_text
Misleading or false information abounds and can catch you unawares because you find it in places you visit all the time like Facebook, Google, Twitter, or other sources. In fact, you may get this information from sources you trust.

Take a look at this chart from the EAVI (European Associations Viewers Interests) and the Center for Media Literacy to see how many of these types of misleading news you recognize.

Can You Recognize These Types of Misleading Information?

Click on the image to launch a bigger version of the chart

# data[].attributes.attachment_link.title
Research Guides: Evaluate Your Sources Using the CRAAP Test: Fake or Misleading News

# data[].attributes.attachment_link.description
Misleading or false information abounds and can catch you unawares because you find it in places you visit all the time like Facebook, Google, Twitter, or other sources. In fact, you may get this information from sources you trust.

# data[].attributes.attachment_link.readable_text
The articles below our sources we had our audience use to implement the C.R.A.A.P. test. Some are examples of sources that failed the C.R.A.A.P. test and should not be considered credible news.

# data[].attributes.attachment_link.title
DSC Library: Fighting Fake With C.R.A.A.P. Presentation: C.R.A.A.P. Test Activity

# data[].attributes.attachment_link.description
The articles below our sources we had our audience use to implement the C.R.A.A.P. test. Some are examples of sources that failed the C.R.A.A.P. test and should not be considered credible news.

# data[].attributes.body
Introducing Students to Information Literacy

# data[].attributes.attachment_link.readable_text
1. Ask students to find several scholarly sources on the same topic that take very different stands. How was it that the authors came to different conclusions? Does it have to do with authority?

2. Have students look at a blog, a video on YouTube, a collection of tweets, or some other type of social media regarding a contemporary event (e.g. demonstrations at Tahrir Square during the "Arab Spring" events). Ask them to describe how they would analyze and evaluate the authority of the author(s) of the information. Are there ways to determine whether the individual was an actual witness or participant in the events? Are there ways to identify whether the individual or group that developed a collection of information has a particular political bias? Can they determine whether the author(s) has a particular status within the group s/he represents or is the individual reporting as an "average citizen"?

3. Assign students a scholar/researcher in the field. Ask students to explore that person's career and ideas by locating biographical information, preparing a bibliography of the scholar's writings, analyzing the reaction of the scholarly community to the researcher's work, and examining the scholarly network in which the scholar works.

(From D. Leonard Corgan Library, King's College. " Term Paper Alternatives: Ideas for Information Based Assignments")

4. Identify significant people in a discipline. Consult a variety of biographical resources and subject encyclopedias to gain a broader appreciation for the context in which important accomplishments were achieved.

# data[].attributes.attachment_link.title
LibGuides: Introducing Students to Information Literacy: Authority Is Contextual

# data[].attributes.attachment_link.description
1. Ask students to find several scholarly sources on the same topic that take very different stands. How was it that the authors came to different conclusions? Does it have to do with authority? 2. Have students look at a blog, a video on YouTube, a collection of tweets, or some other type of social media regarding a contemporary event (e.g.

# data[].attributes.body
IV. CURATE Make meaning for oneself and others by collecting, organizing, and sharing resources of personal relevance.
Learners act on an information need by: 1. Determining the need to gather information. 2. Identifying possible sources of information. 3. Making critical choices about information sources to use.
Learners gather information appropriate to the task by: 1. Seeking a variety of sources. 2. Collecting information representing diverse perspectives. 3. Systematically questioning and assessing the validity and accuracy of information. 4. Organizing information by priority, topic, or other systematic scheme.
Learners select and organize information for a variety of audiences by: 1. Performing ongoing analysis of and reflection on the quality, usefulness, and accuracy of curated resources. 2. Integrating and depicting in a conceptual knowledge network their understanding gained from resources. 3. Openly communicating curation processes for others to use, interpret, and validate.