# data[].attributes.body
for Reading, see p. 4

# data[].attributes.headline
Evidence Protocol for Reading Interventions

# data[].attributes.subject
Evidence Protocol for Reading Interventions

# data[].attributes.attachment_link.readable_text
Glossary

The What Works Clearinghouse (WWC) seeks to provide users with the information they need to be effective consumers of education research. Please use our glossary to better understand the terms used by the WWC.

# data[].attributes.attachment_link.description
As an initiative of the U.S. Department of Education's Institute of Education Sciences (IES), the What Works Clearinghouse (WWC) was created in 2002 to be a central and trusted source of scientific evidence for what works in education.

# data[].attributes.headline
Making Sense of Terms Used to Assess Early Reading

# data[].attributes.subject
Making Sense of Terms Used to Assess Early Reading

# data[].attributes.body
IES/REL Southeast, March 2020

# data[].attributes.attachment_link.readable_text
March 2020

As students learn to read, educators and parents want to know if they are on track to become successful readers. Assessments can help answer this question but a plethora of terms-formative, benchmark, screener, diagnostic, progress monitoring, and outcome-confuse educators and parents and often lead to over testing. The solution is a streamlined framework, where the purpose for each assessment is clearly defined.
A Streamlined Assessment System
In a recent infographic, the Regional Educational Laboratory (REL) Southeast laid out a streamlined assessment system for early reading assessment. The streamlined system contains four steps:

Screening: The purpose of screening is to identify the probability of risk or success in reading achievement.

Diagnosis: The purpose of diagnostic tasks is to identify the need for instructional support in specific skill areas.

Progress monitoring: The purpose of progress monitoring, also called interim or formative assessment, is to determine whether students are learning the skills taught and/or meeting benchmarks.

Summative: The purpose of summative or outcome assessment is to measure whether students are making significant gains or losses in reading performance.

Screening, diagnosis, and progress monitoring can be combined within one assessment system to streamline these steps. The screen can identify who is at risk of reading problems and needs diagnostic tasks to set instructional objectives. Progress monitoring can be administered several times across the school year to ascertain whether instructional goals are being met and growth towards outcome performance is occurring. The summative or outcome assessment can be a standardized reading achievement test based on national norms or a criterion-referenced test based on state standards.
What are Formative Assessments?
Where do formative assessments fit into this assessment system? As originally conceived, formatives reflected the learning progressions that students demonstrated as they acquired skill in a domain, such as literacy (Heritage, 2010). Teachers can informally assess whether children are learning what they have been taught and can diagnose, reteach, and assess progress across the year. In this sense, formatives consist of teacher observations, quizzes, or curriculum-based tests such as end-of-unit tests whose ratings, or scores can serve to inform classroom instruction.

However, the meaning of formatives has shifted with the emphasis on students being proficient on the state's standards-based test. For many educators, formatives have become synonymous with benchmark assessments that indicate whether children are proficient on state standards. For example, if a grade 1 reading standard is to decode words with inflectional endings, a benchmark assessment item might be to read the word jumped. Yet, just because the student correctly reads the word does not guarantee later proficiency on the benchmark test. First, typically more than 10 items are needed to ensure reliable scores. Second, item-level mastery does not ensure proficiency on the standard. Being able to read jumped does not mean that a child will be able to read stopped, created, or running. Knowledge of spelling rules, more complex sound-spelling patterns, and other inflectional endings is required to increase the probability of being proficient on the standard on the outcome assessment. Finally, scoring proficient or not on a benchmark test does not provide instructional guidance for students far below grade-level standards, as is often the case for English learner students or students with disabilities, or for students performing above grade level. Using formative assessments that can adapt to a wide ability range are best for those populations of students (Foorman, Espinosa, Jackson, & Wu, 2016). Thus, formative assessments best fit into the progress monitoring component of a streamlined system, if they tap the skills the teacher is teaching for the diverse ability range of students in the classroom.
The Importance of Valid Measures
A streamlined assessment system must be comprised of valid measures if it is to accurately predict reading proficiency. The constructs that predict reading success are well known (Castles, Rastle, & Nation, 2018): phonological awareness, letter-name and sound knowledge, efficient decoding and encoding, vocabulary and grammatical knowledge, and comprehension strategies. The instructional practices for teaching children these constructs are also well known (Foorman, Beyler, et al., 2016). The challenge in an assessment system is to cover these critical constructs well and with sufficient depth that they predict to the ever-increasing complexity of text used to measure reading comprehension.
The Leap from Understanding Terms to Effectively Implementing Early Reading Assessments
Effective implementation of an early reading assessment system requires â€œbuy-inâ€ from administrators and teachers. In addition, extensive professional development in how to administer the assessments and how to translate data to instruction is needed. Furthermore, district and school leaders must consider the resources available to intervene with students identified as at-risk of a reading problem. In some schools the majority of students may score below the 30th percentile on the screen and, therefore, be eligible for reading intervention beyond the small-group instructional support provided in the classroom. Such situations signal a clear mandate to provide more explicit and systematic reading instruction in the classroom to reduce the numbers of students needing pullout intervention. p>

References

Castles, A., Rastle, K., & Nation, K. (2018). Ending the reading wars: Reading acquisition from novice to expert. Psychological Science in the Public Interest, 19(1), 5-51. https://doi.org/10.1177/1529100618772271

Foorman, B., Beyler, N., Borradaile, K., Coyne, M., Denton, C., Dimino, J., Furgeson, J., Hayes, L., Henke, J., Justice, L., Keating, B., Lewis, W., Sattar, S., Streke, A., Wagner, R., & Wissel, S. (2016). Foundational skills to support reading for understanding in kindergarten through 3rd grade (NCEE 2016-4008). Washington, DC: National Center for Education Evaluation and Regional Assistance (NCEE), Institute of Education Sciences, U.S. Department of Education. Retrieved from http://ies.ed.gov/ncee/edlabs

Foorman, B., Espinosa, A., Jackson, C., Wu, Y. (2016). Using computer-adaptive literacy assessments to monitor the progress of English language learner students. (REL 2016-149). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southeast. Retrieved from http://ies.ed.gov/ncee/edlabs

Heritage, M. (2010). Formative assessment: Making it happen in the classroom. Thousand Oaks, CA: Corwin Press.

# data[].attributes.attachment_link.keywords[]
educational improvement; excellence in education; educational research; statistics; statistical data; statistical surveys; national center for education evaluation and regional assistance.

# data[].attributes.attachment_link.title
Making Sense of Terms Used in Early Reading Assessment

# data[].attributes.attachment_link.description
The National Center for Education Evaluation and Regional Assistance (NCEE) conducts unbiased large-scale evaluations of education programs and practices supported by federal funds, such as Reading First and Title I of the Elementary and Secondary Education Act.

# data[].attributes.body
Making Sense: 2.2021: Shanahan

# data[].attributes.attachment_link.readable_text
Blast from the Past: Originally posted on November 3, 2018; re-posted on January 30, 2021. In this "science of reading" era, what it takes to make sense of reading research is even more timely today than 2 years ago when this entry first posted. Lots of people are hot on the idea of reading instruction based upon science, but how that should work and what we need to do to determine what the science of reading is seem to be big questions for a lot of people. There's much more to say about this topic, so I will try to add a new entry on it sometime soon. Until then, please don't make these common mistakes.
I talk a lot about research in this space.

I argue for research-based instruction and policy.

I point out a dearth of empirical evidence behind some instructional schemes, and champion others that have been validated or verified to my satisfaction.

Some readers are happy to find out what is "known," and others see me as a killjoy because the research findings don't match well with what they claim to "know."

Members of this latter group are often horrified by my conclusions. They often are certain that I'm wrong because they read a book for teachers that had lots of impressive citations that seem contradict my claims.

What is clear from these exchanges is that many educators don't know what research is, why we should rely on it, or how to interpret research findings.

Research is used to try to answer a question, solve a problem, or figure something out. It requires the systematic and formal collection and analysis of empirical data. Research can never prove something with 100 percent certainty, but it can reduce our uncertainty.

"Systematic and formal" means that there are rules or conventions for how data in a research study need to be handled; the rigor of these methods is what make the data trustworthy and allow the research to reduce our uncertainty. Thus, if a researcher wants to compare the effectiveness of two instructional approaches, he or she has to make sure the groups to be taught with these approaches be equivalent at the beginning. Likewise, we are more likely to trust a survey that defines its terms, or an anthropological study that immerses the observer in the environment for a long period of time.

Research reports don't just provide the results or outcomes of an investigation, but they explain-usually in great detail-the methods used to arrive at those results. Most people don't find research reports very interesting because of this kind of detail, but it is that detail that allows us to determine how much weight to place on a study.

Given all of that, here are some guidelines to remember.
1. Just because something is written, doesn't make it research.
Many practitioners think that if an idea is in a book or magazine that it is research. Some even think my blog is research. It is not, and neither is the typical Reading Teacher article or Heinemann book.

That's not a comment on their quality or value, but a recognition of what such writing can provide. In some cases, as with my blog, there is a serious effort to summarize research findings accurately. I work hard trying to distinguish my opinions from actual research findings (wish that was a common practice).

Many publications for teachers are no more than compendia of opinions or personal experiences, which is fine. However, these have all of the limits of that kind of thing.

Just because someone likes what they're doing (e.g., teaching, investing, cooking) and then writes about how well they've done it... doesn't necessarily mean it is really so great. That's why 82% of people believe that they're in the top 30% of drivers; something that obviously can't be right.

As human beings we fall prey to overconfidence, selective memory, and just a plain lack of systematicity in how we gain information about our success.

Often when teachers tell me that kids now love reading as a result of how they teach, I ask how do you know? What evidence do you have? Usually the answer is something like, "A parent told me that their child now likes to read." Of course, that doesn't tell how the other 25 kids are doing, or whether the parent is a good observer of such things, or even the motivation for the, seemingly, offhand comment (that could be aimed at buttering up an average teacher).

Even when you're correct about things improving, it's impossible-from personal experience alone-to know the source of the success. It could be the teaching method, or maybe just the force of your personality. If another teacher adopted your methods, things might not be so magical.

And, then there is opportunity cost. We all struggle with this one. No matter how good an outcome, I can't possibly know how well things might have gone had I done it differently. The roads not traveled may have gotten me someplace less positive-but not necessarily. You simply can't know.

That's where research comes in... it allows us to avoid overconfidence, selective memory, lack of systematicity, lack of reliable evidence, incorrect causal attribution, and the narrowness of individual experience.

Research requires the systematic collection and analysis of data. Research is disciplined or rule-based; that means those data have to be collected an analyzed in ways that have been painstakingly worked out over a long period of time by the research community. Following those rules of practice reduce the chances of error or invalid conclusions. That's why it is so important that research be "refereed." That means that other scientists, with no interest in whether a paper is published or not review the methods and results to determine their value -- without that approval it isn't supposed to be published. (Only one person gets to determine what I publish here and I never provide original analysis of data).

2. Research should not be used selectively.

Many educators use research the same way advertisers and politicians do-selectively, to support their beliefs or claims-rather than trying to figure out how things work or how they could be made to work better.

I wish I had a doughnut for every time a school official has asked me to identify research that could be used to support their new policy! They know what they want to do and want research to sell it. Rather than studying the research to determine what they should do.

Cherry-picking an aberrant study outcome that matches one's claims or ignoring a rigorously designed study in favor of one with a preferred outcome may be acceptable debater's tricks but are bad science. And, they can only lead to bad instructional practice.

When it comes to determining what research means, you must pay attention not just to results that you like. Research is at its best when it challenges us to see things differently.

I vividly remember early in my career when Scott Paris challenged our colleagues to wonder why DISTAR, a scripted teaching approach was so effective, despite that fact that most of us despised it. Clearly, we were missing something; our theories were so strong that they were blinding us to the fact that what we didn't like was positive for kids-at least for some kids or under some conditions (the kinds of things that personal experience can't reveal).

These days there are so many claims about reading -- some supported by extensive amounts of data, others with no proof at all. One example: research overwhelmingly supports the idea of teaching phonics and yet there is no evidence supporting the idea of teaching "decodable text," as that term is defined today. You can't tell that's the case, since many pro-phonics advocates make no distinction between the two claims-- they tell me their support of decodable text is "logical," but they sure don't tell their audiences that the effectiveness of phonics has been proven through research (part of the science of reading) and that the effectiveness of decodable text is a matter of belief and logic without direct empirical evidence (a valuable hypothesis, perhaps, but not part of the science of read -- at least not yet).

3. Research, and the interpretation of research, require consistency.

Admittedly, interpreting research studies is as much an art as science. During the nearly 50 years of my professional career, the interpretation of research has changed dramatically.

It used to be entirely up to the discretion of each individual researcher as to which studies they'd include in a review and what criteria they would use to weigh these studies.

That led to some pretty funky science: research syntheses that identified only studies that supported a particular teaching method or inconsistent criteria for impeaching studies (this study should be ignored because it has a serious design weakness, but then using studies with more acceptable findings even though they suffer the same flaw).

I've been running into this problem a lot lately. Not among researchers, but among practitioners. When I point out a research-supported instructional practice (Reading Recovery) that is inconsistent with phonics theories, I'm told "anything works if it is taught one-on-one." That sounds great, but those same people are offended when there is insufficient attention to phonics instruction, in spite of the evidence supporting phonics such as the National Reading Panel. The problem with this: the instruction in many of those positive phonics studies was delivered one-on-one.

I'm persuaded that both phonics and Reading Recovery work (because they both have multiple studies of sufficient quality showing their effectiveness). That doesn't mean I think they work equally well, or that they are equally efficient, or that they even accomplish the same things for students.

I agree with those who argue against teaching cueing systems, because research evidence reveals that poor readers use non-orthographic information to identify words and that good readers do not. Teaching kids to read like poor readers makes no sense to me. Nevertheless, Reading Recovery clearly gives kids a learning advantage, and we'd be wise to look hard at it to see why (one study found adding more explicit phonics to it improved kids' progress, and that's a clue that may help us understand what it does and what it doesn't).

The point isn't phonics or Reading Recovery: but when we make those kinds of choices, we need to weigh evidence consistently-treating as the same those studies that challenge our deepest beliefs as well as those that are wind beneath our wings. What works in teaching, who it helps, how it helps them... those are complex questions requiring sound evidence and wise analysis rather than rage and cheap "hooray for our side" Tweets.

Let's do better.

# data[].attributes.attachment_link.title
Making Sense of Reading Research | Shanahan on Literacy

# data[].attributes.attachment_link.description
Blast from the Past: Originally posted on November 3, 2018; re-posted on January 30, 2021. In this "science of reading" era, what it takes to make sense of reading research is even more timely today than 2 years ago when this entry first posted.