# data[].attributes.body
assessing what matters

# data[].attributes.attachment_link.readable_text
December 2007/January 2008 | Volume 65 | Number 4
Informative Assessment Pages 20-26

Robert J. Sternberg

This vignette points out in microcosm what may be wrong with the assessments to which we, as a society, have committed ourselves. As a teacher or administrator, how many times have you had to take a multiple-choice or fill-in-the-blank test except perhaps, when you needed to show that you were supposedly qualified for your job? When I look at the skills and concepts I have needed to succeed in my own field, I find a number that are crucial: creativity, common sense, wisdom, ethics, dedication, honesty, teamwork, hard work, knowing how to win and how to lose, a sense of fair play, and lifelong learning. But memorizing books is certainly not one of them.

One can argue, with justification, that one cannot think without content to think with and about. This is indisputable. But when we teach only for facts, rather than for how to go beyond facts, we teach students how to get out of date. For example, the facts that I learned in my introductory psychology course matter little today. An introductory text today contains almost entirely different facts. I know: I am the author of one of those textbooks (Sternberg, 2004). Other fields, such as the hard sciences, political science, economics, and so forth, change at least as rapidly. Even the humanities change: A set of classic works remains, but the interpretations-and even what constitutes such interpretations-change.

So what should we assess? We should assess what students need to become active and engaged citizens of the world in which they will live-in a sense, what it takes to be "expert" citizens. Oddly enough, a lot of models can prepare students for the roles they will play in their world. Traditional schooling just does not happen to be one of them. We should also assess in ways that can help students develop the skills they need for success in school and life.

Consider students on an athletic team. They learn declarative knowledge about the sport. But learning the rules of the game will no more help them in playing the game than memorizing a book of rules on driving will help someone drive. The students also need to learn how to play the sport.

But the most important skills they learn have nothing to do with one sport or another. These skills are very much like those I mentioned previously: dedication, honesty, teamwork, common sense, and the wisdom to distinguish right from wrong. Athletics is not the only model for such learning. Consider the members of an orchestra or of a dance ensemble. They, too, must learn to work together and must develop similar skills.

How might assessments better reflect the kinds of skills that matter-not just in school, but also in life beyond school? This is a question that we in the Center for the Psychology of Abilities, Competencies, and Expertise, formerly at Yale and now at Tufts University, have posed for ourselves. It is a challenge that we have, to some extent, taken as our life work.

Assessing for WICS

The model that underlies our assessments is called WICS, which is an acronym for wisdom, intelligence, and creativity, synthesized (Sternberg, 2003). The basic idea underlying this model is that active and engaged citizenship and especially leadership require individuals to have (1) a creative vision for how they intend to make the world a better place, not just for themselves, but also for their family, friends, colleagues, and others; (2) the analytical intellectual skills to be able to explain why their vision, and that of others, is a good one; (3) the practical intellectual skills to be able to execute their vision and persuade others of its value; and (4) the wisdom to ensure that their ideas represent a common good, not just their own interests or those of their friends or family. Can we apply this model to assessments that can be used in schools? We have done a variety of projects suggesting that we can.

The Successful Intelligence Model

Some of our earlier projects were based on the predecessor of WICS-the model of successful intelligence (Sternberg, 1997). The programs in this model were designed to determine whether we could teach and assess students for memory and for analytical, creative, and practical achievement in the context of any academic subject at any grade level. At that point, wisdom was not separated from practical skills, although it is distinguishable from them. Wisdom involves using academic and practical intelligence, as well as creativity and knowledge, for a common good. If, for example, a used-car salesman convinces customers to buy bad cars, he could be high in practical (or emotional) intelligence without being wise.

As an example, in social studies, we might assess understanding of the American Civil War by asking such questions as (1) Compare and contrast the Civil War and the American Revolution ( analytical); (2) What might the United States be like today if the Civil War had not taken place ( creative)? (3) How has the Civil War affected, even indirectly, the kinds of rights that people have today ( practical)? and (4) Are wars ever justified ( wisdom)?

In English, we might assess understanding of a novel such as The Adventures of Tom Sawyer by asking (1) How was the childhood of Tom Sawyer similar to and different from your own childhood ( analytical)? (2) Write an alternate ending to the story ( creative); (3) What techniques did Tom Sawyer use to persuade his friends to whitewash Aunt Polly's fence ( practical)? and (4) Is it ever justified to use such techniques of persuasion to make people do things they do not really want to do ( wisdom)?

In science, we might ask (1) What is the evidence suggesting that global warming is taking place ( analytical)? (2) What do you think the world will be like in 200 years if global warming continues at its present rate ( creative)? (3) What can you, personally, do to help slow down global warming ( practical)? and (4) What responsibility do we have, if any, to future generations to act on global warming now before it gets much worse ( wisdom)?

In mathematics, we might ask (1) What is the interest after six months on a loan of $4,000 at 4 percent annually ( analytical)? (2) Create a mathematical problem involving interest on a loan ( creative); (3) How would you invest $4,000 to maximize your rate of return without risking more than 10 percent of the principal ( practical)? and (4) Why do states set maximum rates of interest that lenders can charge, and should they do so ( wisdom)?

We have found in studies of reading, social studies, science, and mathematics at a variety of grade levels that teaching for analytical, creative, and practical thinking, as well as for memory, boosts achievement on tests that measure achievement broadly, across subject-matter areas and grade levels (see Grigorenko, Jarvin, & Sternberg, 2002; Sternberg, Grigorenko, Ferrari, & Clinkenbeard, 1999; Sternberg, Torff, & Grigorenko, 1998). Interestingly, even when students are assessed solely for memory, they perform better when taught broadly than when taught just for memory. This is because broader teaching enables students to capitalize on their strengths and correct or compensate for their weaknesses in learning. For example, broader teaching might involve encouraging students who are more visually oriented and less numerically oriented to draw a diagram to help them visualize and solve an algebra problem. Students who are more numerically oriented might proceed directly to constructing a set of equations.

Assessing Creative and Practical Thinking

In our society, a problem with teaching and assessing more broadly is that the kinds of standardized assessments we currently use are quite narrow. For example, the SAT Reasoning Test and the SAT Subject Tests assess primarily remembered knowledge and analytical skills applied to this knowledge. Creativity, practical thinking, and wisdom are assessed minimally or, more likely, not at all. Is there any hope that our society can transport some of these skills to high-stakes assessments?

My collaborators and I decided to find out. In one study, the Rainbow Project, we designed tests of creative and practical thinking that could supplement tests like the SAT Reasoning Test, which measures analytical skills in the verbal and mathematical domains. We tested 1,013 high school students and college freshmen from 15 different schools. We posed analytical questions much like those traditionally found on standardized tests. But we also asked the students to answer creative and practical questions.

The creative tests required the students to stretch their imaginations. For example, they might be asked to write a creative story with a title like The Octopus's Sneakers or 3821. Or they might be shown a collage of pictures, such as of musicians or athletes, and be asked to tell a story about the collage. Or they might be asked to caption an untitled comic strip.

The practical tests required the students to solve everyday problems. Some tests were presented verbally; others, through videos. For example, students might see a movie showing a student about to ask a professor for a letter of recommendation, but also showing the blank look on the professor's face, indicating that he did not know who the student was. The task would be to decide what the student should do. Or students might see a video that shows a group of friends trying to figure out how to move a large bed up a winding staircase.

There were three crucial findings (Sternberg & the Rainbow Project Collaborators, 2006). First, in addition to the information that the tests provided about students' creative and practical thinking capabilities, we learned something important about multiple-choice problem solving: Multiple-choice tests, no matter what they were supposed to measure, clustered together. Students who were better at one multiple-choice test tended to be better at others as well. This result suggested that using multiple-choice tests consistently tends to benefit some students and not others.

Second, we discovered that using broader tests for college admissions can enhance academic excellence. When compared with using SAT scores alone for predicting freshman-year grades, using these broader tests enabled us to double the accuracy of that prediction. Compared with the predictive value of SAT scores and high school grade point average combined, we increased the accuracy of prediction by about 50 percent. In other words, our assessments were not quixotic ventures into esoteric realms. On the contrary, they enhanced our ability to predict who would be more, as opposed to less, successful in college, at least from an academic point of view.

Third, we discovered that we could substantially reduce ethnic group differences with the tests. In other words, using such tests could increase the proportion of ethnic minorities admitted to selective colleges. The tests would not compromise academic excellence, but actually enhance it. Because different ethnic groups have different conceptions of what intelligence is (Sternberg, 2006), they tend to socialize their children to be intelligent in different ways. For example, on our tests, American Indians, on average, performed lower than most other groups on analytical assessments. But on oral storytelling, they had the highest average scores. Different groups excel, on average, in different ways; giving them a chance to show how they excel enables them to show that they can succeed.

Tests like the Rainbow Assessment do not benefit only members of ethnic minority groups. Many students who come from the majority group, and even from well-off homes, learn best in ways that are different from those assessed by conventional standardized tests. Our tests help identify such students.

Increasing Quality and Diversity

It is one thing to have a successful research project, and another actually to implement the procedures in a highstakes situation. We have had the opportunity to do so.

In 2005, I moved from Yale University, where I was the lead collaborator in the Rainbow Project, to Tufts University, where I became dean of the School of Arts and Sciences. Tufts University, under the leadership of its president, Lawrence Bacow, has strongly emphasized the role of active citizenship in education. So it seemed like an ideal setting to put into practice some of the ideas from the Rainbow Project. In collaboration with Linda Abriola, dean of the School of Engineering, and Lee Coffin, dean of admissions, I instituted Project Kaleidoscope, which implements the ideas of Rainbow but goes beyond that project to include in its assessments the construct of wisdom.

On the 2006-07 application for all of the more than 15,000 students applying to the schools of Arts, Sciences, and Engineering at Tufts, we placed questions designed to assess WICS (Sternberg, 2007). Whereas the Rainbow Project was a separate high-stakes test administered with a proctor, the Kaleidoscope Project was a section of the Tufts college application. The advantage of the Kaleidoscope Project is that it got us away from the high-stakes testing situation in which students must answer complex questions in very short amounts of time under incredible pressure. The section was optional this past year, and students were encouraged to answer just a single question.

For example, a creative question asked students to write stories with titles like "The End of MTV" or "Confessions of a Middle School Bully." Another creative question asked students what the world would be like if some historical event had turned out differently, for example, if Rosa Parks had given up her seat on the bus. Yet another creative question, a nonverbal one, gave students an opportunity to design a new product or an advertisement for a new product. A practical question queried how students had persuaded friends to adopt an unpopular idea. A wisdom question asked students how they might apply a passion they had toward the common good.

We now have the results of our first year of implementation, and they are promising. Some stakeholders were afraid that the number of applications would go down; instead, they went up slightly. More notable, the quality of applicants rose substantially. There were fewer students in what before had been the bottom third of the pool in terms of academic quality. Many of those students, seeing the new application, decided not to bother to apply. Other stakeholders were afraid that average SAT scores might plummet. Instead, they went up. This is because the new assessments are not negatively correlated with SAT scores. Rather, they are not much correlated at all.

So adopting these new methods does not result in admitting less-qualified applicants. Rather, admitted applicants are more qualified, but in a broader way. Moreover, after several years in which the number of applications by underrepresented minorities remained relatively flat, this year they increased substantially. In the end, we admitted 30 percent more black students than the year before and 15 percent more Hispanics. Our results, like those of the Rainbow Project, showed that it is possible to increase academic quality and diversity simultaneously and to do so for an entire undergraduate class at a major university. Most important, we sent a message to students, parents, high school guidance counselors, and others that we believe there is more to a person than the narrow spectrum of skills assessed by standardized tests and that we can assess these broader skills in a quantifiable way.

Such projects can be done at any level. We designed an admissions test for a well-known private school, which showed results for a whole class that were comparable to those for the Rainbow Project. We also did a project in a large business school and showed that we could increase the accuracy of prediction and decrease both gender and ethnic group differences in admissions (Hedlund, Wilt, Nebel, Ashford, & Sternberg, 2006). We are currently developing a comparable test for middle school students (Chart, Grigorenko, & Sternberg, in press).

One might wonder how to assess responses to questions that seem so subjective. The answer is through well-developed rubrics. For example, we assess analytical responses on the basis of the extent to which they are analytically sound, balanced, logical, and organized. We assess creative responses on the basis of how original and compelling they are, as well as on the basis of their appropriateness to the task presented. We assess practical responses on the basis of how feasible they are with respect to time, place, and human and material resources. We assess wisdom-based responses on the extent to which they promote the common good by balancing individual interests with others' larger interests, over the long and short terms, through the infusion of positive (prosocial) values.

Promoting Wisdom

Perhaps conventional assessments met the cognitive demands placed on students 100 years ago. They do not meet the cognitive demands of the world today. Active and engaged citizens must be creatively flexible, responding to rapid changes in the environment; able to think critically about what they are told in the media, whether by newscasters, politicians, advertisers, or scientists; able to execute their ideas and persuade others of their value; and, most of all, able to use their knowledge wisely in ways that avoid the horrors of bad leadership, as we have seen in scandals involving Enron, Arthur Andersen, Tyco, Clearstream, and innumerable other organizations.

It may be a hard sell to teach and assess for wisdom. However, wisdom is the most important and yet most neglected aspect of education today (Sternberg, 2001a, 2001b). We have seen in failed leaders the enormous costs of having leaders who are knowledgeable and intelligent-who have "good degrees" from prestigious schools-yet who are unwise. They tend to commit several serious cognitive fallacies. They are (1) unrealistically optimistic, believing that anything they do will turn out well because they are so brilliant; (2) egocentric, believing that the world revolves around them; (3) falsely omniscient, failing to learn from experience because they believe they know everything; (4) falsely omnipotent, believing that they are all-powerful by virtue of their superior skills or education; (5) falsely invulnerable, believing they can get away with almost anything because they are so clever; and (6) ethically disengaged, believing that ethical principles apply only to lesser mortals. In my view, much of what is wrong in the world today stems from people who are simultaneously smart and foolish.

Four caveats are in order here. First, my work on WICS and successful intelligence is not the only theory on the basis of which we might create new, broader assessments. Howard Gardner's (1999) theory of multiple intelligences provides another basis for such assessments, and other theories could be used as well. Second, the assessments do not measure all the skills required for success in everyday life. For example, although I assess teamwork in the courses I teach, the assessments I have described do not measure this skill, at least not directly. Third, the assessments have not been scaled up for use on a statewide or national basis. Doing so would no doubt present new challenges. Fourth, expanded assessments cost more time and money. But when we consider the benefits of opening up possibilities and hope to diverse students who learn and think in a variety of ways-whatever their gender or ethnic background-the costs may be relatively small.

Worthy and Wise

There is another issue we need to face. Traditional assessments provide little help to students in learning how to capitalize on strengths and compensate for or correct weaknesses. They measure only narrow bands of skills. Broader tests can give broader ranges of scores and help students see where they have mastery and where they need to improve. Teachers, in turn, can teach in ways that help students acquire the skills they need to succeed in school and life (Sternberg & Grigorenko, 2000, 2007). From this point of view, instruction and assessment are two sides of the same coin rather than two different coins. Assessment drives instruction.

So let's create assessments that are worthy of such a role. To prepare students for a world in which political, economic, social, and even climatic contexts are rapidly changing, we must focus on more than just facts and figures. Our society needs citizens and leaders who are not just memorizers and who are more than just analytically adept. We need people who are creative, practical, and, especially, wise.

References

Chart, H., Grigorenko, E. L., & Sternberg, R. J. (in press). The Aurora Battery: Toward better identification of giftedness. In C. Callahan & J. Plucker (Eds.), What the research says about: An encyclopedia of research on gifted education. Waco, TX: Prufrock Press.

Gardner, H. (1999). Intelligence reframed: Multiple intelligences for the 21st century. New York: Basic Books.

Grigorenko, E. L., Jarvin, L., & Sternberg, R. J. (2002). School-based tests of the triarchic theory of intelligence: Three settings, three samples, three syllabi. Contemporary Educational Psychology, 27, 167-208.

Hedlund, J., Wilt, J. M., Nebel, K. R., Ashford, S. J., & Sternberg, R. J. (2006). Assessing practical intelligence in business school admissions: A supplement to the graduate management admissions test. Learning and Individual Differences, 16, 101-127.

Sternberg, R. J. (1997). What does it mean to be smart? Educational Leadership, 54(6), 20-24.

Sternberg, R. J. (2001a). Why schools should teach for wisdom: The balance theory of wisdom in educational settings. Educational Psychologist, 36(4), 227-245.

Sternberg, R. J. (2001b). Wisdom and education. Perspectives in Education, 19(4), 1-16.

Sternberg, R. J. (2003). Wisdom, intelligence, and creativity synthesized. New York: Cambridge University Press.

Sternberg, R. J. (2004). Psychology (4th ed.). Belmont, CA: Wadsworth.

Sternberg, R. J. (2006). Recognizing neglected strengths. Educational Leadership, 64(1), 30-35.

Sternberg, R. J. (2007). Finding students who are wise, practical, and creative. The Chronicle of Higher Education, 53(44), B11.

Sternberg, R. J., & Grigorenko, E. L. (2000). Teaching for successful intelligence. Arlington Heights, IL: SkyLight.

Sternberg, R. J., & Grigorenko, E. L. (2007). Teaching for successful intelligence (2nd ed.). Thousand Oaks, CA: Corwin.

Sternberg, R. J., Grigorenko, E. L., Ferrari, M., & Clinkenbeard, P. (1999). A triarchic analysis of an aptitude-treatment interaction. European Journal of Psychological Assessment, 15(1), 1-11.

Sternberg, R. J., & the Rainbow Project Collaborators. (2006). The Rainbow Project: Enhancing the SAT through assessments of analytical, practical, and creative skills. Intelligence, 34(4), 321-350.

Sternberg, R. J., Torff, B., & Grigorenko, E. L. (1998). Teaching for successful intelligence raises school achievement. Phi Delta Kappan, 79, 667-669.

Robert J. Sternberg is Dean of the School of Arts and Sciences, Professor of Psychology, and Adjunct Professor of Education at Tufts University, Medford Massachusetts; Robert.Sternberg@tufts.edu.

# data[].attributes.attachment_link.description
December 2007/January 2008 | Volume 65 | Number 4 Informative Assessment Pages 20-26 Robert J. Sternberg This vignette points out in microcosm what may be wrong with the assessments to which we, as a society, have committed ourselves.

# data[].attributes.body
benefits of polls

# data[].attributes.body
Ed Week

# data[].attributes.attachment_link.provider_name
Education Week - Sara Mead's Policy Notebook

# data[].attributes.attachment_link.readable_text
Dana Goldstein has an interesting Slate piece looking at how states, schools, and districts are assessing children's learning in the performing and visual arts. This is an issue I've struggled with personally in my work as an authorizer--when charter authorizers approve an arts-themed charter school (of which there are many), ideally we want to see that the school is not only delivering student achievement results on state assessments, but that it's also fulfilling its mission to deliver high-quality arts education to students. Identifying appropriate arts-related measures of learning outcomes has proven challenging, however, for both schools and authorizers, so I'm glad Dana took a look at what some states and districts are doing in this area, as well as some well-regarded arts assessments at the high school level. That said, I'm sort of perplexed by an idea that I've seen in Dana's piece as well as elsewhere. Dana writes:

Students are then shown a measure of sheet music and asked to identify which of four electronic recordings matches the notation. The multiple choice section of the state's fourth grade arts exam shows students a picture, such as one of a vase and a bowl of fruit placed on a chair, and asks them to identify the drawing as either a "landscape," "portrait," "non-objective," or "still-life." The question is: Does a student's ability to answer such queries correctly actually indicate arts proficiency? Can such a test measure creativity--or is creativity not the point?

She also expresses concern that such assessments do not measure children's appreciation of art and may incent teachers to spend less time creating art with their students. I'm not sure I completely agree with this challenge. Obviously, if we believe that ability to create visual art or play an instrument at a specific level is a skill we want all children to master, then we need to look at children's actual performance and creation of visual arts--just as, if we want children to be good writers, we need to actually assess their writing. But I'm not entirely sure there actually is a consensus on these goals for students. Moreover, proficient writing builds on and is supported by a host of other specific, concrete skills--such as ability to appropriately use vocabulary and grammatical conventions,to identify the logical implication of a statement or piece of data, and to understand the structure of a coherent argument--which can be assessed through multiple choice and other easier-to score mechanisms. In the same way, musical performance builds on skills, such as the ability to read music, that the test Dana describes above do assess.

What really bugs me, however, is the implication that evaluating students' understanding and mastery of certain arts concepts has little to do with arts appreciation or proficiency. The point of arts education shouldn't be to teach children to simply "enjoy art"--we are, after all free to choose which art we enjoy, or whether we enjoy it at all. Rather, it should be to give children the skills and background knowledge to experience art or music in an informed and more than superficial sense--much of which is about understanding and identifying concepts, vocabulary, and techniques in ways that can be assessed through multiple choice assessments. A major reason that high-quality education needs to include the arts is certain arts-related information--such as names and work of key artists and composers, specific musical or artistic vocabulary and meanings, and artistic movements over time and their relationship to broader historical and social trends--is key cultural knowledge that our students need to be culturally literate. But arts and music instruction in our schools has often ignored cultural literacy and key concepts in favor of performance and "creativity." (This is not a knock on creativity, but creativity is enhanced when children or artists have rich background knowledge and understanding of concepts and history from which to create.) Getting a bit more balance here is not a bad thing.

# data[].attributes.attachment_link.title
What Does it Mean to Teach Children to "Appreciate" Art?

# data[].attributes.attachment_link.description
Dana Goldstein has an interesting Slate piece looking at how states, schools, and districts are assessing children's learning in the performing and visual arts.

# data[].attributes.body
authentic assessment

# data[].attributes.attachment_link.readable_text
This is an excerpt on defining authentic assessment from eduplace.com:

Authentic assessment refers to assessment tasks that resemble reading and writing in the real world and in school (Hiebert, Valencia & Afflerbach, 1994; Wiggins, 1993). Its aim is to assess many different kinds of literacy abilities in contexts that closely resemble actual situations in which those abilities are used. For example, authentic assessments ask students to read real texts, to write for authentic purposes about meaningful topics, and to participate in authentic literacy tasks such as discussing books, keeping journals, writing letters, and revising a piece of writing until it works for the reader. Both the material and the assessment tasks look as natural as possible. Furthermore, authentic assessment values the thinking behind work, the process, as much as the finished product (Pearson & Valencia, 1987; Wiggins, 1989; Wolf, 1989).

(A comment from Jeff: this last sentence really resonates with me, as I have always believed that the journey is just as important as the destination when it comes to learning. We need to strike a balance between the two, if we are going to authentically assess our students.)

Working on authentic tasks is a useful, engaging activity in itself; it becomes an "episode of learning" for the student (Wolf, 1989). From the teacher's perspective, teaching to such tasks guarantees that we are concentrating on worthwhile skills and strategies (Wiggins, 1989). Students are learning and practicing how to apply important knowledge and skills for authentic purposes. They should not simply recall information or circle isolated vowel sounds in words; they should apply what they know to new tasks. For example, consider the difference between asking students to identify all the metaphors in a story and asking them to discuss why the author used particular metaphors and what effect they had on the story. In the latter case, students must put their knowledge and skills to work just as they might do naturally in or out of school.Goals of Authentic Assessment are discussed in the article

Incorporating Authentic Assessment from Park University. Here is an excerpt:

Goals of Authentic Assessment:
There is a great deal more in this article including a chart that compares traditional assessment with authentic assessment, advantages and disadvantages of both, guidelines for creating authentic assessment and much more. Check it out; it's worth a close reading.

Although it doesn't specifically mention integrating the arts, it doesn't take too much effort to see how one could use those ideas with authentic assessment.

# data[].attributes.attachment_link.description
This is an excerpt on defining authentic assessment from eduplace.com: Authentic assessment refers to assessment tasks that resemble reading and writing in the real world and in school (Hiebert, Valencia & Afflerbach, 1994; Wiggins, 1993). Its aim is to assess many different kinds of literacy abilities in contexts that closely resemble actual situations in which those abilities are used.

# data[].attributes.body
why have assessments?

# data[].attributes.attachment_link.description
NCLB demands that states assess their students in core subject matter. This high-stakes standardized testing will not give you an accurate snapshot of student proficiency. In most states, different students take the test every year and the test changes. Because of this, it is impossible to see growth over time.

# data[].attributes.body
meaningful assessments

# data[].attributes.attachment_link.readable_text
When I think about my own definition of a "meaningful assessment," I think the test must meet certain requirements. The assessment must have value other than "because it's on the test." It must intend to impact the world beyond the student "self," whether it is on the school site, in the outlying community, the state, country, world, etc. Additionally, the assessment should incorporate skills that students need for their future. That is, the test must assess skills other than merely content. It must also test how eloquently the students communicate their content.

Criteria for a Meaningful Classroom Assessment

To address these requirements, I ask myself the following guided questions:

1. Does the assessment involve project-based learning?

2. Does it allow for student choice of topics?

3. Is it inquiry based?

4. Does it ask that students use some level of internet literacy to find their answers?

5. Does it involve independent problem solving?

6. Does it incorporate the 4Cs?

7. Do the students need to communicate their knowledge via writing in some way?

8. Does the final draft or project require multiple modalities (visual, oral, data, etc.) in its presentation?

Clearly not all assessments achieve every single characteristic listed above. But in our attempt to address some of these elements, we will have made our classroom assessments so much more meaningful. It is vital that students connect with the value of their assessments. After all, if a student trusts that the assessment is meaningful and will help them later on, it helps with both their achievement and with your own classroom management.

Transparency and Why It's Important

It's important that we inform the students why a particular assessment has value. Some teachers still balk at this job, as if students should just trust that what we do in school has value to what happens outside of school. However, kids are smart. They know that bubbling with a #2 pencil is antiquated. They know that much of the content we teach them can be found through Google. But as savvy as students are, they don't know everything about communicating their content, and we owe it to them to make sure that not only are our tests aligned with skills they must know for their future, but to make sure that we've been transparent in our rationale.

So how can high-stakes assessments be meaningful to students? For one thing, high-stakes tests shouldn't be so high stakes. It's inauthentic. They should and still can be a mere snapshot of ability. Additionally, those occasional assessments need to take a back seat to the real learning and achievement going on in every day assessments observed by the teacher.

The key here, however, is to assess every day. Not in boring, multiple-choice daily quizzes, but with informal, engaging assessments that take more than just a snapshot of a student's knowledge at one moment in time.

But frankly, any assessment that sounds cool can still be made meaningless. It's how the students interact with the test that makes it meaningful. With the 4 Cs in mind, ask if the assessment allows for the following:

Creativity Are they students creating or just regurgitating? Are they being given credit for presenting something other than what was described?

Collaboration Have they spent some time working with others to formulate their thoughts, to brainstorm, or to seek feedback from peers?

Critical Thinking Are the students doing more work than the teacher in seeking out information and problem solving?

Communication Does the assessment emphasize the need to communicate the content well? Is writing involved, as well as other modalities? If asked to teach the content to other students, what methods will the student use to communicate the information and help embed it more deeply?

Rubric on Meaningful Assessments

So as an activity for myself, I created a rubric to look at whenever I was wondering if an assessment was going to be a waste of time or was going to connect with the students. (Click the chart to download the PDF.)

Another way to ensure that an assessment is meaningful, of course, is to simply ask the students what they thought. Design a survey after each major unit or assessment. Or, better yet, if you want to encourage students to really focus on the requirements on a rubric, add a row that's only for them to fill out for you. That way, the rubric's feedback is more of a give-and-take, and you get feedback on the assessment's level of meaningfulness as soon as possible.

Download the example (left) of a quick rubric I designed for a general writing assessment. I included a row that the participants could fill out that actually gave me quick feedback on how meaningful or helpful they believed the assessment was towards their own learning. As an instructor and lesson designer, I want a quick turnaround between when I assign an assessment and if I need to adjust the assessment to meet the needs of future learners. By also giving them a space to fill out, they own the rubric even more, and will pay more attention to what I fill out knowing that I gave them an opportunity to also give me feedback. It's one way the students and I can learn reciprocally.

So how do you ensure that your classroom assessments are meaningful?

# data[].attributes.attachment_link.title
How Can We Make Assessments Meaningful? | Edutopia

# data[].attributes.attachment_link.description
When I think about my own definition of a "meaningful assessment," I think the test must meet certain requirements. The assessment must have value other than "because it's on the test." It must intend to impact the world beyond the student "self," whether it is on the school site, in the outlying community, the state, country, world, etc.

# data[].attributes.body
I wish there was a paint tool so I could create graffiti all over this brick-wall background

# data[].attributes.body
multiple resources

# data[].attributes.body
data supporting skilled assessment

# data[].attributes.attachment_link.readable_text
by Robert J. Marzano

Improving the academic achievement of K-12 students has been a central concern of educators in the United States since at least the early 1890s, when leaders of industry, politicians, parents, and the society at large realized that an educated populus was the closest thing a country could have to a guarantee of a bright future (Ravitch, 1983). Since that time, a wide array of educational innovations have been tried, all of which were designed to enhance student achievement. Educators have experimented with such things as changing the schedule, decreasing the student-to-teacher ratio, increasing the availability and use of technology, and so on. All of these innovations have merit. However, not even the best has demonstrated the impact on student achievement of the most intuitively important variable in the educational system-the classroom teacher.

Virtually every study that has examined the role of the classroom teacher in the process of educating students has come to the same straightforward conclusion: an effective teacher enhances student learning more than any other aspect of schooling that can be controlled. To illustrate, after analyzing test scores of more than 60,000 students across grades 3 through 5, researchers S. Paul Wright, Sandra Horn, and William Sanders (1997) made the following observation:

The results of this study will document that the most important factor affecting student learning is the teacher. In addition, the results show wide variation in effectiveness among teachers. The immediate and clear implication of this finding is that seemingly more can be done to improve education by improving the effectiveness of teachers than by any other single factor. Effective teachers appear to be effective with students of all achievement levels, regardless of the level of heterogeneity in their classrooms. If the teacher is ineffective, students under the teacher's tutelage will show inadequate progress academically regardless of how similar or different they are regarding their academic achievement. (p. 63)

Other studies have corroborated the conclusions of Wright, Horn, and Sanders (for a review of other studies, see Nye, Konstantopoulos, & Hedges, 2004). Kati Haycock (1998) dramatizes the effect of a classroom teacher by comparing what one can expect from a student spending a year with the "most effective teacher" and the "least effective teacher" (for a discussion of how "most effective" and "least effective" teachers are defined, see Technical Note 1.1). Haycock explains that the most effective teacher produces an achievement gain of 52 percentile points in student achievement, whereas the least effective teacher produces a gain of only 14 percentile points-a difference of 38 percentile points. This finding is made even more dramatic when one realizes that it has been estimated that students gain about 6 percentile points in academic achievement simply from growing one year older and gleaning new knowledge and skill from daily life (Cahen & Davis, 1987; Hattie, 1992). The ineffective teacher adds little more than life experience.

Given the potentially strong and positive effect of a classroom teacher, a logical question is, what do highly effective teachers do? Again, many answers have been proposed, most of which focus on lists of instructional and management strategies (see Hattie, 1992; Marzano, Marzano, & Pickering, 2003; Marzano, Pickering, & Pollock, 2001). These lists emphasize the use of strategies such as well-designed practice activities, comparison activities, communicating learning goals, and using pictures, graphs, and pictographs to represent knowledge. Although it is certainly true that "high-yield" instructional strategies and classroom management strategies are a critical part of effective teaching, this book is about one aspect of teaching that is frequently overlooked in discussions of ways to enhance student achievement: classroom assessment.

To the surprise of some educators, major reviews of the research on the effects of classroom assessment indicate that it might be one of the most powerful weapons in a teacher's arsenal. To illustrate, as a result of a synthesis of more than 250 studies, Paul Black and Dylan Wiliam (1998) describe the impact of effective classroom assessment in the following way:

The research reported here shows conclusively that formative assessment does improve learning. The gains in achievement appear to be quite considerable, and as noted earlier, amongst the largest ever reported for educational interventions. As an illustration of just how big these gains are, an effect size of 0.7 [see Technical Note 1.2 for a description of an effect size], if it could be achieved on a nationwide scale, would be equivalent to raising the mathematics attainment score of an "average" country like England, New Zealand or the United States into the "top five" after the Pacific rim countries of Singapore, Korea, Japan and Hong Kong. (p. 61)

It is important to note that Black and Wiliam's (1998) comments address formative as opposed to summative assessments. This distinction is addressed in the next section. To get a sense of Black and Wiliam's conclusions, consider Figure 1.1 (see Technical Note 1.3 for a description of how Figure 1.1 was derived). The upper part of Figure 1.1 depicts a teacher who begins at the 50th percentile in terms of her skill at using classroom assessments and a student in her class who begins at the 50th percentile in terms of his achievement. Over time the teacher increases her effectiveness at using classroom assessment to the 84th percentile. Given Black and Wiliam's findings, one would predict that the student's achievement would increase to the 63rd percentile. The lower part of Figure 1.1 represents an even more dramatic scenario. If the teacher increases from the 50th to the 99th percentile in terms of skill at using classroom assessments, one would predict the student's achievement to increase to the 78th percentile.

Figure 1.1. Effect of Teacher's Increased Skill in Classroom Assessment on Student Achievement

At face value, the findings reported in Figure 1.1 are remarkable-classroom assessment can have a dramatic influence on student achievement. Given these findings, one might be tempted to conclude that assessing students more will automatically increase their learning. Such a conclusion would be wrong. Like most things in education, classroom assessment enhances student achievement under certain conditions only. Fortunately, the research provides some guidance regarding those conditions.

A Brief Review of the Research on Classroom Assessment

Scholars have conducted many reviews of the research on classroom assessment. Some of the more comprehensive reviews are those by Natriello (1987); Fuchs and Fuchs (1986); Crooks (1988); Bangert-Drowns, Kulik, and Kulik (1991); Bangert-Drowns, Kulik, Kulik, and Morgan (1991); Kluger and DeNisi (1996); and Black and Wiliam (1998). The reviews lead to many conclusions that provide insights into effective classroom assessment; however, four generalizations are particularly germane to this book:

Feedback from classroom assessments should give students a clear picture of their progress on learning goals and how they might improve.

Feedback on classroom assessments should encourage students to improve.

Classroom assessment should be formative in nature.

Formative classroom assessments should be frequent.

Providing a Clear Picture of Progress and How to Improve

At a basic level, classroom assessment is a form of feedback to students regarding their progress, and it stands to reason that feedback will enhance learning. Indeed, as a result of reviewing almost 8,000 studies, researcher John Hattie (1992) made the following comment: "The most powerful single modification that enhances achievement is feedback. The simplest prescription for improving education must be 'dollops of feedback'" (p. 9).

As compelling as Hattie's comments are, all forms of feedback are not equally effective. In fact, some forms of feedback might work against learning. To illustrate, consider the research findings depicted in Figure 1.2. The figure presents findings from two major meta-analytic studies-one by Robert Bangert-Drowns, Chen-Lin Kulik, James Kulik, and Mary Teresa Morgan (1991), which reviewed 40 studies on classroom assessment; and one by Lynn Fuchs and Douglas Fuchs (1986), which reviewed 21 studies of assessment. The findings from these two synthesis studies as depicted in Figure 1.2 help one understand this first principle of effective classroom assessment.

Consider the first five rows of Figure 1.2, from the Bangert-Drowns, Kulik, Kulik, and Morgan synthesis. Row 1 indicates that when students receive feedback on a classroom assessment that simply tells them whether their answers are correct or incorrect, learning is negatively influenced. This finding is illustrated by the loss of 3 percentile points. However, when students are provided with the correct answer, learning is influenced in a positive direction. This practice is associated with a gain of 8.5 percentile points in student achievement, as shown in Row 2.

Row 3 of Figure 1.2 addresses whether students are clear about the criteria used to judge their responses. Clarity regarding scoring criteria is associated with a gain of 16 percentile points in student achievement. Row 4 reports a particularly interesting finding-providing students with explanations as to why their responses are correct or incorrect is associated with a gain of 20 percentile points in student achievement. Finally, Row 5 indicates that asking students to continue responding to an assessment until they correctly answer the items is associated with a gain of 20 percentile points.

Rows 6 and 7 of Figure 1.2 are from the Fuchs and Fuchs (1986) study. Row 6 shows the effect of graphically displaying student results. As we shall see in subsequent chapters, displaying assessment results graphically can go a long way to helping students take control of their own learning. However, this practice can also help teachers more accurately judge students' levels of understanding and skill, and it is associated with a gain of 26 percentile points in student achievement. Presumably, seeing a graphic representation of students' scores provides teachers with a more precise and specific frame of reference for making decisions about next instructional steps. Finally, Row 7 addresses the manner in which the teacher interprets assessment results. If the interpretation is done by a set of "rules," student achievement is enhanced by 32 percentile points. In Chapter 3 we will consider this issue in depth. Briefly, though, this finding implies that teachers within a school or a district should have rigorous and uniform ways of interpreting the results of classroom assessments.

Encouraging Students to Improve

One perplexing finding from the research literature is that the manner in which feedback is communicated to students greatly affects whether it has a positive or a negative effect on student achievement. This was one of the major conclusions of a meta-analysis conducted by Avraham Kluger and Angelo DeNisi (1996). After analyzing 607 experimental/control comparisons involving some 23,000 students, the researchers found that in 33 percent of the studies they examined, feedback had a negative impact on achievement. One causal factor they identified for this paradoxical effect is whether feedback encourages or discourages students. To illustrate, Kluger and DeNisi found that when assessment feedback is discouraging to students, it has an effect size of negative .14. This translates into a decrease in student achievement of 5.5 percentile points (see Technical Note 1.2 for a discussion of effect sizes).

Of course, the critical question that emerges from this finding is, what constitutes encouraging versus discouraging feedback? Kluger and DeNisi warn that this question has no simple answers, but the research provides some strong guidance. To understand the dynamics of encouraging versus discouraging feedback, we must consider two foundational aspects of motivation theory-drive theory and attribution theory.

Drive theory postulates that much of human motivation can be explained as a function of two competing forces, or drives-the striving for success and the fear of failure (Atkinson, 1957, 1964, 1987; Atkinson & Raynor, 1974). Over time, people develop tendencies toward one drive or the other-to be either success oriented or failure avoidant. When these tendencies become habituated, they translate into strong expectations regarding new tasks-particularly tasks that are challenging to a student.

Success-oriented students tend to be encouraged by challenges because they anticipate the positive feelings that accompany success. Failure-avoidant students tend to be discouraged by challenges because they anticipate the negative feelings that accompany failure. In fact, failure-avoidant students might use self-handicapping strategies that ensure they fail for reasons other than lack of ability. These strategies include procrastination (Rothblum, Solomon, & Murakami, 1986; Solomon & Rothblum, 1984), setting unattainable goals so that failure is ensured (Snyder, 1984), and admitting to small weaknesses or handicaps to establish an excuse for failing-establishing an "academic wooden leg" (Covington, 1992; Covington, Omelich, & Schwarzer, 1986).

Attribution theory provides another perspective on encouraging versus discouraging feedback. It postulates that the manner in which students explain or attribute failure and success encourages or discourages them (Weiner, 1972, 1974; Weiner et al., 1971). In general, individuals attribute their success to four causes: ability, effort, luck, and task difficulty. Of these, the attribution of effort provides the most encouragement. As Martin Covington (1992) explains:

One of the most important features of attribution theory is its focus on the role of effort in achievement. This emphasis is justified for several reasons. For one thing, if students believe their failures occur for a lack of trying, then they are more likely to remain optimistic about succeeding in the future. For another thing, trying hard is known to increase pride in success and to offset feelings of guilt at having failed. And, perhaps most important of all, the emphasis on the role of effort in achievement is justified because it is widely believed that student effort is modifiable through the actions of teachers. (p. 16)

A fairly straightforward relationship exists between attribution theory and drive theory. Specifically, students who tend to be success oriented also tend to believe in the effort attribution. They perceive that working hard will bring them success. Thus they have a way to succeed, even when faced with challenging tasks. One of the more encouraging aspects of attribution theory is that students who do not believe their efforts produce success can learn over time that they do. Martin Seligman (Seligman, 1975; Seligman, Maier, & Greer, 1968) postulates that students can even cultivate an "explanatory style" that is effort oriented, if they have enough direct experience that effort produces success. Seligman refers to this dynamic as "learned optimism."

Drive theory and attribution theory provide plausible explanations as to why assessment feedback might be encouraging to some students and discouraging to others. Assume that a student has done poorly on an assessment. If the student is failure avoidant, the negative outcome will strengthen the student's belief that he cannot succeed at challenging tasks and the negative feelings associated with such tasks. This combination will most likely discourage the student. However, if the student is success oriented, poor performance on the test will not be as discouraging because the student has a strategy for improvement-to work harder.

In short, drive theory tells us that classroom assessment that is encouraging must have two characteristics. First, teachers must provide students with a way to interpret even low scores in a manner that does not imply failure. If not, failureavoidant students will continually be discouraged when they do not receive high scores. Second, teachers must provide students with evidence that effort on their part results in higher scores.

The Value of Formative Versus Summative Assessments

The terms formative and summative are frequently used in discussions of educational assessments. Actually, the concepts of formative and summative assessment when first developed had little to do with classroom assessment or even with learning.

The distinction between formative and summative assessment was first popularized by Michael Scriven in 1967 as part of an American Educational Research Association monograph series on evaluation. Scriven's original point was that a distinction should be made between programs that are being formulated versus programs that have evolved to their final state. Consequently, evaluation takes on different characteristics and is interpreted differently in formative versus summative situations. This distinction was soon applied to the assessment of students. Specifically, formative assessment was defined as occurring while knowledge is being learned. Summative assessment was defined as occurring at the end of a learning episode-for example, at the end of a course (see McMillan, 2000). More formally, Peter Airasian (1994) defines formative assessments as those that "are interactive and used primarily to form or alter an ongoing process or activity. In contrast, assessments that come at the end of a process or activity, when it is difficult to alter or rectify what has already occurred, are called summative assessments" (pp. 135-136).

Although the terms formative and summative have both been widely used in literature on classroom assessment, formative assessment has received more attention in the research literature. Specifically, formative classroom assessment has been the focus of almost every major attempt to synthesize the research on classroom assessment. Recall the finding from Black and Wiliam's (1998) synthesis of more than 250 studies that formative assessments, as opposed to summative ones, produce the more powerful effect on student learning. In his review of the research, Terrance Crooks (1988) reports that effect sizes for summative assessments are consistently lower than effect sizes for formative assessments. In short, it is formative assessment that has a strong research base supporting its impact on learning.

Unfortunately, within the research literature, formative assessment is not defined consistently. As Black and Wiliam (1998) note, "Formative assessment does not have a tightly defined and widely accepted meaning" (p. 7). For the purposes of this book, I use the definition offered by Black and Wiliam that formative assessment encompasses "all those activities undertaken by teachers and/or by students which provide information to be used as feedback to modify the teaching and learning activities in which they engage" (pp. 7-8). This definition casts a wide net in terms of both types of activities that qualify as assessments and the timing of those activities. By definition, then, formative classroom assessment can and should begin immediately within a learning episode and span its entire duration. Additionally, formative classroom assessment can take a wide variety of formats, both formal (e.g., paper-and-pencil quiz) and informal (e.g., a discussion with a student).

The Importance of Frequency

One of the strongest findings from the research is that the frequency of assessments is related to student academic achievement. This finding was dramatically demonstrated in the meta-analysis by Robert Bangert-Drowns, James Kulik, and Chen-Lin Kulik (1991). They analyzed findings from 29 studies on the frequency of assessments. Their findings are depicted in Figure 1.3.

To interpret the figure, assume that we are examining the learning of a particular student taking a 15-week course. (For a discussion of how this figure was constructed, see Technical Note 1.4.) The figure depicts the increase in learning that one might expect when differing numbers of formative assessments are used during that 15-week session. If the teacher uses 5 assessments, a gain in student achievement of 20 percentile points is expected; if the teacher administers 25 assessments, a gain in student achievement of 28.5 percentile points is expected; and so on. Bangert-Drowns, Kulik, and Kulik (1991) comment on a number of aspects of this finding. First, they emphasize the relatively strong effect of a single assessment-13.5 percentile points, as depicted in Figure 1.3. Second, they highlight the fact that the frequency effect of assessment tapers off over time. As shown in Figure 1.3, the effect of assessment jumps dramatically from 0 to 10 assessments and then tends to level off. The recommendation from Bangert-Drowns, Kulik, and Kulik is not that teachers should use 30 assessments over a 15-week period but that teachers should systematically use classroom assessments as a form of feedback.

Figure 1.3. Gain Associated with Number of Assessments over 15 Weeks

Fuchs and Fuchs (1986) reported this same phenomenon in their metaanalysis of 21 controlled studies. They reported that providing two assessments per week results in an effect size of .85, or a percentile gain of 30 points. Although there is no set number of assessments that should be administered during a unit of instruction or a grading period, the message from the research is clear: systematic use of classroom assessments-weekly or even more frequently-can have a strong positive effect on student achievement.

Summary and Conclusions

Research supports the conclusion that formative classroom assessment is one of the most powerful tools a classroom teacher might use. Formative assessments are defined as any activity that provides sound feedback on student learning. Characteristics of sound feedback include that it should be frequent, give students a clear picture of their progress and how they might improve, and provide encouragement. In the remaining chapters, these principles are used to design a comprehensive system of effective classroom assessment.

# data[].attributes.attachment_link.description
by Robert J. Marzano Improving the academic achievement of K-12 students has been a central concern of educators in the United States since at least the early 1890s, when leaders of industry, politicians, parents, and the society at large realized that an educated populus was the closest thing a country could have to a guarantee of a bright future (Ravitch, 1983).

# data[].attributes.body
good assessment

# data[].attributes.body
assessments with accuracy, meaning, support learning

# data[].attributes.attachment_link.readable_text
Ensure Accuracy, Meaning, Consistency, and Support for Learning

Ken O'Connor

What are the measures that schools and teachers use most? Grades. So "good grades" are the measures that schools really need. Unfortunately, many grading practices are sadly lacking.

Most teachers have little training in how to grade, so grading has developed into a private and idiosyncratic practice. Superintendents, principals, and teachers have important roles to play in designing and implementing assessment, grading, and reporting policies and procedures that will ensure that grades are accurate; meaningful; consistent; and, most important, supportive of learning.

If any of these conditions is lacking, then the grades fail as effective measures. But educators can take steps to ensure that each of the four conditions of quality for grades are met.

First and foremost, accuracy requires that behaviors and attitudes be separated from achievement, so that grades are pure measures of achievement. The most practical way to do this is to use K-12 expanded format report cards that have a section where valued behaviors are reported for each student separately from the grade for achievement of learning goals. For example, in the province of Ontario, for grades 9-12, five "learning skills" are reported for each subject a student takes.

Other necessary practices include

Eliminating the use of penalties for behaviors such as submitting required assessment evidence after due dates, absence, and academic dishonesty;

Eliminating the use of extra credit for activities that have nothing to do with demonstration of achievement of standards;

Ensuring that grades are based on individual achievement, which means that group scores have no place in the determination of grades;

Ensuring that every assessment meets standards for quality assessment-clear targets, clear purpose, and sound design; and

Calculating appropriately, which involves using the median and mode as well as the mean, and finding alternatives to the use of zeros, especially where a percentage grading scale is used.

Make Grades Meaningful

For grades to be meaningful in standards-based systems, they must be based on standards, not assessment methods. This means teachers must do standards-based assessment and organize their grade books with standards as the set and assessment methods as the subset. For example, in that kind of matrix, the vertical organizer would be the target standards or strands, and the horizontal organizer (e.g., achievement evidence) would be the various tests. Each standard could have space to record comments on student strengths, areas for improvement, or observations.

Make Grades Consistent

For consistency, it is essential to have clearly described performance standards that teachers interpret in the same way. A limited number of levels (e.g., 2-7) with well-written generic descriptors become the basis for the scoring tools used at the classroom level. We must eliminate the use of the percentage system, as we cannot clearly describe 101 levels. Percentage scales are incompatible with standards-based systems where the primary focus for achievement levels is whether students are proficient or not.

Make Grades Supportive of Learning

Maybe the most important thing that needs to be done in assessment and grading is to make sure that students understand that the focus of the classroom is on learning, not just accumulating points. This can be achieved by

Using "no mark; comment only" formative assessments and including primarily summative assessments in the determination of grades. This means that quizzes, most homework, and most daily work have no place in grades.

Emphasizing more recent evidence and ignoring early learning failings in the determination of grades: if a student is proficient in May, the fact that they were struggling in December becomes irrelevant.

Involving students in the assessment and grading process by providing choices in the "how" and "what" of assessment, by providing many opportunities for self- and peer assessment, and by involving students in the communication process through the use of student-involved and student-led conferences.

If schools and districts focus on the four conditions of quality for grades and implement the practices suggested above, teachers, students, and parents will have the accurate measures they need to make good decisions that help students to achieve and to develop positive attitudes about learning and school.

Ken O'Connor is an education consultant in Scarborough, Ontario, Canada.

# data[].attributes.attachment_link.description
Ken O'Connor What are the measures that schools and teachers use most? Grades. So "good grades" are the measures that schools really need. Unfortunately, many grading practices are sadly lacking. Most teachers have little training in how to grade, so grading has developed into a private and idiosyncratic practice.

# data[].attributes.body
multiple assessment methods

# data[].attributes.attachment_link.readable_text
Assessment Highlights Range of Students' Multiple Intelligences

Branton Shearer

Schools that make the development of self-understanding an explicit part of their instruction and curriculum give students powerful tools for current and lifelong learning. Howard Gardner's multiple intelligences (MI) theory articulates exactly the kind of global structure needed for assessments that take stock of and highlight the many facets of the student's whole intellect.

MI theory describes the human mind as possessing at least eight distinct forms of intelligence, rather than a single, all-purpose, general intelligence (Gardner, 1993, 1999). These intelligences include linguistic, logical-mathematical (both traditional IQ-related abilities), visual-spatial, kinesthetic, musical, naturalist, intrapersonal, and interpersonal. This powerful concept teaches that the intellectual potential of each student's mind is best described multidimensionally, and that it would be superficial misrepresentation to look only at typical tests of academic performance when judging student potential.

Seeing Strengths First

In 1987, Gardner's MI theory was relatively new, but I considered it ideal for framing interviews I was conducting with family members of traumatic brain injury survivors who were undergoing cognitive rehabilitation. As an educator and counselor, I needed to understand each survivor's strengths and limitations prior to injury so that I could employ strengths-based cognitive remediation strategies. The MI-structured interview evolved into a self-report questionnaire, a psychometrically valid assessment called the Multiple Intelligences Developmental Assessment Scales (MIDAS). MIDAS produced a qualitative and quantitative profile of the respondent's "intellectual disposition" (Shearer, 2007).

MIDAS interested educators who were enthusiastic about the creative possibilities introduced by MI theory, so I adapted the assessment for use as a self-report by students, teachers, counselors, and psychologists. Now used in schools in the United States and 23 other countries, MIDAS, by design, provides research-based, richly detailed descriptions of a student's intellectual and creative life that can be easily validated and then associated with a wide variety of academic and real-world tasks. MIDAS-related interpretative materials guide students to critically reflect on their profile, share it with a family member, compare it to test scores and to class grades, and discuss it with a teacher. Students can then use these other sources of information to adjust or affirm their profile. Once they've verified their profiles, students can match strengths to study strategies, course selections, career pathways, college majors, and community involvement opportunities.

Quantitative and Qualitative

The MIDAS profile consists of eight main scales tied to the multiple intelligences, along with 26 subscales that describe specific skill domains within each. For example, the musical main scale includes four subscales: vocal, musical appreciation, instrumental, and composing. The subscales provide a rich source of information from the student's point of view that is rarely available in a timely, succinct, graphic format. There are also intellectual style scales that describe the strength of three general preferences: leadership, general logic, and innovation.

A careful review of the subscales can play an important role in understanding a student's MI profile because they can reveal strengths and nuances that are otherwise hidden. For example, Joe may be an accomplished vocalist, but have only a moderate score on the musical main scale because of his very low instrumental ability. His very high innovative scale would suggest that he tends more toward the creative than the technical aspects of music and singing.

MIDAS consists of 119 questions about the developed skill, level of participation, and expressed enthusiasm for a wide variety of activities students and adults naturally encounter in daily life. These range from practical activities, such as map reading and technical writing, to creative endeavors in the arts to social-emotional problem solving, like conflict negotiation and stress management. The MIDAS process has three goals: first, to educate students (and other stakeholders) about the eight multiple intelligences; second, to obtain a realistic description of student strengths and limitations through an MI assessment; and third, to connect MI strengths to learning activities and career paths that will maximize success. These interpretative activities are detailed in several of my books for students, teachers, counselors, psychologists, and parents.

A Dialogue of Discovery

Students typically complete the MIDAS questionnaire either online or on paper in 20-30 minutes. Then, a computerized program uses a research-based scoring matrix to generate a three-page report detailing student strengths and limitations in terms of the eight multiple intelligences. The first page of the three-page profile displays a bar graph of the scores for each of the eight main scales along with the three intellectual style scales. The second page provides a listing of all 26 subscales, arranged hierarchically so that the subscales at the top of the list are the highest scoring and those at the bottom are the lowest scoring. The third page provides the scores for all main scales and subscales, which are simple percentages ranging from 0-100.

To understand the meaning and implications of their profiles, the students must learn about each of the multiple intelligences and their application to learning, careers, and everyday life. Students each then write a Brief Learning Summary (BLS) based on their profiles. Beyond the graphic depiction of a student's eight main scales, and four highest and four lowest subscales, the BLS includes brief descriptions of specific activities students associate with their strengths. For example, Joe would have his musical scale in the middle range and his vocal scale in the high end; he then might describe how he sings with two different rock bands and writes rap lyrics and poetry.

After self-reflection and further discussion of the BLS with teachers, family, or friends, the student adjusts the BLS so that it becomes the best description possible of current skills and abilities. The resulting "verified" profile can then be used like a map to focus the thinking of students, parents, and teachers as students make important education and career decisions.

The BLS minimizes the focus on scale scores, which, if not used properly, tend to be misconstrued as labels that limit students, rather than direct their efforts. This avoids reducing students to simplistic statements, such as "Oh, he's just a spatial learner, not a linguistic one." MIDAS, reflecting MI theory, emphasizes that every person has all of the intelligences, each of which can be improved with time, effort, and guidance. The MIDAS process of reflection, critique, and dialogue naturally enhances students' understanding of themselves, providing a realistic appreciation for the value of personal strengths that can enable success and happiness.

In Hong Kong, where the drive for academic excellence can be intense, MIDAS helped a student gain valuable insight on this learning-and some parental understanding, according to Fong Wing Sze, a student advisor at the Vocational Training Council in Hong Kong.

Geoffrey was enrolled in the high diploma program in engineering, but was uninterested in the course content and felt frustrated about his future. Although he was influential with peers, Geoffrey also would "never lose an argument" with teachers and parents, so he was labeled a "bad student," says Fong. The more he was punished, the less motivated he became.

In reviewing his MIDAS profile, Geoffrey and his mother realized that he had strengths in musical and naturalist intelligences, but "in Chinese culture, musical intelligence and naturalist intelligence are usually thought to be interests only" and a diversion from school, explains Fong. "Both Geoffrey and his mother were trapped by this myth."

In learning to appreciate his potential in the naturalist, musical, and interpersonal intelligences, and with more understanding from his parents, Geoffrey increased his motivation to continue his studies, explore other course options in management or government, and work to gain entrance to a university.

When students are taught to use MI language to describe how they think, learn, and perform, they gain enhanced self-confidence and can become strategic learners. When students know how to use their MI strengths to think realistically about career paths that maximize their chances for success as adults, then schools, too, become more deeply engaged in high-level, life-changing education.

Gardner, H. (1993). Frames of mind. (Rev. ed.). New York: Basic Books.

Gardner, H. (1999). Intelligence reframed: Multiple intelligences for the 21st century.
New York: Basic Books.

Shearer, C. B. (2007). The MIDAS: Professional manual. (Rev. ed.). Kent, Ohio:
MI Research and Consulting, Inc.

Branton Shearer, a developmental psychologist and instructor at Kent State University in Ohio, developed the Multiple Intelligences Assessment Scales (MIDAS). For more information go to www.miresearch.org/.

# data[].attributes.attachment_link.title
Unleashing Strengths, Shoring Up Weaknesses

# data[].attributes.attachment_link.description
Branton Shearer Schools that make the development of self-understanding an explicit part of their instruction and curriculum give students powerful tools for current and lifelong learning. Howard Gardner's multiple intelligences (MI) theory articulates exactly the kind of global structure needed for assessments that take stock of and highlight the many facets of the student's whole intellect.

# data[].attributes.body
There is a need for alternative methods of assessing elementary art students that is both effective and efficient.

# data[].attributes.body
Because art teachers use the "B,D,M"(beginning, developing, mastered)
grading scale, there is no extrinsic push to develop more valid assessments

# data[].attributes.body
Problem: Assessment within the art room is often limited to performance assessments. This cuts down on the validity of the assessment

# data[].attributes.body
VALIDITY
Wiggins and McTighe (2005, p.183) we typically pay too much attention to correctness and too little attention to degree of understanding.

In using performance assessments in art, I typically look for understanding in how a student performs on their art work. Seldom is there a right or wrong.

Should I use more traditional assessments to assess some correctness?

# data[].attributes.body
ADVANTAGES OF MULTIPLE CHOICE

Nitko and Brookhart (2011, p.169)

-Assess a greater variety of learning targets

-minimize the opportunity for students to "dress-up" their answer

-do not require writing

-less chance to guess correctly

-distractors can be used to diagnose misconceptions

# data[].attributes.body
SUPPORT FOR MULTIPLE ASSESSMENTS

# data[].attributes.attachment_link.readable_text
Winter 2008 | Number 52
The Case for Multiple Measures

Full Article

Dan Fuller, Kevin Fitzgerald and Ji Sun Lee

These standardized, off-the-shelf-tests provide very little, if any, information to inform learning and teaching. Due to the standardized format, the timing of the tests, and the distribution of results, they offer little in way of diagnosis but have significant weight in labeling the performance of schools and school districts. Under NCLB, many educators have grown concerned that judgments about performance of students, subgroups of students, schools, and districts are based solely on state-approved tests in reading and math.

No single test can tell all there is to know. As the directors of the National Center for Research on Evaluation, Standards, and Student Testing emphasize, "Multiple measures are needed to address the full depth and breadth of our expectations for student learning" (p. 2). Beyond the multiple-choice and short-answer items that are typical of current assessments, "other types of performance measures-essays, applied projects, portfolios, demonstrations, oral presentations, etc.-are needed to represent and guide students' progress" (Herman, Baker, & Linn, 2004, p. 2).

Painting the Full Picture

The purpose of assessments should be to inform teaching and learning. Testing and assessing should be diagnostic; that is, they should provide information about individual student learning strengths and deficiencies to enable educators to adjust methods accordingly. Students learn differently, and teachers should be able to distinguish these differences and determine effective ways to teach students based on how they learn best. This should be the highest priority for all testing and teaching.

ASCD supports accountability and believes we should test all students and measure and disaggregate testing data. However, we do remain concerned with the significant consequences associated with a single test as the sole determinant of performance. The high stakes associated with the state assessments as part of No Child Left Behind lead to many complications and invalid conclusions. Many schools are being unfairly penalized for poor results on these tests. And morale is declining among educators because of the increasing pressure to teach to the test, which leaves little time to extend students' education beyond the tested subjects.

ASCD supports the use of multiple measures of assessment. We believe that decision makers in education, students, parents, educators, community members, and policymakers all need timely access to information from many sources if they are to make informed judgments about student learning and the success of education programs. Using a single achievement test as the sole measure of learning is inappropriate. The success of students, schools, districts, states and provinces, or nations should be determined by multiple assessments of and for learning.

ASCD supports the use of multiple measures in assessment systems that are:

Fair, balanced, and grounded in the art and science of learning and teaching;

Reflective of curricular and developmental goals and representative of content that students have had an opportunity to learn;

Used to inform and improve instruction;

Designed to accommodate nonnative speakers and students with special needs; and

Valid, reliable, and supported by professional scientific and ethical standards designed to fairly assess the unique and diverse abilities and knowledge base of all students.

Single Assessment Is Not the Answer

Initially, public education sorted students from the lowest to highest achievers, to channel individuals into the various segments of the social and economic system. In the last few decades, society changed its mission regarding public education, expanding to provide all citizens equal access to education and the opportunity to succeed, regardless of race, sex, or ability. Therefore, the notion of sorting evolved into a mission of ensuring certain competencies so that individuals had a fighting chance in the increasingly complex global society. Schools are now required to "leave no child behind" by demonstrating through appropriately high scores on statewide accountability assessments that their students are becoming competent readers, writers, and problem solvers.

Unfortunately, in the current climate, the language of equality and accountability in education has been couched in the mistaken belief that a single high-stakes test is an objective and valid measurement of individual students' achievement and knowledge. Monetary investments in U.S. education have reflected this belief, as billions of dollars over six decades have been provided for district, state, national, and international testing programs (Stiggins, 2004).

Despite mounting evidence of the harm standardized testing programs cause for teaching and learning, criticism toward a system that employs a single measure of assessment receives censure from many supporters of NCLB. These individuals claim that a system of multiple measures will lead to less accountability and that, by expanding assessment measures, schools and educators will be free to hide or soften low standardized scores with increases in other, more subjective assessments. Arguments like this, however, are often more inflammatory than accurate.

Ironically, the effects of test misuse have been most unfortunate for the very students they were most intended to help. Many studies have found that students placed in the lowest tracks or in remedial programs-disproportionately low-income and minority students-are most apt to experience instruction geared only to multiple-choice tests, working at a low cognitive level on test-oriented tasks that are profoundly disconnected from the skills they need to learn. Rarely are they given the opportunity to talk about what they know, to read books other than textbooks, to write, or to construct and solve problems (Cooper & Sherk, 1989; Oakes, 1985; Trimble & Sinclair, 1986). The quality of education made available to many students has been undermined by the nature of the testing programs used to monitor and shape their learning.

The shortcomings of tests were less problematic when they were used as only one source of information among many other kinds of assessments about student learning and when they were not directly tied to decisions about students and programs. As schools have begun to "teach to the test," scores have become and even poorer gauge of overall student abilities. Class work has been oriented toward recognizing answers to multiple-choice questions, and narrowed curricula do not heighten proficiency in aspects of subject areas not covered by standardized tests, such as analysis, complex problem solving, and written and oral expression (Darling-Hammond & Wise, 1985; Haney & Madaus, 1986; Koretz, 1988). Chester Finn and Diane Ravitch (2007), former assistant secretaries for the U.S. Department of Education and once prominent NCLB advocates, comment on what will be the outcome of this situation:

Rich kids will study philosophy and art, music and history, while their poor peers fill in bubbles on test sheets. The lucky few will spawn the next generation of tycoons, political leaders, inventors, authors, artists and entrepreneurs. The less lucky masses will see narrower opportunities. (para. 17)

During the 1970s, when test-oriented accountability measures were instituted in U.S. schools, there was a decline in public schools' use of teaching methods appropriate to the development of higher-order skills, such as research projects and laboratory work, student-centered discussions, and the writing of essays or themes (Dearman & Plisko, 1982). Major studies by Boyer (1983), Goodlad (1984), and Sizer (1985) document the negative effects of standardized testing on teaching and learning in high schools, while the disadvantage created for U.S. students by the rote learning stressed in U.S. standardized tests has been documented in international studies of achievement (McKnight et al., 1987).

Finding Alternatives

Rather than providing a single snapshot of student performance, multiple measures-such as oral presentations, writing assessments, and portfolios-afford ongoing information about students' grasp of key concepts and how to enhance their understanding to help them meet or exceed academic requirements. The challenge for states is to balance the use of these measures with large-scale accountability tests that gauge their effectiveness under NCLB. In discussing the use of multiple measures in the state of Ohio, Mitchell D. Chester (2005) writes, "the use of multiple measures to broaden the construct of district and school effectiveness in turn helps to improve the acceptance and legitimacy of the accountability program, and thus promote desired outcomes" (p. 51).

Although many states now seek alternative ways to measure student performance on state standards in conjunction with their statewide tests, Nebraska is the only state that uses a system of local assessments, known as STARS (School-based Teacher-led Assessment and Reporting System), to meet its accountability requirements. Through the use of multiple measures to continually inform teachers and schools of their students' progress, STARS addresses concerns that the large-scale state tests resulting from NCLB (1) do not provide feedback teachers can use to inform instruction and identify students' strengths and needs at the time the learning occurs, and (2) cause schools to focus on students within reach of proficiency goals, depriving low- and high-achieving students of instruction that meets their needs (Forum on Educational Accountability, 2007). Richard Rothstein (2008) sums up the latter concern:

Any single proficiency standard invites sabotaging the goal of teaching all children, because the only ones who matter are those with scores just below passing. Educators call them "bubble kids," a term from poker and basketball, where bubble players or teams are those just on the cusp of elimination. Explicit school policies now demand that teachers ignore already-proficient children to focus only on bubble kids, because inching the bubbles past the standard is all that matters for "adequate yearly progress." (para. 18)

Without mandatory statewide tests to measure the performance of its students, Nebraska must still meet its accountability requirements under NCLB. In a 2006 article for the Phi Delta Kappan, Pat Roschewski of the Nebraska Department of Education and two of her colleagues at the University of Nebraska explain how the state aligns data from locally based assessment systems to standards that are reviewed and rated against the state department of education's content-area criteria. They explain the challenges and benefits of Nebraska's system:

This process honors teachers and relies on their professional judgment, but it also demands hard work and a great deal of leadership from all of the state's educators. Thus Nebraska educators face very specific challenges: to develop high-quality local assessment systems, to ensure that the data collected in those assessment systems are analyzed, and to use the data for improving instructional practice in the classroom. (p. 434)

In requiring districts to create locally developed assessments aligned to the same standards, STARS affords Nebraska educators a system for comparing results at the state level while providing students with learning experiences tailored to their individual needs at the classroom level. At the same time, the collaboration between administrators and teachers required to implement STARS shows a commitment to students that extends through every level of the state's education system.

The Nebraska system has led to gains on the state's rating of student performance. The percentage of districts at the exemplary level in reading rose from 31.8 percent (2000-01) to 66.1 percent (2004-05) at the 4th grade level, from 34.1 percent to 59.2 percent at the 8th grade level, and from 18.9 percent to 41.3 percent at the 11th grade level (Gallagher & Ratzlaff, 2007-2008).

Despite these results, Nebraska's local assessment system has been challenged by the U.S. Department of Education, which has not yet approved its use. The two sides continue to work together to iron out the details that will earn the system approval.

Nebraska's use of locally based assessments and collaboration with the U.S. Department of Education reflects how the current version of NCLB allows states to incorporate multiple measures to demonstrate student performance on state standards. However, the state's difficulty in gaining approval for its alternative system also illustrates why most states choose to administer single standardized tests to comply with the law.

A Growing Trend?

Another approach that could replace the current practice of basing school performance solely on standardized tests is the use of growth models. These models enable schools to show individual student achievement gains over time and allow the academic progress of students as a valid measurement of learning.

To move away from the snapshot view that the one-time statewide tests provide, states like Wyoming are calling for the reauthorization of No Child Left Behind to incorporate the use of growth models as performance measures in their state accountability systems. In 2005, to test their effectiveness as an alternative measure of accountability, the U.S. Department of Education began a pilot program that allowed nine states (Alaska, Arizona, Arkansas, Delaware, Florida, Iowa, North Carolina, Ohio, and Tennessee) to use growth models. In December 2007, the program was opened to all states that wanted to apply.

Some states seem to favor growth models as an alternative to the current system, and many education organizations embrace this approach as well. Of 16 groups who submitted recommendations for the reauthorization of NCLB, most support the inclusion of growth models in the calculation of AYP. Nearly all of these same groups called for significant changes to the AYP measurements and the reliance on one test versus the use of multiple measures and broader measurements (Education Commission of the States, 2007).

What Parents and the Public Want

Not just educators recognize the need for changes to NCLB. The American public is far ahead of the U.S. Congress, according to a September 2007 poll conducted for the Partnership for 21st Century Skills, an organization of business, community, and education leaders that advocates for the infusion of 21st century skills into education and provides tools and resources to help facilitate and drive change. The poll found:

Sixty-six percent of voters say they believe that students need more than just the basics of reading, writing, and math. They believe schools also need to incorporate a broader range of skills.

Eighty-eight percent of voters say they believe that schools can and should incorporate 21st century skills such as critical thinking and problem solving, computers and technology, and communication and self-direction into their curriculum.

Fifty-three percent say they believe schools should place an equal emphasis on 21st century skills and basic skills. (Partnership for 21st Century Skills, 2007, para. 3)

"These new polling results provide education leaders and policymakers the tremendous opportunity to make our education system more aligned with the needs of the 21st century workforce," said Partnership President Ken Kay. "The public strongly supports more rigorous expectations for students that integrate 21st century skills into core academic subjects. Educators want to equip students with these skills, but they need the public policy, professional development, assessment and curricular tools to accomplish this" (Partnership for 21st Century Skills, 2007, para. 9).

These skills are not readily included or developed through the use of standardized testing in reading and math. How can a student demonstrate technological proficiency through filling in the bubbles on a multiple-choice test? How can teamwork or oral presentation skills be reflected through this model of testing?

Time for Change

Alternative measures of student performance like Nebraska's locally based assessment system and growth models provide both a better, more accurate measurement than simple standardized tests, while also ensuring that students are equipped with marketable skills for success in work and life in the 21st century.

The NCLB reauthorization is progressing. Preliminary versions of the act, proposed in a discussion draft advanced by the House of Representatives, included a growth-model option for assessment using multiple measures. ASCD and many other national education organizations applauded this concept. At the time of this writing, it is unclear what the final version will look like; however, many educators are heartened by this approach. We remain hopeful and will actively engage policymakers to ensure the addition of new assessment measures under the reauthorized No Child Left Behind.

References

Boyer, E. L. (1983). High school. New York: Harper & Row.

Chester, M. D. (2005). Making valid and consistent inferences about school effectiveness from multiple measures. Educational Measurement: Issues and Practice, 24(4), 40-52.

Cooper, E., & Sherk, J. (1989). Addressing urban school reform: Issues and alliances. Journal of Negro Education, 58(3), 315-331.

Darling-Hammond, L., & Wise, A. (1985). Beyond standardization: State standards and school improvement. Elementary School Journal, 85(3), 315-336.

Dearman, N. B., & Plisko, V. W. (Eds.). (1982). The condition of education, 1982. Washington, DC: U. S. Department of Education.

Education Commission of the States. (2007). Adequate Yearly Progress (AYP): NCLB reauthorization. Retrieved December 15, 2007, from http://nclbmb.ecs.org/reports/Report.aspx?id=1560

Education Week. (2007). Quality counts 2007: From cradle to career-Connecting American education from birth to adulthood. Education Week, 26(17).

Finn, C. E., Jr., & Ravitch, D. (2007, August 8). Not by geeks alone. The Wall Street Journal, p. A13.

Forum on Educational Accountability. (2007, August). Assessment and accountability for improving schools and learning: Principles and recommendations for federal law and state and local systems. Retrieved January 2, 2008, from www.edaccountability.org/AssessmentFullReportJUNE07.pdf

Gallagher, C. W., & Ratzlaff, S. (2007, December-2008, January). The road less traveled. Educational Leadership, 65(4), 48-53.

Goodlad, J. I. (1984). A place called school: Prospects for the future. New York: McGraw-Hill.

Haney, W., & Madaus, G. (1986). Effects of standardized testing and the future of the National Assessment of Educational Progress. Chestnut Hill, MA: Center for the Study of Testing, Evaluation, and Educational Policy.

Herman, J. L., Baker, E. L., & Linn, R. L. (2004, Spring). Accountability systems in support of students learning: Moving to the next generation. CRESST LINE. Retrieved January 7, 2008, from www.cse.ucla.edu/products/newsletters/cresst_cl2004_2.pdf

Koretz, D. (1988). Arriving in Lake Wobegon: Are standardized tests exaggerating achievement and distorting instruction? American Educator, 12(2), 8-15, 46-52.

McKnight, C. C., Crosswhite, F. J., Dossey, J. A., Kifer, E., Swafford, S. O., Travers, K. J., & Cooney, T. J. (1987). The underachieving curriculum: Assessing U.S. school mathematics from an international perspective. Champaign, IL: Stipes.

Oakes, J. (1985). Keeping track: How schools structure inequality. New Haven, CT: Yale University Press.

Partnership for 21st Century Skills. (2007). U.S. students need 21st century skills to compete in a global economy [Press Release]. Retrieved December 15, 2007, from www.21stcenturyskills.org/index.php?option=com_content&task=view&id=369&Itemid=64

Roschewski, P., Isernhagen, J., & Dappen, L. (2006, February). Nebraska STARS: Achieving results. Phi Delta Kappan, 87(6), 433-437.

Rothstein, R. (2008, January/February). Leaving "No Child Left Behind" behind. The American Prospect, 19(1). Retrieved December 18, 2007, from www.prospect.org/cs/articles?article=leaving_nclb_behind

Sizer, T. (1985). Horace's compromise. Boston: Houghton Mifflin.

Stiggins, R. (2004). New assessment beliefs for a new school mission. Phi Delta Kappan, 86(1), 22-28.

Trimble, K., & Sinclair, R. L. (1986, April). Ability grouping and differing conditions for learning: An analysis of content and instruction in ability-grouped classes. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.

Dan Fuller is ASCD's public policy director, Kevin Fitzgerald is ASCD's policy communications specialist, and Ji Sun Lee is ASCD's former director of research and information.

# data[].attributes.attachment_link.description
Winter 2008 | Number 52 The Case for Multiple Measures Full Article Dan Fuller, Kevin Fitzgerald and Ji Sun Lee These standardized, off-the-shelf-tests provide very little, if any, information to inform learning and teaching.

# data[].attributes.body
Possibly to use as a reference as to how to set up my report

# data[].attributes.body
ADVANTAGE OF SHORT-ANSWER FORMAT
Nitko and Brookhart (2011, p.153) An advantage of the short answer format is that it lowers the probability of getting the answer correct by random guessing

# data[].attributes.body
GUESSING ON TRUE-FALSE
Nitko and Brookhart (2011, p.158) persons guessing randomly can expect to get on the average one half of the true-false items correct.

# data[].attributes.body
Performance assessment

# data[].attributes.body
INDIVIDUAL ACTION PLANNING: ASSESSMENT
Mills (2014, p.166) Findings of a study related to the use of innovative assessment strategies would provide the teacher researcher with specific actions for the next cycle of implementing an innovative assessment strategy.