# data[].attributes.headline
QUESTIONING AND FEEDBACK: TOP TEN STRATEGIES

# data[].attributes.subject
QUESTIONING AND FEEDBACK: TOP TEN STRATEGIES

# data[].attributes.body
In Feedback & Questioning, Teaching & Learning by Alex Quigley

# data[].attributes.attachment_link.readable_text
As part of our whole staff training at Huntington School we have been sharing ideas and collating 'Top Ten Strategies'. This list is the fruits of our labour:

1. Differentiated questioning.

Given the time we take doing it daily, effective questioning may well be the highest impact strategy we can employ. There is no 'one size fits every class' strategy. Whether it is 'hands up' or 'no hands up', it simply needs careful planning and differentiation. As a rule, avoid tennis questions between the teacher and the student, aiming instead for 'basketball questioning' that bounce ideas around the room, forcing students to listen more intently and be active learners. This short article provides some interesting evidence about questioning, with some very useful starting points: https://teal.ed.gov/sites/default/files/Fact-Sheets/12_TEAL_Deeper_Learning_Qs_complete_5_1_0.pdf.

2. Wait time.

Giving waiting time for answers to our questions is something we do instinctively. We think we give students ample time, but we too often don't. Have you ever timed yourself? The average time allowed for students giving a response has been estimated at 1.5 seconds! The optimum time is nearer 5 seconds. Evidence shows that the depth (between 400 and 800% increase in length of response) & quality of answers greatly improves with this extra waiting time of just a few seconds. Try counting explicitly after your questioning, or getting a student to do so on your behalf. See this blog on a teacher using the research and trialling increased ' wait time': http://improvingteaching.co.uk/2013/08/17/increasing-wait-time/.

3. Don't accept "I don't know ".

Being prepared with a strategy for students saying 'I don't know' is clearly essential. There are lots of methods: 'phone a friend'; 'I'll come back to you'; 'can anyone give a clue' etc. Another good strategy is to use a pack of cards (or counters - frankly whatever you have!). Every student starts with a card. When they answer take card in. Ensure that no-one is left with a card so that everyone contributes an answer.

4. DIRT time in lessons.

We have focused on this as a school and will continue to do so. Effectively, it is about making students focus on improving their work in response to feedback, deeply engaging with their learning, rather than skimming the surface and moving on. As a rule, students should be spending more actively responding to their feedback then we spend on giving it. Try John Titmus's DIRTy History approach: http://www.huntingtonlearninghub.com/dirty-history/.
5. "Now try...... Explain why...."
After feedback we need to make sure that we have students actively responding to their feedback. A nice and simple habit is to use ' Now try...Explain why... '. So simple: low, effort and potentially high impact.

6. Wedding cake feedback.

This is a simple analogy to make the layers of assessment clear. The first layer of the wedding cake is self-assessment. They need to be trained to understand what they're looking for and understand the huge value in self-assessing. It is the biggest tier - they should spent the most time assessing what they have done first! The second tier, taking slightly less time, is peer-assessment. Finally, the smallest tier is teacher assessment. We simply assess their assessment and that of their peers.

7. Question level analysis.

Students need to understand the specific command words in their subject discipline and breakdown the different types of questions/topics etc.
8. Model answer - 'what is the question?'
Using exemplar models is a fundamental aspect of good teaching practice. There are lots of ways to go about modeling, such as shared writing, using past exam answers etc. One easy method is to show students a ready-made model answer, but then ask them to formulate the question to the model answer (either simple closed questions or full essay questions).
9. Self-assessment.
The process of explaining (or even defining) the assessment criteria with students can help promote deeper learning. Students gain an understanding of both the subject matter and the assessment task in order to gauge how well they have met the criteria. Establish basics to make it happen regularly, such as including a student feedback box for them to record their reflections/concerns etc. This is a handy post on 'pre-flight checklists' to guide self-assessment: http://improvingteaching.co.uk/2013/08/17/inducing-quality-writing/.
10. 'Token Questions'.
To stop a barrage of low level questions from students during extended tasks you can allocate them two tokens so that they max out how many questions they can ask. This small tweak improves the quality of questions and eliminated the many time-wasting questions that they could likely answer with some thought.

If that wasn't enough, our brilliant coaches produced Top Ten Strategy lists for the other four strands of 'Truly Great Teaching a Huntington' that is the focus of our whole school training. They can all be found on our Huntington Learning Hub website:

Memory for Learning: Top Ten Strategies - see here. Thanks to Claire Speck.

Behaviour for Learning: Top Ten Strategies - see here. Thanks to Jane Elsworth

Differentiation: Top Ten Strategies - see here. Thanks to Garry Littlewood.

Creativity and Challenge: Top Ten Strategies - see here. Thanks to Julie Watson

# data[].attributes.attachment_link.title
Questioning and Feedback: Top Ten Strategies

# data[].attributes.attachment_link.description
As part of our whole staff training at Huntington School we have been sharing ideas and collating 'Top Ten Strategies'. This list is the fruits of our labour: 1. Differentiated questioning. Given the time we take doing it daily, effective questioning may well be the highest impact strategy we can employ.

# data[].attributes.body
Comparing 5 major sources of evidence

# data[].attributes.attachment_link.readable_text
Mike Bell taught science in UK secondary schools and then became interested in evidence. He now runs EBTN: the Evidence Based Teachers Network. About 7000 teachers receive their newsletter. In this blog post, Mike Bell suggests that there is sufficient consensus among those educationalists who look at the evidence to say that we now know how learning happens, why some students struggle and how to improve learning for any learner. By combining lists of effective methods derived from both classroom experiments and psychology and then checking them with the neuroscience to provide a brain-based explanation, it is now possible to implement the evidence in a simple, six-step process.
The experience of teachers

Karen and Kevin are both trainee teachers. They spend part of their week in college (which includes lectures from education department staff), and part in their placement schools (where they both have experienced teachers as mentors). They attend the same college, but different schools.

This week they hear a lecture on 'Learning Styles'. The lecturer explains the theory and even puts up diagrams of the brain showing the different areas used for visual, auditory and kinaesthetic learning. It's all very convincing and both decide to try the ideas in their lessons later that week.

They both discuss the idea with their school mentor. Karen's mentor is dismissive. "Are they still teaching that stuff!? Don't they know it's disproved?" Karen quickly does a Google search and finds a Wikipedia page with links to the research. Yes, it been debunked. Kevin's mentor listens and agrees that he should try it. "I used to use that myself, but haven't for a while. I know that some of my students are visual learners because they struggle with reading and diagrams are better."

Both trainees attend a staff training session at their schools. One school is single sex selective while the other is mixed comprehensive. At each school the principal extols the virtue of their school type and make a persuasive case for it. Kevin and Karen are both convinced - but in the opposite direction.

Throughout their training and teaching they find the same pattern.

Kevin reads a research paper which claims that a particular method is effective. His college lecturer warns against using individual studies and suggests research-reviews, or meta-analyses which combine the results from several experiments. Karen's lecturer warns against the use of meta-analyses as we cannot be sure all the combined methods were actually the same.

They both watch a TV news magazine program. The education minister is outlining a new policy. She uses the same methods they had heard in their lectures: a well-reasoned argument. In response, the program gets reactions from leading 'experts' - but, again, they profoundly disagree. The union leader profoundly disagrees. The professor resolutely agrees. Not only the minister but also the journalist and even parents act as though they are more expert than the teacher.

The experience of other professions

How different this experience is from that of nearly every other profession. Engineers, archaeologists, midwives etc are all respected for their expertise. Government tends to consult them rather than direct their day-to-day activities. If a disagreement arises, these professionals have a mechanism to resolve the dispute and arrive at a general consensus. They look to research and discuss the evidence, not their opinion.

Anita and Amrita are midwives in the UK and India. Despite their geographical and cultural separation, they both study their midwifery from the same textbook.

The need for consensus

Why are the judgements of most professions respected while teachers are treated so badly? Is there a connection between the lack of shared ideas and the lack of respect? Are the politicians and journalists exploiting the lack of shared ideas in the teaching profession? What would it take to become respected?

The main difference between teaching and many respected professions is not just that we disagree, it is that there is no shared understanding of what we are doing. We lack a shared 'theory of learning' to underpin our thinking. This theory creates the questions which researchers then try to solve. The reason midwives Anita and Amrita agree is because they share a theoretical understanding of the birthing process and its complications.

The emerging consensus

The good news is that all the basic ingredients for a shared theory (and from there a respected profession) already exist. Among those who look at the evidence there is very little disagreement. Blog posts by the Learning Scientists do not clash with the output from the Education Endowment Foundation (EEF). Hattie's Visible Learning list is quite compatible with the EEF and other evidence compilations.

There are three main sources for the shared theory:

Educational neuroscience (which explains what is happening in the brain)

Educational psychology (where experiments are done in laboratories)

Classroom evidence (where the methods are tried by teachers with regular students)

Taken individually, each of the three has problems. Most scientists agree that we are very far from being able to say "The neuroscience shows...., therefore, in your classroom you should ...". We may never get there without the usual extensive trails. Psychology experiments are in artificial environments and so we may not be able to translate the results into the classroom. Classroom experiments are problematic for the reasons Karen and Kevin found above.

But, what if we stop relying on one source and apply the evidence only if it is backed up by another? What if we only implement a classroom method if it can be explained by neuroscience? What if we don't implement psychology research unless it has had classroom trials?

At first this may sound far-fetched. "Surly it'll take decades to do all that extra research?" I hear you say. The good news is - we don't have to wait; there already exists enough cross-referencing for us to explain the learning process, suggest effective ways to improve it, understand why some students struggle (in different ways) and have ways to help them.

Examples:

1. We know from psychology that spaced repetitions work better than repetitions massed at the end of the topic. We also know from neuroscience that long-term memories are formed when the links are made permanent by a process called 'long-term potentiation' - and this requires spaced repetitions (listen to this podcast episode for more on spaced practice).

2. We know from classroom experiments that linking the new knowledge to what the student already knows is effective way to improve learning. This is supported by the neuroscience understanding that memories are simply links to the component parts - not separate memories (as they may be on a computer) (see here).

Further support for the idea that we already have a broad consensus is the way so many bloggers, gurus and conferences are saying very similar things. A consensus is already available. Let's join in.

EBTN has worked to compile the evidence into an easy-to-implement order. By combining several different lists of effective methods, derived from classroom psychology experiments and checking them against the brain-science we have come up with a six-step process. The easiest way to see this is to look at the Evidence sources section of our website.

# data[].attributes.attachment_link.description
In this blog post, Mike Bell suggests that there is sufficient consensus among those educationalists who look at the evidence to say that we now know how learning happens, why some students struggle and how to improve learning for any learner.

# data[].attributes.headline
Teachers developing assessment for learning

# data[].attributes.subject
Teachers developing assessment for learning

# data[].attributes.body
Impact on student achievement

# data[].attributes.attachment_link.readable_text
Abstract

While it is generally acknowledged that increased use of formative assessment (or assessment for learning) leads to higher quality learning, it is often claimed that the pressure in schools to improve the results achieved by students in externally‐set tests and examinations precludes its use. This paper reports on the achievement of secondary school students who worked in classrooms where teachers made time to develop formative assessment strategies. A total of 24 teachers (2 science and 2 mathematics teachers, in each of six schools in two LEAs) were supported over a six‐month period in exploring and planning their approach to formative assessment, and then, beginning in September 1999, the teachers put these plans into action with selected classes. In order to compute effect sizes, a measure of prior attainment and at least one comparison group was established for each class (typically either an equivalent class taught in the previous year by the same teacher, or a parallel class taught by another teacher). The mean effect size in favour of the intervention was 0.32.

Introduction

Reviews of research by Natriello () and Crooks () and more recently by Black and Wiliam (a) have demonstrated that substantial learning gains are possible when teachers introduce formative assessment into their classroom practice. It is also clear from these reviews, and from other studies (see Black & Atkin, ) that achieving this is by no means straightforward. As Black and Wiliam (b) point out, these changes are hard to implement even in ideal conditions:

Thus the improvement of formative assessment cannot be a simple matter. There is no 'quick fix' that can be added to existing practice with promise of rapid reward. On the contrary, if the substantial rewards of which the evidence holds out promise are to be secured, this will only come about if each teacher finds his or her own ways of incorporating the lessons and ideas that are set out above into her or his own patterns of classroom work. This can only happen relatively slowly, and through sustained programmes of professional development and support. This does not weaken the message here-indeed, it should be a sign of its authenticity, for lasting and fundamental improvements in teaching and learning can only happen in this way. (p. 15, emphasis in original)

However, the introduction of high‐stakes state‐mandated testing, such as now exists in England and most states in the USA, makes the effective implementation of formative assessment even more difficult. This is because, although work by Nuthall and Alton‐Lee () has shown that teaching for understanding, rather than rote recall, results in better long‐term retention, attempts to maximize student and school scores appear to result in a lack of attention to the kinds of higher‐order thinking involved in formative assessment (Paris et al., ). Indeed, it appears as if there is a widespread belief that teaching well is incompatible with raising test scores.

There have been some studies that have shown that the use of higher‐order goals is compatible with success, even when attainment is measured in such narrow terms as scores on external tests. In a three‐year study of schools in the mid‐west of the USA, Newmann et al. () found that students whose teachers used authentic classroom tasks (defined as requiring construction, rather than reproduction of knowledge, disciplined inquiry, and value beyond school) out‐performed students not given such work, and that the size of the effects (as measured by standardized effect size) was substantial. In reading, writing and mathematics, the standardized effect sizes were 0.43, 0.52 and 0.64 respectively, with significant aptitude‐treatment interactions favouring high‐achievers in reading and low‐achievers in mathematics.

In another three‐year study of two secondary (11-16) schools in England, Boaler () compared two schools. One school (Phoenix Park) used a 'reform' approach to the teaching of mathematics, emphasizing higher‐order thinking, and students' responsibility for their own learning, while the other (Amber Hill) used a 'traditional' approach emphasizing practice of test items. Although matched in terms of prior achievement, students at Phoenix Park outperformed those at Amber Hill in the national school‐leaving examination (the General Certificate of Secondary Education, or GCSE) by, on average, one third of a grade, equivalent to a standardized effect size of 0.21.

These studies are useful in pointing out that attention to higher‐order goals in teaching can result in higher attainment, even when such attainment is measured principally in terms of lower‐order goals. However, since these studies were not based on direct experiments, there is always the possibility that, in Newmann et al.'s () study, the teachers using more authentic activities were just better teachers, and that the choice of authentic activities was incidental to their success. Similarly, in Boaler's () study, it could be that the teachers teaching at Phoenix Park were just better teachers, drawn to the school by its progressive ethos.

In order to draw clear policy implications regarding the utility of formative assessment, we therefore decided that it was necessary to undertake a more direct experiment, in which the confounding of variables, while not being entirely removed, was reduced, by asking teachers to incorporate formative assessment (or assessment for learning as it is sometimes called) into their classroom practice, and comparing the performance of their students with those of other classes at the same school. This work was undertaken in the King's‐Medway‐Oxfordshire Formative Assessment Project (KMOFAP), funded initially by the Nuffield Foundation (as the Developing Classroom Practice in Formative Assessment project) and subsequently by the United States National Science Foundation through their support of our partnership with the Stanford Classroom Assessment Project to Improve Teaching And Learning (CAPITAL; NSF Grant REC‐9909370).

Research strategy

The central tenet of the research project was that if the promise of formative assessment was to be realized, traditional research designs-in which teachers are 'told' what to do by researchers-would not be appropriate. This is not because teachers somehow fail accurately to put into practice the prescriptions of researchers, but because the general principles emerging from the research underdetermine action-put simply, they do not tell you what to do.

Teachers will not take up attractive sounding ideas, albeit based on extensive research, if these are presented as general principles which leave entirely to them the task of translating them into everyday practice-their classroom lives are too busy and too fragile for this to be possible for all but an outstanding few. What they need is a variety of living examples of implementation, by teachers with whom they can identify and from whom they can both derive conviction and confidence that they can do better, and see concrete examples of what doing better means in practice. (Black & Wiliam, Black PJ Wiliam D (1998b) 1998 Inside the black box: raising standards through classroom assessment (London, King's College London School of Education) [Google Scholar] b, pp. 15-16)

This difficulty of 'putting research into practice' is not the fault of the teacher. But nor is it a failing in the research. Because our understanding of the theoretical principles underlying successful classroom action is weak, research can never tell teachers what to do. Indeed, given the complexity of classrooms, it seems likely that the positivist dream of an effective theory of teacher action-which would spell out the 'best' course of action given certain conditions-is not just difficult and a long way off, but impossible in principle (Wiliam, ).

For these reasons we decided that we had to work in a genuinely collaborative way with a small group of teachers, suggesting directions that might be fruitful to explore, and supporting them as well as we could, but avoiding the trap of dispensing 'tips for teachers'. At first, it seems likely that the teachers did not believe this. They seemed to believe that the researchers were operating with a perverted model of discovery learning in which the researchers knew full well what they wanted the teachers to do, but didn't tell them, because they wanted the teachers 'to discover it for themselves'. However, after a while, it became clear that there was no prescribed model of effective classroom action, and each teacher would need to find their own way of implementing these general principles in their own classrooms.

The sample

We began by selecting two local education authorities (LEAs) where we knew there was support from the authority for attempting to develop formative assessment, and, just as importantly, where there was an individual officer who could act as a link between the research team and the schools, and provide a local contact for ad hoc support for the teachers. In this regard, we are very grateful to Sue Swaffield from Medway and Dorothy Kavanagh from Oxfordshire who, on behalf of their authorities, helped to create and nurture our links with the schools. Their involvement in both planning and delivering the formal in‐service sessions, and their support 'on the ground' have been invaluable, and it is certain that the project would not have been as successful without their contributions.

Having identified the two authorities, we asked each authority to select three schools that they felt would be suitable participants in the project. We were very clear that we were not looking for 'representative' or typical schools. From our experiences in curriculum development-for example in graded assessment (Brown, )-we were aware that development is very different from implementation. What we needed were schools that had already begun to think about developing assessment for learning, so that with these teachers we could begin to produce the 'living examples' alluded to earlier to use in further dissemination.

Each authority identified three schools that were interested in exploring further the possibility of their involvement, and three of us (PB, CH and DW) visited each school with the LEA officer to discuss the project with the head teacher and other members of the senior management team. All six schools identified agreed to be involved. Brief details of the six schools are shown in Table 1 (the names of all schools and teachers are, of course, pseudonyms).

In our original proposal to the Nuffield Foundation, we had proposed to work only with mathematics and science teachers, partly because of our greater expertise in these subjects, but also because we believed that the implications for assessment for learning were clearer in these areas. In order to avoid the possible dangers of isolation, our design called for two mathematics and two science teachers at each school to be involved.

The choice of teachers was left to the school, and a variety of methods was used. In some schools, the heads nominated a head of department together with a teacher in their first or second year of teaching. In another school, in order to ensure a commitment to the project, the head teacher insisted that both the heads and deputies of the mathematics and science departments were involved. In other schools, teachers appeared to be selected because, in the words of one head, 'they could do with a bit of inset'. In the event, while our schools were not designed to be representative, there was a considerable range of expertise and experience amongst the 24 teachers selected-five of the teachers were heads of department, five were deputy heads of department and the remaining 14 occupied a range of positions within their schools, mostly at a relatively junior level.

The intervention

The intervention had two main components:

1.

2.

In our original proposal, we had envisaged a series of nine half‐day in‐service sessions, some, involving all the teachers, to be held in London, and others conducted in the LEA in order to reduce the teachers' travelling time. In the event, only one such LEA‐based session was held, because the teachers felt that they gained a great deal from working with teachers in the other authority. As a result, since most of the teachers would spend two or three hours travelling to reach London, all the inset sessions, apart from the one LEA‐based session, took the form of full‐day sessions (typically 10 a.m. to 4 p.m.). Although we paid the costs of travel and replacement teaching, since not all teachers could attend all the insets, the savings allowed a total of six‐and‐a‐half days' inset (rather than the proposed four‐and‐a‐half).

The pattern of insets is shown in Table 2 (subsequent insets were held as part of the NSF‐funded work on the CAPITAL project, but the data reported here relate to the original project, from January 1999 to August 2000.

The key feature of the inset sessions was the development of action plans. Since we were aware from other studies that effective implementation of formative assessment requires teachers to re‐negotiate the 'learning contract' (c.f. Brousseau, ) that they had evolved with their students, we decided that implementing formative assessment would best be done at the beginning of a new school year. For the first six months of the project, therefore, we encouraged the teachers to experiment with some of the strategies and techniques suggested by the research, such as rich questioning, comment‐only marking, sharing criteria with learners, and student peer‐assessment and self‐assessment. Each teacher was then asked to draw up, and later to refine, an action plan specifying which aspects of formative assessment they wished to develop in their practice and to identify a focal class with whom these strategies would be introduced in September 1999. Although there was no inherent structure in these plans (see below), the teachers being free to explore whatever they wished, we did find that they could be organized under the broad headings shown in Table 3. In all, the 24 teachers included a total of 102 activities in their action plans-an average of just over four each.

Most of the teachers' plans contained reference to two or three important areas in their teaching where they were seeking to increase their use of formative assessment, generally followed by details of strategies that would be used to make this happen. In almost all cases the plan was given in some detail, although many teachers used phrases whose meanings differed from teacher to teacher (even within the same school).

Practically every plan contained some reference to focusing on or improving the teacher's own questioning techniques although only 11 gave details on how they were going to do this (for example, using more open questions, allowing students more time to think of answers or starting the lesson with a focal question). Others were less precise (for example, using more sustained questioning of individuals, or improving questioning techniques in general). Some teachers mentioned planning and recording their questions. Many teachers also mentioned involving students more in setting questions (for homework, or for each other in class). Some teachers also saw existing national curriculum tests as a source of good questions.

Using comment‐only marking was specifically mentioned by nearly half the teachers, although only 6 of the teachers included it as a specific element in their action plans. Some of the teachers wanted to reduce the use of marks and grades, but foresaw problems with this, given school policies on marking of student work. Four teachers planned for a module test to be taken well before the end of the module thus providing time for remediation.

Sharing the objectives of lessons or topics was mentioned by most of the teachers, through a variety of techniques (using a question that the students should be able to answer at the end of the lesson, stating the objectives clearly at the start of the lesson, getting the students to round up the lesson with an account of what they had learned). About half the plans included references to helping the students understand the marking criteria used for investigative or exploratory work, generally using exemplars from students from previous years. Exemplar material was mentioned in other contexts such as having work on display and asking students to mark work using a set of criteria provided by the teacher.

Almost all the teachers mentioned some form of self‐assessment in their plans, ranging from using red, amber or green 'traffic lights' to indicate the student's perception of the extent to which a topic or lesson had been understood, to strategies that encouraged self‐assessment via targets which placed responsibility on students (e.g., 'One of these twenty answers is wrong: find it and fix it!'). Traffic lights (or smiley faces-an equivalent that did not require coloured pens or pencils!) were seen in about half of the plans and in practically all cases their use was combined with strategies to follow up the cases where the students signalled incomplete understanding. Several teachers mentioned their conviction that group work provided important reinforcement for students, as well as providing the teacher with insights into their students' understanding of the work.

We were interested in whether the choices of activities by the different teachers showed any structure (e.g. do particular combinations of strategies occur together?). However, use of cluster analysis and multidimensional scaling (Schiffman et al., ) revealed no tendency for particular strategies to be found together. In this sense, the strategies and techniques appear to be relatively independent of one another.

The other component of the intervention, the visits to the schools, provided an opportunity for project staff to discuss with the teachers what they were doing, and how this related to their efforts to put their action plans into practice. The interactions were not directive, but more like a holding up of a mirror to the teachers. Since project staff were frequently seen as 'experts' in either mathematics or science education, there was a tendency sometimes for teachers to invest questions from a member of the project team with a particular significance, and for this reason, these discussions were often more effective when science teachers were observed by mathematics specialists, and vice‐versa.

We aimed for each teacher to be observed at least once each half term, although releasing teachers to discuss their lessons either before or afterwards was occasionally a problem (and schools that had guaranteed teacher release for this purpose at the beginning of the project were sometimes unable to provide for it).

A detailed description of the qualitative changes in teachers' practices is beyond the scope of this paper (see Black et al., , for a full account), but it is worth noting here that the teachers' practices were slow to change, and that most of the changes in practice that we observed occurred towards the end of the year, so that the actual size of the effects found are likely to be underestimates of what could be achieved when teachers are emphasizing formative assessment as an integral part of their practice.

Research design

Given the nature of the intervention, which was designed to build on the professionalism of teachers (rather than imposing a model of 'good formative assessment' on them), we felt that to utilize a traditional research design on the teachers would have been inconsistent. Furthermore, it would have been impractical. Since each teacher was free to choose which class would be the focus for this work, there was no possibility of standardizing either the 'input' or 'output' variables. For this reason, the collection of empirical quantitative data on the size of effects was based on an approach which we have termed 'local design'. Drawing more on interpretivist than positivist paradigms, we sought to make use of whatever assessment instruments would have been administered by the school in the normal course of events. In many cases, these were the results on the national tests for 14‐year‐olds or the grades on the national school‐leaving examination (the GCSE), but in some cases we made use of scores from school assessments (particularly in science, where modular approaches meant that scores on end‐of‐module tests were available).

Using externally mandated tests and examinations as 'input' and 'output' variables has both weaknesses and strengths. On the minus side, such tests might lack curricular validity (McClung, ) in that they may not accurately reflect what the teachers were teaching in their classrooms. On the other hand, to require teachers to develop additional assessments specifically related to what they had been teaching would have been an unacceptable addition to their already heavy workloads. Nor would providing our own assessments have been a satisfactory solution, since this would immediately raise questions of whether they captured what the teachers were trying to achieve. Furthermore, all the teachers were happy with the 'output' variables we had suggested as satisfactory measures of what they were trying to achieve in their classrooms, suggesting a considerable degree of 'alignment' between their teaching and the assessments used (although it is worth noting here that the teachers were critical of these assessments, because they felt that the assessments did not assess the important aspects of the subject). While the use of external tests therefore raises many issues, we do not think that any other approach would have been appropriate.

For each focal class we therefore had a focal variable (that is, dependent variable or 'output') and, in all but a few cases, we also had reference variables (that is, independent variables or 'inputs'). In order to be able to interpret the outcomes we discussed the local circumstances in the school with each teacher and set up the best possible comparison group consistent with not disrupting the work of the school. In some cases this was a parallel class taught by the same teacher in previous years (and in one case in the same year). In other cases, we used a parallel class taught by a different teacher and, failing that, a non‐parallel class taught by the same or different teacher. We also made use of national norms where these were available. In almost all cases, we were able to condition the focal variable on one or more reference variables, although in some cases the reference variables were measures of general ability (e.g. the National Foundation for Educational Research-NFER's Cognitive Abilities Test) while in others they were measures of achievement in that subject (e.g. end‐of‐year‐8 tests).

In order to be able to compare the results, raw differences between experimental and comparison groups were standardized by dividing by the pooled standard deviation of the experimental and comparison scores. This measure, called either the 'standardized effect size', or sometimes just 'effect size', and denoted d provides a way of comparing experimental results achieved with different measures (although see the discussion of problems in interpreting effect sizes below).

Results

Of the 24 teachers originally selected, 22 remained part of the project until its conclusion in July 2000. Peter from Brownfields School formally withdrew from the project and Lisa left Riverside School, to be replaced by Patrick. However, several teachers left their schools at the end of the project, and reliable data were available for only 19 teachers, four of whom had decided to have two focal classes each, resulting in data on 23 classes. For two of the classes (Nancy and James) there were two possible comparison groups. In the case of James, the effects are comparable ( d=0.29 and d=0.38). However, in the case of Nancy, comparison with another teacher (actually Patrick, who was not originally part of the study) yields a negative effect ( d=−0.31) while a comparison with a similar set taught in the previous year by Nancy yields a very large positive effect ( d=1.15). For reasons of completeness, both results for Nancy and James have been included, giving a total of 25 effect sizes, which are shown in Table 4, and summarized in stem‐and‐leaf form in Figure 1.

As can be seen, the majority of effect sizes are around 0.2 to 0.3, with a median value of 0.27. Given the fact that each of these results is a separate, 'mini‐experiment', care needs to be taken in drawing any general conclusions about the net effect of the adoption of formative assessment (see 'Discussion' below). The mean effect size is 0.34, but is clearly influenced by some extreme values, and since the effect sizes are not normally distributed, the jack‐knife procedure recommended by Mosteller and Tukey () was used to provide an estimate of the true mean effect as 0.32 and a 95% confidence interval of the true effect size as (0.16, 0.48).

In order to examine the relationship between a teacher's practice and the effect sizes, we classified teachers into one of four groups, according to their use of formative assessment strategies in their classrooms, as shown in Figure 2.

These characterisations had emerged from our observations of each teacher's practice, and were based on their use of key strategies during the main period of the project. Independent classification of the 24 teachers by two of us (CH and CL) produced identical classification for all but two teachers, and these were resolved after discussion. These classifications were produced before the results were known. The effect sizes by teacher type are shown in Table 5. Although there is no obvious trend in terms of average effect size, as one moves from less to more expert teachers, the interquartile range of effect sizes reduces, indicating further support for the attribution of the effects to the quality of formative assessment.

An analysis of the effects by different forms of comparison group in the form of side‐by‐side stem‐and‐leaf diagrams ( Figure 2) shows that no significant difference in effect sizes for the different form of comparisons is apparent.

There was no difference in the mean effect size for groups of different ages, although it is worth pointing out that the year 11 focal groups-where the 'output' measure was the grade on the GCSE examination-all had positive effect sizes. There was no systematic variation in effect size with the ability of the focal class although an analysis by subject shows that all the negative effect sizes were found for the mathematics groups (the median effect sizes for the mathematics and science groups were, however, almost identical).

Discussion

By its very nature, the quantitative evidence provided here is difficult to interpret. The comparisons are not equally robust. In some cases, we have comparisons with the same teacher teaching a parallel class in previous years, which, in terms of the main question (that is, has the intervention had an effect?) is probably the best form of comparison. In other cases, we have comparisons with a different teacher teaching a parallel set, so it could be that in some cases a positive effect indicates only that the teacher participating in the project is a better teacher than the teacher teaching the comparison class. In other cases, the comparison class is another class (and sometimes a parallel class) taught by the same teacher, and while there are examples of positive effect sizes here (in the case of Robert, for example) it is also reasonable to assume that the observed size of such effects will be attenuated by what we have termed 'uncontrolled dissemination'. Indeed, while two of our teachers did view involvement in the project as a short‐term commitment (after which they would return to teaching 'normally') for the vast majority of our teachers, involvement in the project has not just spread to all their classes, but has fundamentally altered their views of themselves as professionals. In some cases, the only comparisons available were the classes of different teachers teaching non‐parallel classes, and given the prevalence of ability grouping in mathematics and science, and its effect on achievement (see Wiliam & Bartholomew, ), disentangling the effect of our interventions from contextual factors is quite impossible.

In particular, the problematic nature of the comparison groups makes interpretation of the four negative effects difficult. The case of Nancy has been discussed above. For Lily, an inexperienced female teacher teaching in an all‐boys school, the only comparisons possible were with more experienced, male teachers, and therefore a negative effect is not surprising (and although this may seem like special pleading, it is our belief, from observations of her teaching, that Lily did improve substantially in her teaching during the project). We could also attempt to 'explain away' the negative effects for the two mathematics teachers at Century Island School by citing their limited attendance at the in‐service sessions. Such exclusions would be warranted if we were seeking to establish the effectiveness of formative assessment, since their engagement with formative assessment was, in fact, very limited. However, since our focus in this paper is not whether formative assessment is effective in raising achievement (because there is significant research in existence to show that it is), but on how to support teachers in developing their practice, we believe that it is appropriate to include these results.

Although a variety of measures were used as inputs and outputs, the very fact that these were either national tests and examinations, or assessments put in place by the school, gives us a degree of confidence that these measures have some validity in terms of what the teachers were trying to achieve. There is also a problem inherent in the standard definition of standardized effect size in the literature (Glass et al., for example). While standardized effect sizes are more comparable than raw scores, and allow different assessments to be placed on a common metric, there are nevertheless significant problems of interpretation. Dividing the difference between comparison and experimental groups by the pooled standard deviation clearly makes sense in that an improvement of 5 marks from (say) 60 to 65 represents a huge improvement if the standard deviation is 5 marks, but only a modest improvement if the standard deviation is 20 marks. However, the same logic dictates that an average improvement of half a grade per student at GCSE is to be regarded as a bigger effect if it is achieved in a top set where the standard deviation is one grade ( d=0.5), than if it is achieved in a mixed‐ability class where the standard deviation is two GCSE grades ( d=0.25).

There is also the question of what lessons can be drawn from these six schools for policy and practice elsewhere. After all, the schools in this study were not typical, in that they had identified themselves as interested in exploring the development of formative assessment, and they were given a much greater degree of support than is available to most teachers. In response, we can only agree, and repeat the usual researchers' litany that 'more research needs to be done'. However, in defence of the idea that further research in this area is worth undertaking, we would make two points.

The first is that while the schools were a selective sample, the teachers were much less so, representing a range of expertise and experience, and almost all the teachers appear to have improved. Furthermore, these teachers have generated a series of 'living examples of implementation' that have served to make it easier to introduce these ideas to other teachers (Black Black P Harrison C Lee C Marshall B Wiliam D (2002) et al., 2002 Working inside the black box (London, King's College London Department of Education and Professional Studies) ). The experience of the participating teachers is already being built upon in their schools and Local Education Authorities (and more broadly-see Black & Wiliam, ), but of course it remains to be seen to what extent this work can be scaled up to an LEA or a country.

The second relates to the cost of the support. We estimate that the cost of providing the support (as opposed to researching its effects) was around £2000 ($3000) per teacher or approximately 8% of the salary costs for one teacher for one year. While this is much more than most schools have available per teacher for professional development, it is a relatively small proportion of the annual cost of each teacher (especially if, as appears to be the case, this is a one‐off, rather than a recurrent cost).

In conclusion, despite the cautions noted above, we believe that the results presented here provide firm evidence that improving formative assessment does produce tangible benefits in terms of externally mandated assessments (such as key stage 3 tests and GCSE examinations in England). Placing a quantitative estimate on the size of the effect is difficult but it seems likely that improvements equivalent to approximately one‐half of a GCSE grade per student per subject are achievable. While these improvements might sound small, if replicated across a whole school, they would raise the performance of a school at the 25th percentile of achievement nationally into the upper half. At the very least, these data suggest that teachers do not, as is sometimes reported, have to choose between teaching well and getting good results.

Notes on contributors

Dylan Wiliam took up the post of Director of the Learning and Teaching Research Center at ETS in September 2003. Before that he was Professor of Educational Assessment at King's College London, which he joined after teaching mathematics and science in secondary schools in London. At King's, he worked on various research projects related to assessment, including Graded Assessment in Mathematics, the Consortium for Assessment and Testing in Schools, as well as teaching on PGCE, Masters, Ed.D. and Ph.D. programmes.

Paul Black is Emeritus Professor at King's College London. During his career he has been involved in a range of Nuffield curriculum projects and in many research projects, mainly in science education and assessment. In 1987-8 he chaired the task group (TGAT) that advised ministers on the new national assessment and testing policy. Since his retirement he has concentrated on the study of formative assessment.

Chris Harrison taught science in several schools in the London area before she joined King's College in 1993. She now spends part of her time working with trainee teachers. Her work in formative assessment at King's has led to several research and professional development projects in the UK and abroad at both primary and secondary level. She has also worked as Assessment for Learning consultant on national projects in England, Scotland and Jersey.

Clare Lee was the research fellow on the KMOFAP project at King's, and had previously taught secondary mathematics for over twenty years in several schools. She has also worked with teachers on Action Research projects based at Oxford University. Currently she is Teacher Advisor for Assessment for Warwickshire working on many Assessment for Learning projects at primary and secondary level within the LEA.

School

Abbreviation

Description

Category

Activity

Frequency

School

Subj

Teacher

Yr

Set

n

Focal variable

Reference variables

Comparison group

n

SD

Raw effect

d

p

Key Focal variables KS3 Key Stage 3 tests SBn School‐produced test at beginning of year n SEn School‐produced test at end of year n Reference variables Cn CAT scores in year n SEn School produced tests at end of year n Comparisons I Parallel set taught by same teacher in same year S Similar set taught by same teacher in previous year P Parallel set taught by different teacher in same year L Similar set taught by different teacher in previous year D Non‐parallel set taught by different teacher in same year N National norms *Non‐representative comparison

Group

Count

Median

H‐spread*

Experts

7

0.25

0.07

Moving pioneers

10

0.31

0.25

Static pioneers

2

1.38

0.35

Triallers

6

0.15

0.64

*The 'hinge‐spread' or 'H‐spread' is analogous to the inter‐quartile range, and tends to it as the number of data‐points increases, but is more easily interpreted for small samples. See Tukey ( Tukey JW (1977) 1977 Exploratory data analysis (Reading, MA, Addison‐Wesley) [Google Scholar] ) for details.

Notes

This is a revised version of a paper presented at the 27th annual conference of the British Educational Research Association, University of Leeds, September 2001. In revising the paper, we have taken into account the comments of two anonymous reviewers, whose help we gratefully acknowledge, although we are entirely responsible for any remaining errors.

References

Black P Harrison C Lee C Marshall B Wiliam D (2002) Working inside the black box (London, King's College London Department of Education and Professional Studies)

Black P Harrison C Lee C Marshall B Wiliam D (2003) Assessment for learning: putting it into practice (Buckingham, Open University Press)

Black, P and Wiliam, D . (2003). In praise of educational research: formative assessment. British Educational Research Journal, 29(5): 623-637.

Black PJ Atkin JM (Eds) (1996) Changing the subject: innovations in science, mathematics and technology education (London, Routledge)

Black, PJ and Wiliam, D . (1998a). Assessment and classroom learning. Assessment in Education: principles, policy & practice, 5(1): 7-73.

Black PJ Wiliam D (1998b) Inside the black box: raising standards through classroom assessment (London, King's College London School of Education)

Boaler J (2002) Experiencing school mathematics: traditional and reform approaches to teaching and their impact on student Learning (Mahwah, NJ, Lawrence Erlbaum Associates)

Brousseau G (1984) The crucial role of the didactical contract in the analysis and construction of situations in teaching and learning mathematics in: H‐G. Steiner (Ed.) Theory of mathematics education: ICME 5 topic area and miniconference (Bielefeld, Germany, Institut für Didaktik der Mathematik der Universität Bielefeld) 110 119

Brown ML (Ed.) (1988) Graded Assessment in Mathematics development pack: teacher's handbook )Basingstoke, Macmillan)

Crooks, TJ . (1988). The impact of classroom evaluation practices on students. Review of Educational Research, 58(4): 438-481.

Glass GV McGaw B Smith M (1981) Meta‐analysis in social research (Beverly Hills, CA, Sage)

McClung, MS . (1978). Are competency testing programs fair? Legal?. Phi Delta Kappan, 59(6): 397-400.

Mosteller FW Tukey JW (1977) Data analysis and regression: a second course in statistics (Reading, MA, Addison‐Wesley)

Natriello, G . (1987). The impact of evaluation processes on students. Educational Psychologist, 22(2): 155-175.

Newmann FM Bryk AS Nagaoka JK (2001) Authentic intellectual work and standardized tests: conflict or coexistence? (Chicago, IL, Consortium on Chicago School Research)

Nuthall, G and Alton‐Lee, A . (1995). Assessing classroom learning: how students use their knowledge and experience to answer classroom achievement test questions in science and social studies. American Educational Research Journal, 32(1): 185-223.

Paris, SG , Lawton, T , Turner, J and Roth, J . (1991). A developmental perspective on standardised achievement testing. Educational Researcher, 20(5): 12-20.

Schiffman SS Reynolds ML Young FW (1981) Introduction to multidimensional scaling: theory, methods, and applications (New York, Academic Press)

Tukey JW (1977) Exploratory data analysis (Reading, MA, Addison‐Wesley)

Wiliam D (2003) The impact of educational research on mathematics education in: A. Bishop, M. A. Clements, C. Keitel, J. Kilpatrick & F. K. S. Leung (Eds) Second international handbook of mathematics education (Dordrecht, Netherlands, Kluwer Academic Publishers)

Wiliam, D and Bartholomew, H . (2004). It's not which school but which set you're in that matters: the influence of ability‐grouping practices on student progress in mathematics. British Educational Research Journal, 30(2): 279-293.

Additional information

Author information

# data[].attributes.attachment_link.title
Teachers developing assessment for learning: impact on student achievement

# data[].attributes.attachment_link.description
Abstract While it is generally acknowledged that increased use of formative assessment (or assessment for learning) leads to higher quality learning, it is often claimed that the pressure in schools to improve the results achieved by students in externally‐set tests and examinations precludes its use.

# data[].attributes.body
PRACTICE BRINGING INFORMATION TO MIND

# data[].attributes.attachment_link.title
Identify, support and fund learners with additional learning needs.

# data[].attributes.attachment_link.description
CognAssist is an innovative solution developed to enable learning providers to identify, support and fund learners with additional learning needs.

# data[].attributes.attachment_link.readable_text
Time - the Holy Grail of the teacher. There was never enough of it, and now there's arguably even less. This blog was inspired by a conversation with a great colleague over half term about how to teach effectively efficiently. Here are 12 time saving pedagogies:

Barak Rosenshine, Principles of Instruction: Research-Based Strategies That All Teachers Should Know argues that " the most effective teachers... understood the importance of practice, and they began their lessons with a five-to-eight minute review of previously covered material ."

Use your starter to ask students:

To retrieve key knowledge from last lesson

To retrieve key knowledge from last week

To retrieve key knowledge from last term

To retrieve key knowledge from last lesson and connect it to knowledge from last term

According to Rosenshine, "The more successful teachers did not overwhelm their students by presenting too much new material at once... the more effective teachers spent more time presenting new material and guiding student practice than did the less effective teachers. In a study of mathematics instruction, for instance, the most effective mathematics teachers spent about 23 minutes of a 40-minute period in lecture, demonstration, questioning, and working examples. In contrast, the least effective teachers spent only 11 minutes presenting new material. The more effective teachers used this extra time to provide additional explanations, give many examples, check for student understanding, and provide sufficient instruction so that the students could learn to work independently without difficulty."

Finally, don't waste time marking substandard work. Hattie and Timperley remind us that, "With inefficient learners it is better for a teacher to provide elaborations through instruction than to provide feedback on poorly understood concepts."

Rosenshine states that, "Students need to practice new material... The most successful teachers in these studies spent more than half of the class time lecturing, demonstrating, and asking questions. Questions allow a teacher to determine how well the material has been learned and whether there is a need for additional instruction. The most effective teachers also ask students to explain the process they used to answer the question, to explain how the answer was found. Less successful teachers ask fewer questions and almost no process questions."

Tom Sherrington @headguruteacher suggests, " every time students give a verbal answer and before they are asked to write anything, ask them to re-form their initial responses into well-constructed sentences using the key words and phrases you've discussed. Do it relentlessly, every time.
What does the graph tell us?
* First attempt: It goes up.
* Second attempt: The speed on impact increases as the mass of the trolley increases."

Use your vocabulary to develop a growth mind-set:

Insist on the word 'yet'

Use 'what do you think?' to deflect questions back to the student

Rather than spending time at the end of the lesson trying to assess what students have learned, ask them what they have struggled with that lesson? What was hard? Why was it hard? Then use this to plan your next lesson. Not only will this make your next lesson more focused, but it also fosters the attitude that 'struggle is good'.

Dylan Wiliam reminds us that feedback should involve more work for the student than the teacher. Alex Quigley gives a warning about " feedback [that] no longer becomes purposeful for the learning of students, but instead becomes a visible indicator for inspection teams."

This strategy has saved my time with A Level History students. Before I used it, I found myself re-writing the same targets for students time and again as they forgot what I'd written on their last essay and didn't address my improvement strategies.

I write strategies for improvement on an essay

The student fills in the boxes below at the top of their next essay

I comment briefly on their progress with this improvement strategy. It's also useful to read how the student feels that they have addressed the improvement strategy

# data[].attributes.attachment_link.description
Time - the Holy Grail of the teacher. There was never enough of it, and now there's arguably even less. This blog was inspired by a conversation with a great colleague over half term about how to teach effectively efficiently. Here are 12 time saving pedagogies: 1.&a

# data[].attributes.body
Exploring the transformative potential of recognition of prior informal learning for learners

# data[].attributes.attachment_link.title
Exploring the transformative potential of recognition of prior informal learning for learners: A case study in Scotland

# data[].attributes.attachment_link.description
This article presents an overview of the findings of a recently completed study exploring the potentially transformative impact upon learners of recognition of prior informal learning (RPL). The specific transformative dimension being reported is learner identity.

# data[].attributes.attachment_link.readable_text
Why is feedback and feed forward important?

"Conventionally, feedback is conceptualised as an issue of 'correction of errors' or 'knowledge of results'. Much more important is how the provision of feedback affects student learning behaviour - how feedback results in students taking action that involves, or does not involve, further learning."
Transforming the Experience of Students through Assessment (TESTA)

Feedback provides information to learners about where they are in relation to their learning goals so that they can evaluate their progress, identify gaps or misconceptions in their understanding and take remedial action. Generated by tutors, peers, mentors, supervisors, a computer, or as a result of self-assessment, feedback is a vital component of effective learning.

Feedback should be constructive, specific, honest and supportive.

While feedback focuses on a student's current performance, and may simply justify the grade awarded, feed forward looks ahead to subsequent assignments and offers constructive guidance on how to do better. A combination of both feedback and feed forward helps ensure that assessment has a developmental impact on learning.

Effective feedback should also stimulate action on the part of the student. The most effective practice treats feedback as an ongoing dialogue and a process rather than a product.

What are the common problems?

Timeliness

Feed forward can only be effective if it's timely ie, received at a point when meaningful action can be taken. High-stakes assessments are often set towards the end of a module, term or semester, reducing opportunities for students to apply any feedback they receive.

Regularity

Feedback needs to be quite regular and hence on relatively small chunks of course content to be useful. One piece of detailed feedback on an extended essay or design task after ten weeks of study is unlikely to support learning across a whole course very well.

Approaches to feedback

Academic staff can often ignore the need to discuss feedback approaches. The feedback given can then be inconsistent or weak in other ways ie, skewed towards a particular type of observation (such as praise) or short term and too focused on the assignment in hand rather than truly developmental.

A common Ofsted report observation in failing colleges is that tutors (and workplace assessors in the case of apprentices) don't provide feedback to students that helps them understand how they can improve.

Missed opportunities

Students often don't collect or read feedback. This is sometimes because it arrives too late to be useful. In some cases the problem is as simple as the students not realising the feedback is available.

Alternatively students can focus on the overall mark and not understand the benefits of making use of the feedback. We discuss this further in the section on returning marks and feedback where we suggest that disaggregating marks and feedback can encourage students to engage better with the feedback.

Passive learners

Students can often be passive recipients of feedback, viewing it as the tutor's role to deliver feedback to them, rather than understanding the need for them to engage in meaningful dialogue around the feedback to aid their development.

Lack of overview

It can often be difficult for both students and tutors (especially tutors with a pastoral or personal tutoring role) to gain an overview of feedback. This may be because feedback is online but stored at a module level or because it is in a more ephemeral format such as paper or verbal feedback.

Understanding feedback

Feedback that appears self-evident to tutors may be difficult for students to understand. There could be difficulties with the format eg, indecipherable hand writing or with how the feedback is expressed and contextualised.

The Open University asked a group of students to work through their feedback talking out loud about what they understood and what they didn't understand.

How might we use technology and what are the benefits?

Technology can :

Improve clarity about marking and feedback deadlines and provide students with a personalised schedule

Help students store and access feedback easily if it's provided in a digital format

Support individual learners or subjects where certain digital formats are more suitable eg, audio feedback for language courses

Help teachers make efficiency gains by using a feedback format that best suits them eg, a slow typist may provide better quality feedback by making an audio recording.

How does feedback and feed forward relate to the lifecycle?

This theme runs right through the lifecycle:

At the specifying stage think about your approach to feedback

During the setting stage ensure that the overall submission and marking schedules allow for timely feedback that can inform the next assignment. Infom students on how to make use of feedback and also provide formative opportunities

At the marking and production stage generate feedback for students

When returning marks and feedback adopt an approach that is most likely engage students

At the reflecting stage engage in dialogue with students about their feedback and reflect on the feedback you have given, how useful it's been and any changes you should make in the future.

What resources can help?

Our feedback and feed forward guide is a good starting point on the use of technology to support student progression. You can also look at these individual resources from the guide:

Our case study outlines the use of technology to improve feedback dialogue at the University of Dundee

Our video explores reconceptualising feedback:

Further useful resources

"Neglecting dialogue can lead to dissatisfaction with feedback. The transmission model of feedback ignores these factors and importantly the role of the student in learning from the feedback. Simply providing feedback does not ensure that students read it, understand it, or use it to promote learning."

# data[].attributes.attachment_link.description
Why is feedback and feed forward important?"Conventionally, feedback is conceptualised as an issue of 'correction of errors' or 'knowledge of results'.

# data[].attributes.body
Graphic Organisers

# data[].attributes.attachment_link.readable_text
As Alex Quigley points out , it can also be tempting to mistake correlation for causation: " This is of course of crucial importance in schools. We are constantly being sold silver bullets whose evidence is based on loose correlation (or worse) and nothing like causation ."

Strategies to accurately assess learning
David Didau has written extensively on the fact that learning is invisible .

Myth 2: Learning is intuitive

" From the moment we are born we make sense of the world through our five senses. However, neuro-linguistic programming (NLP) practitioners argue that those five senses may not contribute equally to that perception and that individuals may have a sensory preference for receiving and making sense of new information and ideas. They have identified three types of learner."

Many people do not fit one particular style

"The relationship between what people say about how they learn and how they actually learn is weak."

At least 71 learning styles have been identified.

Research by Clark found that "learner preference was typically uncorrelated or negatively correlated to learning and learning outcomes. That is, learners who reported preferring a particular instructional technique typically did not derive any instructional benefit from experiencing it."

Similarly, Kirschner, Sweller and Clark found that learners' preferences can be unhelpful: " Less able learners who choose less guided approaches tend to like the experience even though they learn less from it." In comparison, task-specific learning strategies embedded in instructional presentations "require explicit, attention-driven effort on the part of the learners and so tend not to be liked, even though they are helpful to learning." Conversely, " Higher aptitude students who chose highly structured approaches tended to like them but achieve at a lower level...[because they] have acquired implicit, task-specific learning strategies that are more effective for them than those embedded in the structured versions of the course...[but] believe that they will achieve the required learning with a minimum of effort. "

Daniel Willingham on Why Learning Styles Don't Exist
David Didau explodes Right Brain/ Left Brain Bollocks

Myth 3: Learning must be collaborative

It may not be as effective or efficient as individual work in ensuring all students think hard about the learning

Student-to-student interaction may be lower level than teacher-involved interaction

It may focus on proxies for learning such as engagement or student interaction

It may make assessment of individual learning more difficult

It may prevent conditions of desirable difficulty

David Didau on Should Group Work be Imposed?

Myth 4: Growth mindset alone is the answer

It's a bit more complicated than that. Astbury and Plomin point out in G is for Genes genetic as well as environmental reasons, it will be harder for some people to develop a growth mindset than others ."

Carl Hendrick warns against reductionist 'pop psychology' in " to be " that " promising areas" like Growth Mindset, which fails to take into account students' multidimensional self-concept and underestimates the impact of achievement on self-perception, which Muijs and Reynolds found stronger than the effect of self-concept on achievement." Similarly, David Didau warns growth mindset has been so universally heralded as a 'good thing' that it's in danger of becoming one of those memes we think with rather than about."

Nick Rose points out that school interventions designed to improve students' metacognition and 'character' can be problematic. A 2015 review on, The potential for school-based interventions that target executive function to improve academic achievement suggested that:
"There is surprisingly little rigorous empirical research that explores the nature of the association between executive function and achievement and almost no research that critically examines whether the association is causal. From the existing research it is not clear whether improving executive functioning skills among students would cause their achievement to rise as a result."
The review found that none of the skills-based approaches that it investigated appeared to directly improve student outcomes.

Scott Alexander critiques growth mindset
Nick Rose Growth Mindset: It's not magic
How to use Growth Mindset effectively: Growth Mindset - so What's Next? by Alex Quigley.

Myth 5: research alone is the answer

Again, it's more complicated than that. Tom Bennett argues that:

"T eachers need to interact with what the best evidence is saying and translate it through the lens of their experience... Teaching is not, and can never be a research-based, or research-led profession. Research can't tell us what the right questions to ask are, nor can it authoritatively speak for all circumstances and contexts... [but] it can assist bringing together the shared wisdom of the teaching community... Teaching can - and needs to be -research informed, possibly research augmented. The craft, the art of it, is at the heart of it. Working out what works also means working out what we mean by 'works', and where science, heart and wisdom overlap and where they don't ."

Daniel Willingham suggests that the main reason for a cognitive dissonance between teachers' belief in the validity of research and the techniques they use in the classroom, is fear of losing autonomy. However, Carl Hendrick cites that research in which teachers are involved is in fact " Stenhouse's argument the route to teacher emancipation."

What next and where? 8 ideas

As Andy Tharby points out , " What is most critical... is that we question, test and evaluate our assumptions and intuitions, whatever our position in the educational system - from trainee teacher to Secretary of State. Unless the evidence is incontrovertible (and it almost never is) always avoid the simplistic answer, however seductive it sounds."

Looking for evidence of our impact

Asking for evidence

Improving our teaching using research

Developing Research Lead roles in schools

4. Geoff Petty suggests how to use evidence to improve teaching

5. Highbury Grove School's vision for the research-engaged school

7. Keep up to date with research through twitter. The following blogs are great places to start:

8. Check out schools developing research-informed practice

# data[].attributes.attachment_link.description
As Alex Quigley points out , it can also be tempting to mistake correlation for causation: " This is of course of crucial importance in schools. We are constantly being sold silver bullets whose evidence is based on loose correlation (or worse) and nothing like causation ."

# data[].attributes.headline
What is ipsative assessment and why would I use it?

# data[].attributes.subject
What is ipsative assessment and why would I use it?

# data[].attributes.body
Question mark blog

# data[].attributes.attachment_link.readable_text
Posted by John Kleeman

As I'm writing this, I've just got back from the gym, where I beat my personal best distance on an exercise bike. What's this got to do with computerized assessment, you might ask? Hear me out.

You're probably familiar with norm-referenced testing and criterion-referenced testing :

A norm-referenced test compares a test-taker against his or her peers. For example, you might compare my results with those of my Questionmark colleagues. (If you did, then seeing how energetic many are in the gym, I suspect my performance would not compare well!)

A criterion-referenced test measures a test-taker against external criteria. For example, it might be that people of a certain age should be expected to reach a certain distance in a certain time on an exercise bike.

A third type is sometimes called ipsative assessment.

An ipsative assessment in an education/learning context compares a test-taker's results against his or her previous results. This is how I measure myself at the gym - I am pleased that I am doing better than I have before. I'm not worried if this meets some external criteria or if I'm better or worse than other people.

It's very common to use criterion-referenced tests as computerized assessments because they help us measure competence. If you want to be sure that your employees know the rules, if you want to validate a pilot to fly a plane, or if you want to check that someone has understood training, a criterion-referenced test is usually the way to go.

But an advantage of ipsative assessment is that it measures progress and development - a test-taker can see if he or she is improving and whether or not he/she is taking advantage of feedback from previous assessments. Using ipsative assessment can help all test-takers improve: A weaker performer will be encouraged by seeing performance improvements over earlier attempts, and a stronger performer can be challenged to do better. This can deal with the risks of the weaker performer becoming demotivated from a poor test result and the strong performer complacent from a good one. Ipsative assessment can be used for objective measures (e.g. did I get a better score?) and also for more subjective measures (e.g. am I more confident about something?).

Questionmark software makes it easy to produce coaching reports on each attempt at an assessment, and these can easily be used to allow test-takers to compare results from previous attempts and see how they've improved. This is particularly useful for observational assessments, which measure skill and performance - areas where everyone wants to improve and there can never be a perfect score.

To learn more on ipsative assessment in education and learning, one resource is this study by Dr Gwyneth Huges of the Institute of Education. (As a heads-up, the term ipsative measure is also used in a different, technical way in psychological testing as a within-person measure.)

Expertise is built up by deliberate practice, and being tested can help identify where that practice is needed. I think it's helpful for all of us to remember that progress and improvement is a useful thing to measure as well as achievement and competency.

# data[].attributes.attachment_link.title
What is ipsative assessment and why would I use it?

# data[].attributes.attachment_link.description
What is ipsative assessment and why would I use it? | Posted by John Kleeman As I'm writing this, I've just got back from the gym, where I beat my personal best distance on an exercise bike. What's this got to do with computerized assessment, you might ask? Hear me out.

# data[].attributes.attachment_link.readable_text
Excuse the cheap pun of a title. With all the public furore about the Daily Mail I couldn't help but be mindful of a tabloid headline! , if you didn't know, is an acronym. Apologies if you are tired of acronyms, but admit it, sometimes they are just plain useful. DIRT stands for ' Dedicated Improvement and Reflection Time '. It has become a useful short-hand for checking, drafting, proof reading and for labelling the age-old process of students spending crucial time on improving their work.

I originally came across its use in Jackie Beere's book, ' The Perfect OFSTED Lesson '. In the two years hence it has gradually become a staple feature in our English department. After a short presentation to the whole staff last year, it has begun to slowly spread as a language to harness time to enhance the standard of students' work as well as using it to give quality feedback. DIRTy work then is that old fashioned stuff - getting students working really hard to ensure that the standard of their work is the best it can be.

Time is the thing. Too often we spend a great deal of time marking student work and giving them excellent feedback to improve. Crucially, however, students often spend a relatively short amount of time scanning the feedback for a grade or a level and then move on. Similarly, with a draft of their work they give the feedback a cursory glance, but they hurry on with supposed improvements and make the same mistakes once more. DIRT is about redressing that issue.

When I came across the acronym is was at a similar time to reading Ron Berger's excellent book, ' An Ethic of Excellence '. Ron used the following to describe the essence of DIRTy work and its value:

"Most discussions of assessment start in the wrong place. The most important assessment that goes on in a school isn't done to students but goes on inside students. Every student walks around with a picture of what is acceptable, what is good enough. Each time he works on something he looks at it and assesses it. Is this good enough? Do I feel comfortable handing this in? Does it meet my standards? Changing assessment at this level should be the most important assessment goal of every school. How do we get inside students' heads and turn up the knob that regulates quality and effort" (P103, 'An Ethic of Excellence')

Essentially, DIRT is about having the highest expectations of students and them having the highest expectations of themselves.

This week, in our whole staff training on DIRT, we shined a light on the excellent practice already existing in different subject areas. In Food Technology they used booklets which scaffold student understanding helpfully, step by step, with the expectation that students act upon their feedback using DIRT - writing appropriate comments in response to teacher feedback. In History they have invested in a bulk purchase of green pens which signal a drafting and proof reading focus in their exercise books. Their books are regularly daubed in green and a clear signal is sent that improving their writing is a basic, but crucial expectation of their learning. For the History department, DIRTy time is about improving extended writing and ensuring students proof read their work automatically. In Maths there is a real focus upon reflection and responding to precision feedback. There is a forensic analysis of their own work, question by question, with DIRT providing the impetus for such reflection in real terms.

Through our experiences and the shared expertise across departments, I was able to define a top five tips list for DIRT.
1. Keep it focused.
If you simply hand back work to students and tell them to improve it all then the response will invariably less than successful! Other than the few students whose work was lazily hashed together at the last minute, most students' work is faulty because they don't know how to make it better. They need specific support and to avoid overloading students we need to focus in upon specific improvements to sir work.

In English, this often includes drafting and proof reading their extended writing. To ensure students can best manage their improvements, we would often narrow down the focus, such as reflecting upon spelling strategies and/or punctuation usage, then ask students to make improvements in these specific areas. Given this learning focus, students then need a really clear focus in terms of time. With clear task instructions, including timing and outlining exact expectations, students can be more focused in their DIRT time and considerably more effective.
2. Model and scaffold.
Once more, letting students loose for twenty minutes to overhaul their work to reach new heights doesn't simply happen. Chatter and disruption is more likely! A range of models and resources to scaffold their understanding are required. In English, our focus is mostly concentrated upon literacy standards. That requires the obvious scaffolded support of the teacher, but it can be supplemented by tools, such as a dictionary and thesaurus, literacy mats, their school planner etc.

Models of work, with specific strengths or weaknesses, are crucially effective in determining what Berger describes as the assessment going on inside the head of students. Seeing an outstanding exemplar of a particular genre of writing, for example, helps lessen the load and gives students a high standard to reach for with their work. Reviewing a faulty example, picking apart its flaws with the teacher, or improving upon a weak example of work also helps scaffold their understanding about what is required to improve their own work. DIRT time may seemly be about independent work, but in actuality there is a great of reliance upon scaffolded teacher expertise.
3. Targeted feedback.
If students are receiving regular quality feedback that is targeted and precise in each of their subject areas then cumulatively they should learn clear patterns regarding how they need to improve in specific subject areas as well as recognising common patterns. Marking is therefore crucial - it determines teacher planning and it can be a defining factor for successful DIRT.

The evidence about the importance of quality feedback is well founded. Put simply, feedback and DIRT are essential bedfellows. If we give great feedback, with specific targets to improve, then DIRT is the crucial next step to deal with that feedback. As a rule of thumb, we can expect students to spend twice their time reflecting on their feedback as we devoted to giving feedback. Otherwise, really, what is the point of marking work and giving oral feedback?

4. Make oral feedback matter.

Oral feedback is too often relegated as the ugly step-sister to written feedback; however, we shouldn't neglect using it. As proven for us in our 'one to one oral feedback' matched trail, oral feedback works. It is simply a staple of good teaching. If we establish a really clear focus for DIRT, with quality models, scaffolds and targets for improvement, then students should be sufficiently focused to allow the teacher to undertake quality 'one to one feedback' whilst DIRT is taking place.

We have moved to recording oral feedback sessions, such as using stamps, with students noting their feedback around the stamp, or using an A5 pro forma, to ensure that such feedback is recorded so that students can respond to it fully and to ensure no advice is simply lost because they have forgotten.
5. Exploit the power of peers.
Peer assessment is often derided or done badly, by both students and teachers. For students it can be a poor substitute for teacher feedback. What we can do is use peers as a positive support tool during DIRT time. If my year 8 group are working on spelling and punctuation, then working in pairs is often an effective strategy to lighten their load and help them make minor improvements.

Once more, any peer work needs real focus. If we have peers analysing the work of one another for spelling, punctuation and grammar improvements, then guidelines and expectations need to be explicit. Small details, like getting pairs to sign their feedback can be a small, but powerful way to get students to fully engage in the task. Any pairings of students needs to be carefully considered of course, like any good seating plan.
Useful resources:
The following blog posts helpfully relate to effective DIRTy work:

The inimitable 'Learning Spy', David Didau has a really useful DIRT archive: see here.

On 'making your marking policy a feedback policy', with an explanation of how we use DIRT in our English department: see here.

A post outlining a range of strategies for improving written feedback: see here.

A post detailed a range of useful oral feedback strategies: see here.

# data[].attributes.attachment_link.description
Excuse the cheap pun of a title. With all the public furore about the Daily Mail I couldn't help but be mindful of a tabloid headline! DIRT, if you didn't know, is an acronym. Apologies if you are tired of acronyms, but admit it, sometimes they are just plain useful.

# data[].attributes.attachment_link.readable_text
Dweck argues that, "] that It's not either-or. It's not nature or nurture, genes or environment.... People may start with different temperaments and different aptitudes, but it is clear that experience, training and personal effort take them the rest of the way". Similarly, Neisser found [page 11 "schooling itself changes mental abilities."

Doug Lemov argues that success can be encoded with " disciplined, deliberate, intelligent practice."

4. All learners are unique

Nuthall found in The Hidden Lives of Learners that the processes of learning are essentially the same for all learners. Differences emerge due to differing prior knowledge, motivation and individual experiences, meaning that around 30% of what each student learns in a lesson is unique to them.

5. Struggle is necessary

" Learning happens when people have to think hard" [Coe]. Over-guiding students prevents this: " The aim of classroom instruction is ultimate mastery not error free learning" [Rohrer]. Shaun Allison writes about keeping students in the 'struggle zone' between cognitive overload and lack of challenge.

6. Over-differentiation can be dangerous

Andy Tharby reflects on some dangers of differentiation, including differentiation as a " life sentence," while Harry Webb writes here: " isn't it rather dangerous to use [differentiation] to decide to not teach certain children certain things? Learning is messy with lots of looping back and leaping forward - by assuming a theoretical progression and then imposing that on students, we might decide not to teach them ideas that they really could

But what about Ofsted...

Jane Jones, Ofsted's National Lead for Mathematics, writes that " Ofsted does not have fixed expectations of...how teachers differentiate to meet pupils' learning needs... The national curriculum makes it clear that the majority of pupils are expected to move through the programmes of study at broadly the same pace ."

So what is effective differentiation...

"Differentiation does not mean that you must have tiered resources and tasks in every lesson. It does not mean you should have must-should-could learning objectives... Differentiation needs to be seen as the aggregation of the hundreds of subtly different interactions that you have with each of your students... real differentiation [is]: pushing, prodding, nudging, stretching...slow, subtle, nuanced, a step at a time, working around the class from lesson to lesson " Tom Sherrington .

35 effective differentiation strategies

Differentiation through planning

Reconsider differentiated learning goals: " For years, learning objectives up and down the country have been shaped around 'all...most...some.' This is potentially really dangerous I think, as it lowers our expectations of what students can achieve. Much better to have a single, challenging objective and use questioning, feedback and other means of support to encourage all students to aspire towards it " [Shaun Allison ].

1. Use 'fluid' rather than 'autopilot' differentiation: what differentiation is appropriate for mastery of the particular concept or skill being taught?

2. Group students based on their level of mastery of the concept or skill being taught so that they can access appropriate tasks, support and resources.

Differentiation through explanation

" Differentiation should... be about how the teacher helps all pupils in the class to understand new concepts and techniques. " Jane Jones HMI.

1. Increase guided practice: " The most successful teachers... spent more than half of the class time lecturing, demonstrating, and asking questions " [ Principles of Instruction ]

2. Create podcasts of explanations to differentiate through increased exposure

3. Use a range of differentiated explanation types:

Example sequences:The limits of the concept by negative examples
i. A range of positive examples of concept
ii.
iii. Minimally different examples

Elaborative interrogation

# data[].attributes.attachment_link.description
Dweck argues that, "] that It's not either-or. It's not nature or nurture, genes or environment.... People may start with different temperaments and different aptitudes, but it is clear that experience, training and personal effort take them the rest of the way". Similarly, Neisser found [page 11 "schooling itself changes mental abilities."

# data[].attributes.body
How to Promote Student Engagement in Inclusive Classrooms

# data[].attributes.attachment_link.readable_text
Students with learning disabilities are often educated in inclusive classrooms alongside their typically developing peers. Although differentiated small-group instruction is ideal for students with learning disabilities, whole-group instruction continues to be the predominant instructional model in inclusive classrooms. This can create major challenges for teachers as they aim to actively engage all students, including students with learning disabilities. There are variations of whole-group response strategies, however, that teachers can use to accommodate a range of individual student needs. Collecting formative assessment data during whole-group instruction also can inform instructional decision making.

To be successfully included in general education settings, students with learning disabilities must have a sense of belonging. Many teachers may find it challenging to actively engage all students, including students with learning disabilities who struggle with academic risk taking and perseverance, resulting in lower levels of participation and peer interactions ( Gurganus, 2007; McIntosh, Vaughn, Schumm, & Haager, 1994). Students with learning disabilities face challenges across subject areas due to deficits in organizational skills, higher-order thinking, working memory, retention, and making connections ( Alloway, 2011; Gurganus, 2007; Montague, Krawec, Enders, & Dietz, 2013). Despite the success of various research-based academic interventions, many students with learning disabilities require additional help understanding how to assess their own level of comprehension as well as how to approach active engagement in learning ( Montague, Warger, & Morgan, 2000).

Although small-group differentiated instruction has been demonstrated as particularly effective for students with learning disabilities ( Kim, Linan-Thompson, & Misquitta, 2012; Montague & Rinaldi, 2001), whole-group instruction continues to be the predominant instructional model in inclusive classrooms ( DiCarlo, Pierce, Baumgartner, Harris, & Ota, 2012). Given how often whole-group instruction is implemented, teachers should strive to incorporate strategies that increase opportunities for student participation, engagement, and self-evaluation. Teachers can use a continuum of strategies during whole-group instruction to maintain student engagement, gather information to inform future instruction, and monitor student progress. This continuum ranges from proactive strategies that promote engagement to reactive strategies in response to students who become disengaged ( Simonsen, Fairbanks, Briesch, Myers, & Sugai, 2008). Although the continuum of strategies is needed to address student learning and behavioral needs, proactive strategies are preferred because they maintain active learning and have potential to prevent problematic behaviors.

Teachers can use a continuum of strategies during whole-group instruction to maintain student engagement, gather information to inform future instruction, and monitor student progress.

Proactive strategies encourage active engagement of all students, including students with learning disabilities ( Reglin, Akpo-Sanni, & Losike-Sedimo, 2010). Specific examples of proactive strategies that can be implemented during whole-group instruction include proximity, high rates of opportunities to respond, high-probability requests, and choice making ( Wehby & Lane, 2009). For example, whole-group questioning-and-response systems include proactive strategies, such as high rates of opportunities to respond and high-probability requests, by prompting all students to participate through verbal, gestural, written, or digital modes of responding. Such proactive strategies are important during whole-group instruction in inclusive classrooms, where traditionally the number of opportunities to respond is limited due to prompting techniques that rely on individual student responses ( Maheady, Michielli-Pendi, Harper, & Mallette, 2006). Given that high-achieving students are more likely to volunteer responses than students with learning disabilities, finding ways to question all students can promote student interest in learning, activate prior knowledge, and improve comprehension in an inclusive manner ( Maheady et al., 2006).

When implemented effectively, proactive whole-group instruction can help students with learning disabilities feel they belong in the learning community and provide opportunities for peer interaction and active participation ( McLaughlin & Allen, 2009). Further, whole-group response systems create a method by which teachers can track student participation and measure current performance or understanding of all students at the same time through formative assessment. Formative assessment occurs during instruction, has low or no stakes attached to it, and is intended to inform teachers' future instructional decisions ( Cornelius, 2013). This type of assessment means more than simply tracking if students with disabilities are "on task." Teachers should also measure how students engage in their own learning during whole-group instruction. For example, teachers can monitor progress on students' individualized education program (IEP) goals and evaluate their learning of curriculum standards in the general education classroom ( Alexandrin, 2003). The goal of tracking active student engagement is to capture evidence of opportunities for students to answer questions, work with peers, and complete tasks independently ( Maheady et al., 2006), which are all necessary if students with learning disabilities are to find success in inclusive classrooms.

There are ways teachers can effectively include students with learning disabilities in whole-group instruction. Specifically, variations of whole-group response strategies can accommodate a range of individual student needs. Hand signals, response cards, and written response strategies are three possible response strategies. In addition, collecting formative assessment data during whole-group instruction can inform student groupings, identify areas of relative need and strength, and track progress toward IEP goals.

Hand signals can be used to promote student engagement and check for comprehension. For example, teachers can use a hand signal strategy to facilitate class discussion. Specifically, teachers can guide discussion by selecting students who want to share new ideas (students holding up one finger) or add to the current idea (students holding up two fingers), as illustrated in Figure 1. This strategy can prevent the discussion from veering off topic and allows teachers to ensure a comprehensive discussion of one idea by selecting students who want to add to the same point before moving on to a new idea. This strategy is a way to scaffold the discussion because students must consider not only what they want to share but also how their idea contributes to the conversation. Scaffolding in this manner increases comprehension for students with learning disabilities by targeting both cognitive and metacognitive development ( Frey & Fisher, 2010).

Whole-group hand signals also can be used to check for comprehension. Comprehension checks using whole-group hand signals can help teachers redirect the lesson before students become frustrated or disengaged. This type of whole-group responding also promotes engagement because it holds students accountable for their learning by asking students to self-evaluate and reengage in the learning process. Comprehension checks are important for students with learning disabilities, who can be left behind during a lesson because they struggle with working memory or the ability to simultaneously store and process information to complete cognitive tasks ( Alloway, 2011; Dehn, 2008).

Incorporating comprehension checks into a lesson at specific points allows teachers to note patterns of student comprehension surrounding specific concepts ( Berry, 2006). Teachers who use hand signals to facilitate comprehension checks should include a scaled response system similar to the four-point system illustrated in Figure 1 rather than a dichotomous "I understand" or "I don't." Valuable learning can occur when students reflect upon their own learning to offer a degree of understanding rather than being asked discreetly, "Do you understand?" or "Does this make sense?" ( Haydon, MacSuga-Gage, Simonsen, & Hawkins, 2012). Special educators can create peer discussion opportunities for students based on the degree to which they understand. Students who feel confident to turn and teach a classmate can pair up with those who are not sure if they understand a given concept ( Maheady et al., 2006). By engaging in peer learning opportunities, students with disabilities can hear classmates use age-appropriate language to emphasize the concepts taught and self-correct their own thinking ( Fuchs & Fuchs, 2005). Teachers can use this opportunity to measure student learning and understanding of concepts by listening to student conversations and jotting down notes on the formative assessment template they are using to monitor understanding ( Alexandrin, 2003; Cornelius, 2013).

Capturing information using formative assessment charts allows teachers to make decisions about future lessons; this is otherwise known as assessment-driven instruction or data-based decision making. In the example in Figure 1, the teacher collected information about students' self-reported comprehension levels. The teacher noted repeated comprehension issues (students who responded using 1s and 2s), and also noticed that both Anne and Joyce struggled during independent tasks throughout the lesson. In this case, the special educator can co-plan with the general educator to build in a small-group preteaching opportunity to model independent tasks for Anne and Joyce. The information collected helped the teacher notice students who needed greater levels of support to feel confident about independent tasks during whole-group instruction. To access, download, customize, and print the formative example chart illustrated in the example in Figure 1, visit http://trackstudentlearning.weebly.com/.

Teachers often use whole-group responding by posing a question in which all students verbally answer in unison, also known as choral responding. However, it can be hard to track individual student accuracy during choral responding, and students with learning disabilities who lack confidence can become passive learners. In some instances, students will not participate at all when prompted to respond verbally ( Berry, 2006). One alternative to verbal or choral responding that has been shown to increase student participation is the use of response cards ( Randolph, 2007). Response cards involve students holding up cards with predetermined answers to respond to a teacher-initiated prompt, eliminating the need for verbal or written responding. Many students with learning disabilities struggle with writing mechanics, such as handwriting, spelling, vocabulary, and text structure ( Gregg & Mather, 2002). When students face such challenges, teachers can use response cards to encourage active participation.

Figure 2 provides examples of flashcard-sized response cards that can be used with a wide range of students and situations. Response cards can be true-false or multiple choice or more content-specific, such as a set of graphemes, vocabulary words, parts of speech, or story elements. Response cards can even be a list of key words that students hold up during a multistep problem to further monitor comprehension and sustain student attention, particularly students with learning disabilities who struggle with organizational skills and making connections ( Alloway, 2011; Montague et al., 2013). The purpose is to create a positive learning community so all students, including students who would otherwise not participate, have frequent opportunities to respond and actively learn ( Heward et al., 1996).

Some students may require additional wait time or prompting to generate a correct response ( Johnson & Parker, 2013). Peer supports and wait time can be used during response-card activities to address the needs of students with processing delays. For example, think-pair-share-where students think through their response, express their reasoning to a peer and obtain immediate feedback on their understanding, and then share their response card out in the whole group-has been found to sustain engagement and enhance critical thinking ( Tyminski, Richardson, & Winarski, 2010). Students with learning disabilities, who may lack metacognitive skills, can be taught appropriate peer-to-peer interactions during whole-group instruction by reviewing a "look, lean, and whisper" approach ( Archer, 2008).

Figure 2 also illustrates how student data, including response and accuracy rates, can be collected during whole-group instruction that includes response cards. Tracking the rate and accuracy of student participation can help predict the academic performance of students with learning disabilities ( Gersten et al., 2009). In this example, the teacher was working with 11 students during a direct instruction block, and the objective was to identify pennies, nickels, dimes, and quarters by holding up the corresponding response card when prompted. The teacher used a chart that matched her seating arrangement, and each desk on the chart represented one student. Throughout the math lesson, the teacher noted whether students held up correct or incorrect response cards when prompted to identify types of coins. Using this method, the teacher was able to monitor her students' progress toward mastery of the content. This type of formative assessment tracking chart can be used to track progress toward IEP goals or to provide specific information about student participation and accuracy for a given learning objective. Recording progress toward individualized goals and objectives becomes as simple as noting how many opportunities the student had, to identify coins, for example, and how often he or she was accurate. Using the example provided in Figure 2, the special educator may decide to pull Simon (S.H.) and Anna (A.N.) into a small group to reteach using guided practice, scaffolded instruction, or a gradual-release model because these students answered only four out of the 10 questions correctly.

Whole-group response strategies can include written responses on exit tickets, open-ended poll questions, surveys, and dry-erase boards. Written responses may be more appropriate than hand signals or response cards in situations where teachers need to accurately capture and make judgments about student learning related to instructional objectives ( Thiede et al., 2015). Teacher prompts can be content specific or can target learning and thinking skills, such as taking a stance on a topic or explaining one's own reasoning to promote reflection ( Miranda & Hermann, 2015). Further, whole-group probes that require written responses can be open-ended or require students to show their work as opposed to closed-ended-only probes during response cards or hand signals. Rather than asking only a portion of the students to actively engage in solving an equation during math class, all students can work on solving the math equation at the same time using whole-group written response systems. This also prevents students from skipping ahead because they are all working only on a singular probe at the same time ( Archer, 2008).

Figure 3 illustrates how to use dry-erase boards during whole-group instruction. Students can write a vocabulary term, rewrite a sentence using correct punctuation, or construct an extended response to a probe. Students also can create pictorial or representational written responses when the teacher probes allow for it. Building in classwide wait time or using a visual timer to indicate when students can respond provide all students with an opportunity to participate (see Johnson & Parker, 2013). When asking for written responses beyond one sentence, consider including sentence starters or a mnemonic device such as POW ( pick my ideas, organize my notes, write and say more), because students with learning disabilities require planning time and a way to organize their thoughts before writing ( Graham & Harris, 2003).

The formative assessment chart in Figure 3 includes data collected throughout an entire week to track how students performed during a 5-minute spelling check before lunch each day. Students were asked spelling words from their list, and the teacher simply wanted to track if students were error free or not during this 5-minute activity. The hope was that by Friday, all students would write spelling words without error. A chart like this can be used in many different ways to track student participation and learning. In this example, the teacher was checking for content mastery and used this information to determine which spelling words to include in area or station work throughout the week. This type of formative assessment chart allows for differentiation and a way to guide co-planning. For example, Becky, Alyssa, and Tyrod were introduced to more complex spelling patterns during station work with the general educator because they reached and maintained mastery of the class spelling words early in the week. Katlyn, Blanca, and Dixon, on the other hand, worked with the special educator to create self-correcting materials that reinforced the spelling patterns they were struggling with during whole-group spelling checks.

Teachers need methods for engaging students with learning disabilities in a positive learning community when whole-group instruction is unavoidable. Increasing students' opportunities to respond and providing student comprehension self-checks as well as capturing this information using formative assessments are very important topics for all teachers. The strategies we have discussed in this article can be used across content areas and in a variety of settings. Teachers can use the information gathered through increasing students' opportunities to respond as a way to both monitor progress and inform instruction.

Increasing students' opportunities to respond and providing student comprehension self-checks as well as capturing this information using formative assessments are very important topics for all teachers.

The formative assessment templates will become a student data log over time. There is no need to create extra work by transferring the data to another form. Simply photocopy the completed templates and keep them in each student's file for future reference. Look for patterns within subject areas or over a set period of time. Note whether specific students respond to one type of whole-group response strategy over another to make decisions about future class activities. Whether they are addressing student engagement, content understanding, or targeted IEP goals, teachers will have evidence at their fingertips to reflect on as they plan lessons. Teachers can group students for cooperative activities based on patterns among student answers as well as determine which students require small-group specialized instruction to preteach or reteach key concepts. Together, these tools can provide teachers with valuable data they need to make important instructional decisions.

Proactive whole-group response systems paired with formative assessment charts have the potential to result in more effective instruction that actively engages students in the learning process. These strategies can be implemented easily in classrooms with minimal additional resources and are applicable across grade levels and content areas with appropriate modifications. Students with learning disabilities have demonstrated deeper engagement and understanding as well as positive student-teacher and peer-to-peer interactions using the strategies we discuss ( Berry, 2006; Clunies-Ross, Little, & Kienhuis, 2008; Randolph, 2007). To achieve positive results, it is important that teachers create safe learning environments where students with learning disabilities are encouraged and feel confident to take risks in revealing responses that reflect their own understanding. Using repetitive whole-group response strategies can build predictability into the lesson, which has been shown to lower anxiety and increase participation (see Heritage & Heritage, 2013). Teachers can reinforce a safe learning community by moving around the classroom, observing student work to ensure it matches their responses, and having students engage in peer discussion ( Alexandrin, 2003). Proactive whole-group response strategies combined with formative assessment charts promote active student engagement and streamline progress monitoring for special education teachers.

Alloway, T. P. (2011). Improving working memory: Supporting students' learning. Thousand Oaks, CA: SAGE. Google Scholar

Berry, R. A. (2006). Teacher talk during whole-class lessons: Engagement strategies to support the verbal participation of students with learning disabilities. Learning Disabilities Research & Practice, 21, 211-232. doi:10.1111/j.1540-5826.2006.00219.x , Google Scholar Crossref

Dehn, M. J. (2008). Working memory and academic learning: Assessment and intervention. Hoboken, NJ: Wiley. Google Scholar

DiCarlo, C. F., Pierce, S. H., Baumgartner, J., Harris, M. E., Ota, C. (2012). Whole-group instruction practices and children's attention: A preliminary report. Journal of Research in Childhood Education, 26, 154-168. doi:10.1080/02568543.2012.657744 , Google Scholar Crossref

Graham, S., Harris, K. R. (2003). Students with learning disabilities and the process of writing: A meta-analysis of SRSD studies. In Swanson, H. L., Harris, K. R., Graham, S. (Eds.), Handbook of learning disabilities (pp. 323-344). New York, NY: Guilford. Google Scholar

Gurganus, S. P. (2007). Math instruction for students with learning problems. Boston, MA: Pearson Education. Google Scholar

McLaughlin, M., Allen, M. B. (2009). Teacher-directed whole-group instruction. In Guided comprehension in Grades 3-8 (pp. 17-32). Newark, DE: International Reading Association. Google Scholar

Montague, M., Krawec, J., Enders, C., Dietz, S. (2013). The effects of cognitive strategy instruction on math problem solving of middle-school students of varying ability. Journal of Educational Psychology. Advance online publication. doi:10.1037/a0035176 , Google Scholar Crossref, ISI

Simonsen, B., Fairbanks, S., Briesch, A., Myers, D., Sugai, G. (2008). Evidence-based practices in classroom management: Considerations for research to practice. Education & Treatment of Children, 31, 351-380. doi:10.1353/etc.0.007 , Google Scholar Crossref

Wehby, J. H., Lane, K. L. (2009). Proactive instructional strategies for classroom management In Akin-Little, A., Little, S., Bray, M., Kehle, T. (Eds.), Handbook of behavioral interventions in schools (pp. 141-156). Washington, DC: American Psychological Association. doi:10.1037/11886-009 , Google Scholar Crossref

# data[].attributes.attachment_link.title
Whole-Group Response Strategies to Promote Student Engagement in Inclusive Classrooms

# data[].attributes.attachment_link.description
Students with learning disabilities are often educated in inclusive classrooms alongside their typically developing peers. Although differentiated small-group instruction is ideal for students with learning disabilities, whole-group instruction continues to be the predominant instructional model in inclusive classrooms. This can create major challenges for teachers as they aim to actively engage all students, including students with learning disabilities.

# data[].attributes.body
SPACE OUT YOUR STUDYING OVER TIME

# data[].attributes.body
blog about questioning students in lessons

# data[].attributes.attachment_link.readable_text
A couple of years ago, I first blogged about Pose, Pause, Pounce, Bounce! If you are not familiar with this superb questioning strategy, then I would strongly recommend that you download my resource for your classroom. Then read the supporting blog to explain the resource in fuller detail. Download the PDF here.

Related

Posted in Tagged Assessment, @JohnSayers, @Mr_Haines, CPD, Institute of Education, Pam Fearnley, Pose Pause Pounce Bounce, PPPB, Question Matrix, Questioning, socratic questioning, Speed Dating, Tao of Pooh, Teaching and Learning, The Matrix, The Matrix Film, Winnie The Pooh Basic Account, Resources, Teaching and Learning

# data[].attributes.body
Geoff Petty

# data[].attributes.attachment_link.title
Feedback: Medals and Missions - Geoff Petty

# data[].attributes.attachment_link.description
Geoff Petty author of Teaching Today and Evidence Based Teaching. Teaching methods. Active Learning. Teaching skills, creativity, self assessment and Professional Development. Improving the use of the creative process. Evidence Based teaching of creativity.

# data[].attributes.body
EXPLAIN AND DESCRIBE IDEAS WITH MANY DETAILS