# data[].attributes.attachment_link.readable_text
September 2020 | Volume 78 | Number 1
Grade Expectations

Susan Brookhart, Thomas R. Guskey, Jay McTighe and Dylan Wiliam

At the 2019 Learning Sciences International's National Formative Assessment Conference, the four of us participated in a panel session to explore the place of grading as an important component of comprehensive and balanced district assessment systems. After this panel discussion, we met to summarize our discussion and make recommendations for how grading might be improved to enhance student learning, supporting our points with research and our practical experiences. Here are some of our thoughts on needed improvements.

Defining Grading, Identifying Assumptions

We define grades as the symbols assigned to individual pieces of student work or to composite measures of student performance created for report cards. Grades are usually letters, numbers, figures, or a set of descriptors that designate different levels of performance.

Discussions of grading touch on complex, occasionally emotional questions and bring out strikingly differing views on seemingly straightforward things like the role of evaluating student performance. It helps to clarify from the start one's view on some of these questions. Before we made recommendations for improving grading practices, we noted some things we could agree on.

First, we believe that the primary goal of grading and reporting is communication-that is, providing information to interested stakeholders about student achievement and often other qualities (such as dispositions) in such a way that interested parties can clearly understand it and use it.

Second, we acknowledge that, historically, grading practices have been notably variable. Teachers often use a mix of achievement-related information and other factors (like progress or attendance) to calculate and assign grades-both for individual assignments and report cards. Teachers even decide what factors to include-and how to weigh them-differently for different students. Many of the grading practices in U.S. schools are based on tradition, not evidence. There's a lot of room for improvement.

Third, in our discussion after our panel we talked about whether the usual methods of grading should be reformed or replaced with something completely different-and we come down for reform and improve. We considered two realities in debating this: (1) grading practices are at least partly rooted in teachers' foundational beliefs about learning and assessment, and (2) grading is related to significant institutional practices, such as admission into higher education institutions. Since we can't erase and replace teachers' beliefs and prior experiences or immediately alter institutional practices, we believe grading will have to be reformed, not changed in the twinkling of an eye.

Deep-rooted reform will help our grading systems fulfill their purpose. Done well, grading can become a useful source of information in a comprehensive and balanced district assessment system. Grades communicate information that summarizes larger learning domains-like unit goals or standards-than does lesson-by-lesson formative assessment. In addition, grades are more contextualized to school learning than standardized achievement tests; they report classroom achievement that's closer to the time, place, and manner in which the students learned. Effectively determined grades can help students, parents, teachers, and administrators understand what has been learned and select more appropriate next steps in teaching, planning, and resourcing.

How to Steer Toward Reform? Eight Ways

To that end, we have identified eight principles we believe leaders should employ to improve grading practices so these practices are most effective in communicating student progress.

1. Clarify purpose. Educators, parents, and students should know the specific purpose of their grading and reporting system: what it will communicate, what it does not communicate, and what additional information is available. There is no perfect grading system. All systems involve some trade-offs involving specificity, recency, and precision of information. For example, some standards-based grading systems report only on selected "power standards," for the sake of having a concise, actionable report card. When this is the case, stakeholders should know it-and should know how and where they can get information about other standards if they desire. We've found that in most districts and schools, the primary purpose of grading is to communicate students' current status on the learning outcomes in curriculum and standards, but schools don't always spell out this purpose to all involved.

2. Establish goals. Clear learning goals unite curriculum, instruction, and assessment and are the basis of a sound grading system. A clear goal, for instance, might state "Use place value understanding to round multi-digit whole numbers to any place." When teachers are clear about learning goals, they can provide appropriate instruction and use assessments that enable valid inferences about student learning. When students are clear about goals, they're better able to regulate their own learning; they can set a goal and work toward it, monitoring their understandings and adjusting their work as they go. Assessments of student performance, and the associated grades that result, should be closely aligned to targeted goals.

Educators have asked us what we think about grading social-emotional learning as part of school learning goals for grading. We believe social-emotional learning goals should not be evaluated and graded in a traditional manner (such as an A- F scale), because SEL goals are hard to measure-and most measures of these skills are easily gamed. However, both teachers and students can collect evidence of SEL-type skills or dispositions. Students can reflect, self-assess, and set personal goals for improvement in specific areas. Teachers can give feedback on SEL goals as well, noting progress and giving suggestions for improvement.

3. Base grades on a collection of evidence. The quality of the evidence makes a great deal of difference. Each piece of evidence, whether student work on an assignment or teacher observation of what a student does or says, should support valid conclusions about whatever learning outcome or learning skill it's being used to rate, and should be interpreted accurately and without bias.

4. Reflect current achievement. Grades should be based on a synthesis of evidence reflecting students' current level of learning or accomplishment, not an average of performance over a period of time. Where students were at the beginning or halfway through a learning sequence doesn't matter. How many times they fell short during that sequence doesn't matter. What matters is what they have learned and are able to do currently or "at this time."

5. Use established criteria aligned to targeted learning goals. Ideally, districts would establish sets of evaluative criteria and associated scoring tools (like common rubrics) aligned with key standards. Having such well-developed evaluation tools would make it more likely that teacher judgments of student performance-and the concomitant grades assigned-will be more consistent with the judgments of other teachers. We acknowledge that this recommendation can be difficult to enact, because agreement on criteria and performance levels may be difficult to reach, but we think it is worth the effort.

6. Set conditions for opportunity to learn-including feedback. Students must be given appropriate opportunities to practice and receive feedback before their work is graded. Learning precedes assessing and reporting what has been learned. Students need opportunities to apply new learning and receive feedback before they're evaluated. Ongoing formative assessments can provide that feedback, and the results of such assessments should not be graded. The feedback students receive on ongoing work should be based on the same criteria as will be used for grading summative assessments. Effective feedback fuels the formative learning cycle, guides student self-regulation of learning, and helps students connect the practice and learning work they do with the grades they receive.

7. Report achievement separately. The main grades on a report card should report only students' current status on achieving intended learning outcomes, indicated by the quality of work on well-designed assessments, performances, or demonstrations. This body of work should represent with fidelity the domain of learning described by the standard. Other factors, like SEL skills, maybe, or work habits (completing homework on time, say), collaboration, behavior, and attendance should be reported in a separate section on the report card (sometimes called Learning Skills). We recommend reporting and rating such factors using a different scale from the achievement grades so the measurement areas won't be confused.

8. Use scales with fewer gradations. For report cards, we recommend scales with fewer categories ( A, B, C, D, F, or ratings on proficiency-based rubrics) as opposed to more categories (like 0-100). Fewer is better because as the number of categories rises, inter-rater reliability goes down (meaning that different teachers looking at the same body of evidence are less likely to assign exactly the same grade). The trade-off is that when using fewer categories, it isn't always possible to report nuanced differences in performance. A leader might get around this by allowing for individual grades to differ during a marking period. For example, a teacher could report percentages correct on tests but report proficiency ratings along a learning progression for performance assessments, as long as she used a defensible method of aggregation for the marking-period grade.

Urgency for Change

Changing grading and reporting practices can be very challenging. The pace of reforms will vary from place to place. While we've offered recommendations to improve current grading practices, we can envision future approaches to grading and reporting that would depart more deeply from tradition. For example, grading against a continuum of proficiency ratings that range across grade levels (unlike the current standards we use, which are linked to students' grade levels) might provide a more universal gauge of performance and address grading issues for a wider range of learners. This would mean, for example, that a 2nd grader and a 4th grader with the same proficiency in narrative writing would receive the same proficiency rating for narrative writing. In addition, online "Learning Board" systems for collecting, displaying and assessing evidence of student work could describe student learning in more depth than do current report cards.

Grading should be a part of a comprehensive, balanced assessment system. These principles can help it be such. Grades should be based on clear learning outcomes and appropriate assessments of those outcomes and should support a reporting system that clearly summarizes student achievement.

There is urgency for reform: Poor grading practices at best don't support learning, and at worst, actually harm students by giving them inaccurate or inappropriate information on which to base important school and life decisions.

Endnote

The white paper we prepared presenting our ideas for grading reform in more detail is available at https://www.dylanwiliamcenter.com/Grading-Policy-Paper.

Susan Brookhart is Professor Emeritus in the School of Education at Duquesne University and author of How to Use Grading to Improve Learning (ASCD, 2017). Thomas R. Guskey is Professor Emeritus in the College of Education, University of Kentucky, and coauthor of What We Know About Grading: What Works, What Doesn't, and What's Next (ASCD, 2019). Jay McTighe is an author, consultant, and co-developer of the Understanding by Design framework with Grant Wiggins. Dylan Wiliam is Emeritus Professor of Educational Assessment at UCL Institute of Education and author of Creating the Schools our Children Need (Learning Sciences International, 2018).

KEYWORDS

# data[].attributes.attachment_link.title
Eight Essential Principles for Improving Grading

# data[].attributes.attachment_link.description
September 2020 | Volume 78 | Number 1 Grade Expectations Susan Brookhart, Thomas R.

# data[].attributes.attachment_link.title
Assessment and Grading in Distance Learning - How to Get Better

# data[].attributes.attachment_link.description
Assessment expert Dr. Thomas Guskey reflects on the positives and negatives of assessments during distance learning. He...

# data[].attributes.attachment_link.keywords[]
the chromebook infused classroom: using blended learning to create engaging student centered classrooms

# data[].attributes.attachment_link.keywords[]
chromebook; technology and education; teaching with technology; remote learning; online learning

# data[].attributes.attachment_link.title
The Chromebook Infused Classroom: Using Blended Learning to Create Engaging Student Centered Classrooms

# data[].attributes.attachment_link.description
The Chromebook Infused Classroom: Using Blended Learning to Create Engaging Student Centered Classrooms [Clark, Holly] on Amazon.com. *FREE* shipping on qualifying offers. The Chromebook Infused Classroom: Using Blended Learning to Create Engaging Student Centered Classrooms

# data[].attributes.headline
Assessment in the Time of Pandemic: A Panic-free Guide

# data[].attributes.subject
Assessment in the Time of Pandemic: A Panic-free Guide

# data[].attributes.attachment_link.readable_text
In September 2020 we are running activities about online assessment as part of our Virtual Symposium. Follow the link for details.
Tim Burnett and Stefania Paredes Fuentes, University of Warwick
Written March 2020
Published on this site June 2020

Abstract

Under the current scenario, university lecturers need to rethink their assessment strategy. Many will need to shift from closed-books, time-limited examinations, to remote assessments. The choice of remote assessment should not be an after-thought as it may define our response to the current crisis and will affect Universities' academic standards. Students' performance, marks and final degree classification depend on assessment and we owe to set up fair assessments in line with learning outcomes. We offer a five-step guide on how to change assessments and discuss six alternatives. For each alternative we discuss their potential advantages and disadvantages for students' learning and engagement, but also adoption and organisational procedures. Finally, given that students engaging academic malpractice (i.e. cheating) is one of the most common concerns regarding any sort of remote assessment, we explain how good assessment design can help to mitigate some of these problems.

1 Introduction

If you are a university lecturer, very likely you have been asked to change/reconsider your final year exam, given the current extraordinary circumstances. If you are an engaged teacher with a diversified portfolio of assessments for your modules, you probably do not need any help doing this and it is likely that you are helping others with their assessments, or even advising your university (in which case you do not need to read this guide, though you may find it useful).

However, if you have less experience with different assessment types (e.g. you are a new lecturer or have been teaching the same material for a long time), you may feel somewhat overwhelmed by the task of rearranging and redesigning your assessments at such short notice. Rest assured, we (and others in the academic community) are here to help! We have been working for a while trying to change our assessments to best reflect students' learning capabilities, to engage diversity, and promote higher-level of learning. In doing so, things have not always run smoothly; we might have made mistakes, theory and practice could have diverged at points, or it might have taken us time to reach to best solution to a problem. Given this, we hope that by sharing our experiences we might be able to help you (and others) achieve the best outcomes possible in this unprecedented period.

Many of us will be tempted to look for quick solutions; be wary, however, since lack of careful consideration on assessments may negatively affect the perception of the quality of education provided and further disrupt students' learning. How we respond to this challenge will affect students' perceptions of their university experience and, given the importance of assessments for students' progression and degree classification, assessment may have a stronger impact on universities reputation than teaching will.

The rest of this document is structured as follows: first, we provide a guide to the guide which explains what this is about (and what it is not) and how to use it. Section 2 explains where to start in five simple steps and introduces a framework to reflect about any assessment we may set. Section 3 provides six alternatives that you can transition to under the current scenario; we aim to explain the advantages, disadvantages and offer some tips on implementations. Section 4 adds some final remarks on adapting our assessments to these difficult times.

1.1 About this guide

This should not be considered a general guide to setting assessments-this is dealt with in the extensive literature on good and appropriate assessment design. Instead, we have written this as a 'survival kit' to help you over this first (significant) hurdle, and, maybe/hopefully, you may be inspired to dig a bit deeper when you design your future assessments.

As an opening proposition, it should be said that assessments should not be an after- thought of the module design, but should be considered as one of the cornerstones of module preparation (What do I want to teach students? How will I test they have learned it? Given these two points, how should I teach?). Understandably, because we are not in the preparation stage, but rather 'firefighting', this guide is somewhat working backward: This is what I taught, this is what students have done so far, how can I assess it remotely (i.e. without a sit-down exam)?

Given the timing, this guide will focus on assessments that take place after teaching has finished. As stated, in (too) many cases, these would normally be time-limited exams that students sit in-presentia in university premises. We will avoid any discussion on our profession's over-reliance on these exams (we reserve the right to do this later!), and simply work on the assumptions that, if you are reading this, you need to find an alternative.

Another important consideration we make is that (most of the) students are not going to be located on campus when taking these assessments. While this is clearly challenging for you, it is also likely to be a new experience for students. Thus, this guide will also be providing advice on minimising the effects of shocks associated with changing the assessment methods.

2 Where to start?

The task ahead of you may appear Herculean, but we would like to assure you that it need not be. The task of remotely assessing students has confronted many academics before us, especially those involved in distance and remote learning. Thus, we have at our disposal a wealth of knowledge and experience which can be applied to the problem at hand.

2.1 Five basic steps

A. DON'T PANIC! Easier said than done but, believe us, this may not take as long as you think. Also, try to avoid being too grumpy because you have been asked to change assessments; while there are many academic tasks that we are asked to do which may not be immediately relevant to our jobs, if you teach a module, you are the best person to set that module's assessment! You never know, you may gain something from this experience.

B. Consider what assessments students have already done for your module. Ideally, we want to deploy a variety of methods for assessment so that the same students are not always disadvantaged (Knight, 2002). If students have already sat an exam-type assessment (e.g. a time-limited test), another exam may not help to assess your module's learning outcomes in any case - so you may want to take this opportunity to rethink your overall assessment strategy for future years. Also, read again the final assessment (e.g. exam) that you have already written. As the following sections stress, changing assessment type does not mean throwing away all the work you put into the existing assessment. If you wrote this a while ago chances are you may have forgotten what is in there, so read it again - a fresh look at your content will help you understand what type of revised assessment you might set, and what can be saved.

C. Read university guidelines (if any). This is a very frantic period and many universities are putting in place contingency plans. These will likely give you a good idea of time constraints for your redesign, technologies available, and what support is available to help you. Also, do not worry if you think ' my university's guidance isn't actually helpful '-they may be very busy and/or working on it, and that is why we wrote this.

D. Write a new assessment. Just like that! Not really. This is the hard part and for this reason we dedicate Section 3 to outline the various options available with some hints and tips on transitioning to these assessment types.

E. Provide students with clear guidelines on the new assessment method. Students have been preparing for the assessment you planned. Moreover, if in-presentia examinations are the status quo within your department, then students will be accustomed this form of assessment. Assessment which deviates from the norm may challenge students in more ways than just content knowledge, so you will need to ensure that students are sufficiently briefed.

2.2 Beyond the basics: Understanding the module's Learning Outcomes

There is one further point to consider before we continue, and it is an important one: Look at your module's learning outcomes.

This might sound incredibly obvious, routine, or, alternatively, it may sound like completely new and unknown territory for you. You might even think this is not necessary-being that you may never have consciously linked outcomes to assessment previously (this is really common, you are not alone).

Many module leaders are not familiar with their modules' learning outcomes; indeed, in many cases these have been written by someone else (Head of Teaching, previous lecturer who delivered the module, etc). Often, we, as module leaders, 'inherit' modules, and time constraints or competing priorities mean that we continue to reproduce the same material and assessment we inherited.

While, clearly, this is not a great practice, all is not lost! There is no time like the present to familiarise yourself with what your module is aiming to do. Notwithstanding that this might also help you with your teaching in future, understanding what knowledge and/or skills students should acquire for your module is also essential information for the purposes of effectively and efficiently designing your alternative assessment.

2.3 What are we assessing?

Even within the category of remote assessments, there still remains a broad number of possible approaches which can be pursued (as outlined in Section 3). While some types of alternative assignment follow similar principles to conventional sit-down exams, others differ markedly. Each type of assessment is also characterised by its own advantages and disadvantages (you may be surprised to learn that there is not one single type of assessment which dominates others)-these may be in terms of the practicality, or otherwise in terms of the type and depth of knowledge and understanding that alternative approaches can assess.

With regard to the last point, a common way of analysing teaching, learning, and assessment, is through the lens of Bloom's Taxonomy of Learning (introduced in the 1950s and revised in 2001, detailed in Krathwohl (2002)). Whilst trying to avoid unnecessary levels of technicality, we do make use of Bloom in comparing some of the benefits and downsides of the various assessment approaches.

Bloom's (revised) Taxonomy is a conceptual model which considers levels of cognition as being represented by a set of actions an individual (or student) is able to perform; depicted in Figure 1:

Figure 1: Bloom's (Revised) Taxonomy of Learning

Source: Shabatura (2018)

Shabatura (2018) offers a clear explanation of how to use Bloom's Taxonomy to consider the process of learning (bold in the original source), and we reproduce her explanation here:

In addition to thinking more broadly about cognition, this type of framework allows us to evaluate and reflect upon what level of learning an assessment is examining (i.e. does my assessment merely test student's knowledge, or does it require students to demonstrate higher-order cognitive skills?).

Below, and throughout the remainder of this section, we discuss the pros and cons of various remote assessment techniques and provide some advice for their implementation under the current scenario. Using Bloom's Taxonomy as a guide, we also describe how effectively these various approaches can be used to assess different levels of cognition.

Finally, given the level of concern amongst some sections of the academic community, we also provide a critical consideration of issues of academic malpractice - both with respect to the individual assessment types, and more generally in section 3.7.

3 Type of assessments

Here we propose six different options and some ideas on how to implement them, and the potential issues you may encounter during implementation.

One common advice in this period is: "it is not a time to innovate or be creative ". We do not completely agree with this. At a time when we face one of the major challenges to which we need to provide solutions to ensure we continue to deliver our courses and support our students; we need to be creative and sometimes adopt changes that are outside our comfort zone.

That said, we need to counterbalance assessment innovation with the disruption this may create for students. While it is a busy period for us, for students getting a degree is an important moment in their lives and assessments will affect decisions on grades, progression, degree classification and employability opportunities. Therefore, we need to set assessments that test students' engagement with the module, but do not add technical, organisational or implementation burden that may affect their performance.

Therefore, good communication and support on any assessment decision we make must be an integral part of our strategy, even if we set an assessment that is as closed as possible to what we have planned, the circumstances through which students will be working on these assessments are not, and universities need to work on how to adapt to this.

3.1 The Nuclear Option: Cancel it

Dependent upon your institution's regulations, the simplest approach for dealing with your assessment may be to cancel it. There are a few things you need to consider if pursuing this option:

1. Have you already assessed all the learning outcomes associated with the module, such as through other assessments or coursework?

2. Have students had an opportunity to demonstrate their competence such that you can infer their likely overall performance from the assessments already taken?

3. Linked to the above, is the assessment you need to rearrange sufficiently low weighted that you can reasonably say that students have undertaken enough assessment to test their competencies?

If your answer is yes to two or all these questions, you may seriously consider PRESSING THE RED BUTTON! (i.e. cancelling the final assessment). This option is clearly subject to institutional regulations and/or the measures they put in place to deal with the present circumstances.

Of course, this is not challenge-free. Some students may be counting on the final assessment to improve their marks, with some putting lots of efforts into addressing feedback they received in previous assessments in which case, any extrapolation of existing marks is unlikely to reflect their actual preparation.

However, if this is a low-stake assessment, this will have little effect on their overall degree. There is an extensive literature (e.g. Barry et al., 2010; Knekta, 2017; Wise and DeMars, 2005) which looks into student effort and motivation where stakes are low; finding (amongst other insights) the idea that low stakes test exhibit much greater variability in student motivation and effort than high stakes. A non-negligible benefit of cancelling low-stake assessments is that students will have more time to focus on their remaining assessments.

3.2 Time-limited on-line exams

Time-limited on-line exams are, arguably, the closest we get to traditional exams. This consists of making the exam questions available on the university Virtual Learning Environment (VLE)[1] such as Moodle or Blackboard, and giving students a limited amount of time to complete the task (perhaps comparable to the original exam time limit).

If replacing traditional exams, many universities may be opting for this type of assessment, and there are some valid reasons for this to be the default option. First, if exams are already written, lecturers may not need to write a whole new assessment; such re-writing exams could put a further burden on (professional services and academic) staff time, who may already be very busy. Second, this type of assessment will be very similar to that which students are expecting; in the UK, students usually know the structure of the exam in advanced and past exams are available to help them prepare - so this may be an opportunity to offer some continuity in a very challenging period. There are, however, also reasons why the use of exams is less valid; the prime example of which is where exams are chosen solely because they offer a path of least resistance (probably one of the main reasons so many courses rely so heavily on exams).

Irrespective of the rationale for the employment of this approach, there are several considerations which cannot be overlooked. It is important to ensure that exam questions do not simply ask students to replicate information that it is easy to find in textbooks or other re- sources. While this will be poor practice even for in-house, invigilated exams, remote written assessments become de-facto 'open book', such that students have access to their notes and course texts while completing the exam. In this case, assessment is not a useful an indicator of students' performance and engagement with the subject; possibly failing even to satisfy the lower reaches of Bloom's Taxonomy. In this case you may wish to redesign questions such that they require students to demonstrate understanding, or apply their knowledge to a case study, rather than simply reproducing theory or content (Race, 2014).

Multiple choice can be employed in these types of exams and most virtual learning environments have inbuilt tools to allow their creation. Devising and/or preparing online multiple choice tests will clearly impact on the time it takes you to revise your assessment and setting good multiple choice questions that assess more than simply the capacity to choose the correct definition takes some time, but pays off when marking submissions (as does any type of multiple-choice test). Online multiple-choice testing also usually permits the drawing of a set number of random questions from a larger bank; this can help reduce the risk of student communication, but obviously comes with the need to ensure that all questions are of comparable difficulty.

Time-zones represent one of the biggest challenges to these assessments. Given the level of internationalisation in the UK universities[2] students will be sitting exams at different times and it may be very difficult (almost impossible) to find a time that is not inconvenient for some students e.g. in the middle of the night.[3]

One way to overcome this may be using dynamic timing: Students have a time window in which the test is available to view or download, but once begun students have a strictly limited amount of time to complete the work. In this case, the initial windows needs to be broad enough to allow all time-zones to access the exam at a convenient time (e.g. any time between 9am and 2pm UK-time should work for 3-hour examinations) and complete the exam. Some VLEs such as Moodle may allow to set these exams, but otherwise you need to check with the university IT services on the technical feasibility of this option. Another alternative would be having two similar scripts, one to be released in the morning and one in the afternoon, so students will have the option to choose the one is more convenient for their time zone. Of course, you need to write two different scripts and guarantee that these are of similar difficulty.

We cannot forget that some students need special arrangements when sitting exams. If these are dealt with by offering extra-time, this may not be a big challenge, but if they require other type of arrangements (e.g. special IT equipment), these may more difficult to set up under the current scenario, and will add unnecessary stress and anxiety to these students.

You may also encounter issues with resources. If you rely on one or few textbooks that have no e-book versions, it is important that students have access to these for their preparation, and during the exam. Many students still rely on library copies for revision, and given that libraries are closing, some students may struggle during revision period. This will be bad for almost every assessment but will be particularly intensified in open-book time-limited exams, when having the book is a clear advantage.

Similarly, access to internet may be restricted in some countries. If this is the case you may need to check that resources available on your module's VLE (Moodle, Blackboard, etc.) page are not restricted or under commonly restricted websites (e.g. YouTube).

Another challenge with time-limited online exams is how to deal with late submissions (in the case that the submission is not automatic). In an exam hall, we simply collect the exam at the end of the established time, but this becomes more complicated for online submissions. Rather than refusing to accept late submissions, you may want to apply late penalties.

Finally, during exams, every single minute is important for students, so they will work until the end. Therefore, there are going to potentially be many students submitting at the same time and universities need to ensure the systems can cope with this since, with short time frames, potential system failures may have large effects. For many universities, it will be the first time experiencing this type of scenario, so if the system fails you need to be prepared for how you will deal with such a situation.

3.3 Take-home assessments with more conventional time windows

Assuming your module does not feature " students will be able to reproduce course knowledge in a time-pressured environment " amongst its learning outcomes, another option may be to simply set a conventional take-home assignment. Discussion- or essay-based exams would lend themselves well to become take-home assignments where students have, potentially, several weeks to complete the task. How long students are given may be related to the overall length of the exam, but you might adopt a rule of thumb, e.g. one week for each hour the exam was scheduled to last.

This type of assessment may present fewer technical and organisational barriers than other types of alternative assessments making it very appealing. There are no problems with time zones, and there should not be technical issues (not more than in 'normal' times), but this is not the only reason to choose this type of assessment.

As with regular, well-designed, essay-type coursework In order to work on this assessment, students will rely less on one single source (e.g. textbook) for which an electronic copy may not be available. This will address the issue of not physical having library access in this period but remind students on how to access online resources using their library credentials, for example by providing a link to the library resources.

Written assessment with longer time for completion (essays being the typical example) may be associated, variously, with the opportunity to integrate and apply relatively complex ideas (Cook and Watson, 2013); as such they present an appealing alternative to the potentially superficial responses associated with unseen exams.

Should you choose to set a take-home assessment with longer time for submission, there are also many options available beyond simple essays, including reviews and annotated biographies which require students to engage with literature (Race, 2014), or broader problem-based approaches (outlined extensively in Bean (2011)) such as research projects, responding to to positional statements,[4] blog entries, etc.

Whilst we have this amazing and exciting range of alternative take-home assignments that we might consider in lieu of an exam, it is important to make some considerations. While they may encourage a deeper level of analysis, individual one-off assignments placed in lieu of an exam can result in uneven assessment of the course topic: always consider coverage when setting an alternative take-home assignment - does it assess all the points of the course that would have been assessed by the exam. Given previous assignments on the module, does it need to?

Another potential disadvantage of this type of assessment may be the marking work-load. Because well-designed essays encourage students to think at a deeper level, they are often more challenging to mark. So, if you have a large cohort, this may not be the best option, even with support of extra markers; a marking team needs to be coordinated, and some good moderation involved in order to guarantee consistency across the different markers.

Finally, if moving beyond a conventional essay, it is important to ensure that students are sufficiently prepared for the requirements of the task; all assignments are themselves tests of students' ability to understand the task. Confronting students with a task for which they are unprepared is likely to result in under-performance and student dissatisfaction.

Last but not least, given the high stake that final assessments usually carry, you may be worried about students colluding, plagiarising their answers, or even buying their work. While these practices DO exist, appropriate assessment design can help to mitigate against misconduct. Good assessments which require students to engage in creative and original thinking, or require students to produce original ideas and solve problems, make it far harder for students to draw their ideas directly from other students or sources. Examples include requiring students to make an informed judgement about something, e.g. recent political events or an economic policy decision. There is not a single right answer for these assessments, so you will be marking students' reasoning, interpretation of the problem, and how they use theories and concepts in the decision-making and problem-solving process.

Even in the best designed take-home assessments, students may communicate with each other in order to exchange ideas on the assessment, but this is no guarantee that they will reach (or even agree on) the same conclusion; such communication and negotiation of knowledge and understanding can even be beneficial to students. Moreover, should you still need convincing, remember that online submissions and the use of plagiarism software (e.g. Turnitin) can deter students from collusion and plagiarism, especially if the assessment has a high weight in the final mark (but be sure to remind students that you use these methods). To deal with students buying essays from essay mills may require a slightly different approach and we discuss this further in Section 3.7 (see Burnett, 2020).

3.4 24h/48h Assessments

Even where the exam itself does not naturally lend itself to a longer take-home window (as you might expect with more technical or mathematical exams), you may not want to rule out the possibility of setting a different take-home assignment in lieu of the exam. Instead of a long deadline assessment, you can opt for a shorter time window to complete it, such as 24 or 48 hours.

This assessment has a series of organisational advantages, none less that students may be familiar with them, as they are usually used as formative[5] (or low-stakes summative[6]) assessments. They also help to deal with time-zones issues as all students have 24-hours from the release of the homework to complete the task, so there is less pressure on working at inconvenient hours. To this regard, 48-hours assessments might work even better. We need to keep in mind that there are going to be few assessments happening at the same time and, if students have other examinations in a similar time-frame, 48h may help to reduce time-pressures and allow better organisation (and some sleeping time!).

There are also educational advantages. Compared to assessments with extended sub- mission times, students will be less prone to focus on what they believe is relevant for the assessment, and engage with the syllabus more broadly, as they will prepare in advanced. There is also less time to engage in academic malpractices (e.g. buying essay answers). Compared to online exams with shorter-time limits, 24h assessments offer the advantage that students can reflect and improve the quality of their answers, but we need to set questions that promote this reflection.

Again, as with the previous types of assessment, 24-hour window exams have their own specific challenges:

You need to give particular thought to the design of the assessment. Questions should be set in a way that require students to go well beyond recalling information/ concepts/theories that can be easily retrieved (and hopefully this is the case for all assessments). Similar to assessments with long-submission times, questions should require students to engage in original thinking and critical assessment, but in this case, they have a much shorter time-frame.

When designing the questions, keeping in mind the time-frame available is important. Some may be tempted to set up questions that require very long-answers: after all, students have 24 (or 48) hours to write an answer. However, we should not be designing assessments, such that students must spend 24 hours writing. In fact, for each question we should set a (realistic) word-limit so students are not tempted to just add all the information available, but need to be concise and selective in their expression (i.e. need to judge the value of information/ideas to demonstrate the higher order skills of Bloom's taxonomy; outlined in Section 2.3). Word-limits will also facilitate the marking process, as you limit what you have to read and mark.

24-hour assessments can be also used for mathematical/statistical questions, that may require some interpretation of the problem or data provided. However, there is the risk that students do collude and, if the answers only require the computational results, it will be very difficult to detect collusion. Requiring students to interpret the results, and given a greater weight to this interpretation should (at least) partially address this issue.

You can also use 24/48h assessments in more interesting/engaging ways. For instance, you can require students to write policy briefs or short-videos of determined topics of the syllabus (remember to add a word-limit). Those students who are familiar with the syllabus material, will spend more time in using this information to write the brief (or make the video), but those who have done little preparation will spend more time in reading the course material and their briefs may not provide enough in-depth consideration of the issue.

As with any other reorganisations of assessment, it is important that students know the structure and understand what is expected from them before the questions are released. This will help them to organise the material and be ready to focus on the assessment.

3.5 Online Vivas (oral examinations)

In oral examinations (viva, viva-voce), students are interrogated about selected parts of the syllabus through various questions. This is the typical examination method for PhD candidates, but it is less common for undergraduate or taught postgraduate modules in the UK. Under the current scenario, this could be one examination option.

By now most of us have had at least one online meeting by Zoom, Skype, Microsoft Teams, etc. Not sure how you feel about these, but some may be wondering ' why we don't do this more often?', especially in those occasions when we are technically free at at specific time, but in a different location.[7] We can use this technology to carry out viva examinations.

On the organisational side, one advantage is that we can set up times that suit all time zones, so students do not have to be examined at inconvenient hours.[8] It is also quite easy to set as we have all technology available: Skype, Zoom, which allow to even record the Viva exam for future moderation. On the educational side, there is very little risk for students to engage with academic malpractice (i.e. cheating), which may be the main worry with remote assessments.

Race (2014) explains advantages and disadvantages of vivas. They offer the opportunity to develop practical skills such as practice for employment interviews, and help to evaluate verbal communication skills. However, some students may feel less comfortable with this type of assessment which may lead them to under-perform. Additionally, a second marker may be needed during the interview, so that both markers can discuss on the final mark immediately after this takes place.

3.6 Assessing more than one subject

Most of the decisions on assessments are based on a modular basis i.e. there is one assessment per each module/subject but, given the circumstances, why not set assessments that assess more than one subject?

Taking as example Economics graduates, they will rarely be tackling an issue only from a macroeconomic perspective rather than from an industrial economics angle, and more likely employment tasks will require them to provide a discussion/solution/interpretation of a problem, and they will need to use their knowledge background to do this.

In order to assess more than one subject, you can be quite creative with the structure of assessment. While it may not make much sense to set time-limited online assessments to assess more than one subject, almost all the other assessments here described can be used. An assessment with long-submission time or 24/48 hours assessments that requires students to use ideas/theories/models from more than one module, may require higher- order thinking skills as, depending on the question(s), students need to combine concepts learnt in different modules, in order to provide a good answer.

An assessment for Economics students based on writing a policy briefing or research report on " The effects of a pandemic on the UK labour market " can easily be used to test Macroeconomics, as students need to make macroeconomic considerations (e.g. budget constraints, redistribution policies), Labour Economics, as this will affect the labour market, and even more statistical modules such as Statistics or Econometrics, if we add a data analysis component.

An important aspect for the success of these assessment is to design and set clear marking criteria and to communicate this to students. You need to consider how the mark will be distributed across all subjects e.g. would all subjects get the same mark, or different sections of the problem set will be marked differently according to which subject they relate?

In addition, students' ability of merging and using knowledge from different subjects need to be credited, and you need to decide how this extra-credit will be distributed across the different subjects. This requires collaboration with colleagues teaching other subjects, and coordination on the assessment methods.

Of course, another challenge is that students do not all do the same modules, as many of them will have the opportunity to choose subjects from a list of options, and even from other disciplines. So perhaps these assessments may work better for 'core modules' (i.e. those modules that are compulsory for a degree course) or a combination of one core and one optional module, which requires more coordination among different parties.

Despite the challenges, this assessment can be a very enriching experience for all. Academic staff would be able to set up meaningful teaching-collaborations beyond co-teaching a module, that unlike research-collaborations may not happen that often. For students, being able to use their knowledge from various subjects in one assessment may offer the opportunity to develop important employability skills.

Being able to apply subject specific course knowledge to 'real world problems' is highly appreciated skill by employers, but students may not have the opportunity to develop enough during their careers (see e.g. Jenkins and Lane (2019)).

3.7 An additional note on "cheating"

We understand cheating is one of the most common concerns regarding any sort of take-home assessment. In fact, the main reason why in-presentia, time-limited exams are the preferred assessment method is that we consider they minimise students' likelihood of 'cheating' or other disallowed behaviour. However, evidence suggests that we underestimate the probability and incidence of exam room misconduct, while simultaneously overestimating the dangers of misconduct associated with take-home assessment (Bretag et al., 2019a).

It is important, first, to understand that cheating does take place in exams, perhaps more than we expect. Bengtsson (2019), Bretag et al. (2019a), and Burnett (2020) all suggest that cheating in exams is a relatively common practice. Specifically, Bretag et al. (2019a) find exam-based cheating to be more prevalent than any other type of misconduct, while Burnett (2020) finds that self-declared exam-based cheating amongst UK students is nearly three times more common than commissioning of written work. Furthermore, although the physical presence of academic staff in exam rooms might give the illusion of control and effective surveillance, academic staff are frequently not trained or prepared for the task of identifying malpractice in this environment.

Take-home assessments differ in the way that detection takes place, being carried out after the work has been submitted, rather than trying to detect while work is taking place. In most cases monitoring also features the delegation of detection to plagiarism software (such as Turnitin). This makes detection of plagiarism and collusion much more reliable and far less reliant on human vigilance (as in the case of examinations).

We understand that the use of conventional plagiarism software still leaves the spectre of commissioning of written work by students. while the full extent to which students actually buy work is unknown (estimates vary wildly), there is a growing consensus that assessment design can significantly mitigate against the risk of commissioning. To this end, it is important to understand that some assignments/questions are easier to contract out than others; essay titles which simply require students to demonstrate their knowledge of, or describe a theory can be very easily completed by any individual with a basic knowledge of a subject and a willingness to open a textbook. On the other hand, assessments which promote higher order learning, and which draw on specific aspects of your course are much more difficult to 'farm out' (Bretag et al., 2019b; Rafalin, nd). Good examples might include asking students to personally reflect on specific examples you provided in class, or to work with theories/concepts unique to your course; these types of reflective task make it much easier to identify where content has been produced by an external individual.

A final point regarding malpractice is to bear in mind that this is a behavioural issue. Punitive policies, enhanced detection, and exam-room surveillance are all technical solutions to the issue, but none seek to directly change the underlying behavioural aspects; all exhibit elements of ramping up measures to treat symptoms without addressing the root cause.

Survey evidence in Burnett (2020) suggests that, for almost all types of misconduct, students believe the primary motive for engaging in malpractice is a lack of motivation (or 'laziness'), generally followed by a lack of engagement with the course. Crucially, the only category of misconduct where students believe the primary motive is a desire to cheat or gain an advantage, is exam-based malpractice. The implication of these results is two-fold:

1. Academic malpractice in take-home assignments may be the product of disengaged and/or disinterested students and, thus, can be reduced through

2. Owing to the different motives involved, similar measures are unlikely to be effective in addressing in-exam cheating.

In short, in terms of the academic integrity, we may be promoting the idea that exams are more 'secure' than the evidence suggests, while overly concerning ourselves about mal- practice in written assignments, where we have the tools to deal with it.

4 Some final thoughts on assessment

Under the current scenario, universities and staff will face a number of challenges and changes, including how to set final assessments for students. While all this will inevitable create some extra-work, we really hope this guide helps to think about how to set these alternative assessments, but also that we can take some of this learning and apply it in the future design of our modules.

Assessment for lecturers can be an after-thought: something you work on once you have finished the lectures. Even dedicated colleagues, who put lots of thoughts and efforts in organising lectures and teaching material, might give less consideration to the assessments. Thinking of what are we assessing, and why are we assessing while designing our modules, has the potential to engage students with the subject and even improve our design of more effective teaching material in the future.

While we do understand the need to make fast decisions as these need to be communicated to students, we hope these are made taking into account not just practicalities at institutional level, but considering what is best for the students and the subject-level specificities. Lack of careful planning and defaulting in the obvious options may save sometime now, but has the potential to create more work later.

Also, when considering the shift from an in-presentia, time-limited, unseen exam towards a remote assessment, one must consider carefully the learning objectives associated with the module. The assignment need not replicate perfectly the assessment of the same learning outcomes as the exam it will replace, however as a module convenor, it is important to ensure that all outcomes have been assessed at conclusion of the module.

We also hope that we convey the message that there are numerous positive aspects and advantages to take-home assessments, not least the promotion of higher-order thinking skills and that good assessment design will help to at least reduce potential academic malpractice behaviour from students, and these aspects may outweigh the potential negatives.

One extra point in favour of take-home assessments: research shows they tend to reduce student's anxiety, which are usually high around exam period (Bengtsson, 2019). However we need to keep in mind that these are not normal times, and that students are likely to be exceptionally worried about their assessments, so independently on the option we adopt, we need to offer adequate support.

Good communication and managing students' expectations are very important under the current scenario. Even if we set an assessment that is very similar to the original planned, the conditions under which students will take these assessments are not the same, and therefore we cannot assume students' performance will not be impacted. For instance, access to resources is key for students' preparation, and students may not have the same access to resources at home: they may not have the textbook, or even when available online, material may not be accessible to students in countries with restrictive firewalls.[9]

Despite all our efforts, some students may simply not be able to engage with assessments during the exam period. We need to build this in the current plans, and think of what additional support is required. This may mean allowing these students to return later this year or in the next academic year to continue/finish their degree. Finally, remember to create a plan to keep yourself safe and supported, as these are hard times for everybody, including ourselves.

[1] A Virtual Learning Environment (VLE) is the platform in which we upload teaching resources and other types of communication to students. Moodle and Blackboard are quite popular in the UK.

[2] There were 485,645 international students enrolled in UK universities in the academic year 2018/19. This represents 20% of the HE student enrolment for that academic year (HESA, 2019).

[3] e.g. if the exam starts at 9am this means a very early start (or a late night) for those based in Mexico City (3am) and in the US (ranges from 2am to 4am).

[4] A positional statement is one that asserts a particular position and provides students the opportunity to provide an evidenced and reasoned response; an example of this might be "Monetary policy as an economic tool is/is not sufficient to deal with the challenges of the Corona virus."

[5] Those assessments set to monitor student learning and provide ongoing feedback. These usually do not carry any weight in the final mark.

[6] These are used to evaluate students' learning and do count towards the final mark.

[7] Whether you feel like this is almost certainly related to how your Skype/Zoom/hangouts meeting(s) have gone. If it was dreadful, we understand that you may have no enthusiasm and simply ask you to trust us that it is possible to have good meetings.

[8] Though the opposite may be true in that you need to coordinate meeting times across time zones.

[9] Internet restrictions are not just in place in China, but also in Turkey, United Arab Emirates, Egypt, Vietnam, amongst others.

References

Barry, C. L., Horst, S. J., Finney, S. J., Brown, A. R., and Kopp, J. P. (2010). Do examinees have similar test-taking effort? a high-stakes question for low-stakes testing. International Journal of Testing, 10(4):342-363.

Bean, J. C. (2011). Bean-Writing-Assignments. In The Professor's Guide to Integrating Writing, Critical Thinking, and Active Learning in the Classroom, chapter 6, pages 89-119. Jossey-Bass, 2 edition.

Bengtsson, L. (2019). Take-home exams in higher education: A systematic review. Education Sciences, 9(4).

Bretag, T., Harper, R., Burton, M., Ellis, C., Newton, P., Rozenberg, P., Saddiqui, S., and van Haeringen, K. (2019a). Contract cheating: a survey of Australian university students. Studies in Higher Education, 44(11):1837-1856.

Bretag, T., Harper, R., Burton, M., Ellis, C., Newton, P., van Haeringen, K., Saddiqui, S., and Rozenberg, P. (2019b). Contract cheating and assessment design: exploring the relationship. Assessment and Evaluation in Higher Education, 44(5):676-691.

Burnett, T. (2020). Understanding and developing implementable best practice in the design of academic integrity policies for international students studying in the UK. Technical report, UK Council for International Student Affairs.

Cook, S. and Watson, D. (2013). Assessment Design and Methods. The Economics Network.

Jenkins, C. and Lane, S. (2019). "Employability Skills in UK Economics Degrees." Report for the Economics Network. Technical report, The Economics Network.

Knekta, E. (2017). Are all Pupils Equally Motivated to do Their Best on all Tests? Differences in Reported Test-Taking Motivation within and between Tests with Different Stakes. Scandinavian Journal of Educational Research, 61(1):95-111.

Knight, P. T. (2002). Summative Assessment in Higher Education: practices in disarray. Studies in Higher Education, 27(3):275-286.

Krathwohl, D. R. (2002). A Revision of Bloom's Taxonomy: An Overview. Theory into Practice, 41(4):212-218.

Race, P. (2014). Designing assessment and feedback to enhance learning. In The Lecturer's Toolkit : A Practical Guide to Assessment, chapter 2, pages 29-131. Routledge, London, 3rd edition.

Rafalin, D. (n.d.). Designing assessment to minimise the possibility of contract cheating. City University, London.

Shabatura, J. (2018). Using Bloom's Taxonomy to Write Effective Learning Objectives.

Wise, S. L. and DeMars, C. E. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10(1):1-17.

# data[].attributes.attachment_link.title
Assessment in the Time of Pandemic: A Panic-free Guide

# data[].attributes.attachment_link.description
In September 2020 we are running activities about online assessment as part of our Virtual Symposium. Follow the link for details. Tim Burnett and Stefania Paredes Fuentes, University of Warwick Written March 2020 Published on this site June 2020 Abstract Under the current scenario, university lecturers need to rethink their assessment strategy.

# data[].attributes.headline
Practical assessment strategies to prevent students from plagiarizing

# data[].attributes.subject
Practical assessment strategies to prevent students from plagiarizing

# data[].attributes.attachment_link.readable_text
As colleges ask faculty to prepare for a possible online, hybrid or altered in-person fall semester-or all three simultaneously-many instructors are wondering how to best measure student learning.

Even during the pandemic's early days, some professors decided to toss out their grade books, figuring that high-stress assessments weren't appropriate given the uncertain circumstances, which saw students sent away from campuses, sometimes without tech tools or internet access.

In the latest episode of EdSurge Live, we interviewed two professors who have long questioned traditional methods of grading about how to approach exams, essays and other assessments next semester:

Susan D. Blum, professor of anthropology at the University of Notre Dame, is the author of "I Love Learning; I Hate School: An Anthropology of College," and a contributor to the forthcoming book, "Ungrading: Why Rating Students Undermines Learning (and What to Do Instead)."

Tony Crider, professor of astrophysics at Elon University, offers his students "epic finales" instead of final exams.

Listen to the conversation, or read a partial transcript below, lightly edited for clarity.

EdSurge: I thought we could start out by asking both panelists to tell us what their approach to grading and assessment was when they first started teaching at the college level.

Susan D. Blum: I've been teaching over 30 years, so I've had a lot of time to evolve. And when I was first teaching, I was pretty much a stickler for control and for judgment. And I was very parsimonious in giving good grades. And I was suspicious of students who were cutting corners and possibly cheating. And I have been studying this subject anthropologically and pedagogically and theoretically for 15 years, and I've completely transformed my teaching, top to bottom.

I wrote a book about plagiarism in which I basically concluded that most of the time, the problem is an educational problem and a motivation problem rather than an enforcement problem. The goal is to create meaningful work that students are engaged in so that they have the tools to achieve what we've asked them to do. And that if they don't do that, then it's an educational problem that we have to solve. Clearly, there are exceptions here and there.

Since then, I have written more and learned more about what students are doing in higher education [broadly, and about] what the purpose is. And I no longer think that having a kind of uniformity of content delivery is really a goal. And I aim really to understand what each needs to get out of the class, and work with them as much as possible to develop their own goals. And then assessment is really a way of figuring out how well they've learned, which is really the only thing I care about. And if people learn in different ways, then that's appropriate because they're all different kinds of people. So the approach that has emerged from that is something that I and others call "ungrading." We basically don't grade until we're required to at the end of the semester, but there is a lot of feedback.

Tony Crider: I came at it from the perspective of the physics education research, and looking at it as a way to say, How do we give tests at the beginning and test at the end to try to gauge what they learn over time? And it was very focused on treating each student like a data point-that they were all representing some measure of the effectiveness of the class as a whole. And I probably spent like five to seven years focused very much on that, trying to turn it into a science and optimize that so that it would happen as quickly as possible, from scantrons to online, [with] very traditional exams.

I think that probably the reason I'm on this call is, about 10 years ago or so, I started to ramp up the concept of "epic finales," where you would treat the end of a semester not so much like it's that final assessment, that time where you fill in the blanks and then quietly leave the room. But it was an experience that would give the instructor a chance to see who did his students or her students turn into over the semester? You don't know that if you're just giving a traditional assessment. So ... [I] like to have that final epic finale at the end of the semester where I can sort of see what students are capable of in a new and different situation.

Susan, can you tell us more about this idea of ungrading, where it came from and how you put it into practice in your classroom?

Blum: I began to read the research on motivation, and that, of course, distinguishes intrinsic and extrinsic motivation. And the goal was always to increase intrinsic motivation. But like a lot of others, I was influenced by behaviorism, which tells you that if you increase extrinsic motivation enough, if you give enough points, if you weight things enough, people will discover intrinsic motivation. But in fact, the research shows the opposite. The more extrinsic motivation you provide, the less students feel that it's up to them to really be enthusiastic and excited about things. So that was kind of a theoretical perspective. I wanted to diminish the extrinsic motivation focus as much as possible.

So I began by just telling students don't care about your grades, but I was still giving them, and I was still calculating the points. And how many points do you lose if you have absences above a certain number and how do you measure participation in how many blog posts do you have to have and how many responses to other people's blog posts? And it was very mechanistic and very much about students having a checklist to respond to. When I realized, when I started to read Alfie Kohn and even more Starr Sackstein, and some of the other people who really write a lot about grades, I realized that I needed to get rid of them-and that it was possible. I don't get rid of learning. We just don't focus on the grades. We focus a lot on learning and we have a lot of conversations about it.

I try every semester to have the students generate their own kind of baseline ... when they come in, what do they know? What do they think they're going to do? What do they want to do? What's hard? What's interesting? And I really encourage them to do a lot of honest reflection and self-assessment, so that they really see that, OK, this is an opportunity to learn something that matters to me, even if it's a requirement. Surely there's something about it that matters to each learner because I truly believe that all humans are wildly curious. You just have to figure out how to tap into that superpower of ours.

And [students] are appreciated by their classmates. If you do something well in the class and your classmates say, wow, that was interesting. That's a kind of assessment. They don't need me to be micromanaging.

How it works at the end of the semester [is that] we have a conference where the students suggest a grade. They say, I think what I've learned and produced this semester is worthy of a B-plus, or it's worthy of an A-minus, or an A. Every now and then there's a student who says, I've only really done C work. And that's really wonderful to me that they are honest like that because sometimes we don't care, and that's OK. It's certainly better than having everybody pretend that they've done their maximum for an A, when we know in real life outside school, we aren't always functioning at the maximum. So it fosters reflection, it fosters honesty, it fosters interaction and it leaves open the possibility for more learning. And the class is just a beginning, in my experience, of their engagement with the subject.
Tony, you were recently on the EdSurge Podcast, talking a bit about your unusual and awesome approach to final exams. And I wondered if you could, for people who missed that episode, just briefly tell us what the concept is and give one example of what it means.
Crider: I came up from a different perspective on motivation theory, looking through gamer motivations and research on why people are motivated intrinsically to play video games, and then [I've] have taken those into the class. People tend to want to master something to win, or they want to have some social experience or they want to be immersed in some world. And so I want to provide something for them to think about for the rest of their lives.

[The final exam is] the last time you're gonna spend three hours with them. In fact, it might be the only time that you spend three consecutive hours with them, depending on your university. ... Even if you're still doing a multiple choice test, it doesn't have to be the last three hours that you spend with them.

I craft experiences for the students. The most recent one, in ... a class where we looked at the scientific revolution and then the Industrial Revolution, and then the revolution that we're living in right now, and compared and contrasted those.

And then for their epic finale, I told them, show up to this building. I actually had to tell him, bring four sets of clothes. You'll need some Elon University gear. You'll need some business attire. You'll need some black clothes. And you'll need some sweaters. And then when they walked in the room, I sort of played a character [and said,] alright, here we go, kids, I've invented time travel, and we're going to have you go into the future. And they'd sort of take a pill, like in the Matrix, although they were Mike and Ikes. And then [they'd] go into the room and they'd have to pretend they were five years, 10 years, 25 years and 50 years in the future. And they would each draw cards telling them what their possible life might be like. Maybe they're living at home. Maybe they lost their job. Maybe their job was taken by a robot. And then they'd record themselves on their phones for most of this time. And my goal for this was to get them to see the connections between the sort of possible futures that I've mapped out.

Some students, they would just play it out as written, but others would say, wait, this is what we talked about in the Industrial Revolution. I didn't just lose my job. I lost my job because AI became powerful enough to take my job.

That's what I was looking for in terms of an assessment, but [it's] also ridiculously memorable. The goal is to make an exam that they will talk about for the rest of their lives because that's your last three hours with them. And so I've been doing that for years.

And I'll tell you what, grading these epic finales is the most fun that you ever have. I have grading parties where I sometimes have other professors come in, and we sit and we watch hours of this play out and go through the rubric and say, How did this student do and how did it play out?

I'm curious what your thoughts are about assessments are for this time of pandemic and the upcoming semester. What approach are you taking? What do you recommend?

Blum: Now is a great time to try something new. You know, we are in a really weird moment right now. This is nothing like any of us have ever lived through. Personally, I'm going to be fully online this semester for the first time in my teaching career, aside from, you know, those seven weeks in March and April.

One of the things that a lot of us have really been thinking hard about is equity, and equity when people are in uneven circumstances doesn't mean sameness. And so I think ungrading makes a lot of sense in these particular circumstances. Most of my students have Wi-Fi, and most of them have the tools they need at least on campus. But when they went home in March, there were people who were very crammed into housing, and they were sharing Wi-Fi, [or had] no Wi-Fi. They didn't have good computers with them. And so if I had been a stickler for uniformity and precision-grading and points and attendance and participation in synchronous meetings-it would basically have measured people's economic comfort. And that doesn't seem to me what we're after.

So I think these are the perfect moments in which to really figure out what is possible for each student and what will help them learn given the reality of their life right now. And it seems to me that if we're not just preparing our students for school, but if we're preparing them for life after school, this is a very, very real life lesson to really figure out, what can you do? What's possible?

Crider: I'm at an institution where there's a lot of technology that's the same. Economically, they're very similar. There are 95 percent of them that have an iPhone. Which makes it such that I can design for that. And I know that's not the case at other universities where you've got a wider range of tech or maybe poor connection when they get out of there. And if they're not able to come to campus, that's going to have to be on a case-by-case basis with instructors and the students, sort of trying to figure out what the minimum technology is in that class and how to do that.

The tech, it's causing us problems, but it provides opportunities. I tend to look at the pandemic we're in in terms of technology and education. We've all suddenly been thrust forward 10 years into the future. ... We'll adapt to it.

# data[].attributes.attachment_link.title
How to Assess Learning in a Pandemic - EdSurge News

# data[].attributes.attachment_link.description
As colleges ask faculty to prepare for a possible online, hybrid or altered in-person fall semester-or all three simultaneously-many instructors are wondering how to best measure student learning. Even during the pandemic's early days, some professors decided to toss out their grade books, figuring that high-stress assessments weren't appropriate given the uncertain circumstances, which saw students sent away from campuses, sometimes without tech tools or internet access.

# data[].attributes.attachment_link.readable_text
Transcript

John: Many faculty are finding themselves teaching a fully online course for the first time this fall. In this episode we discuss how faculty can use the research-based SUNY Online Course Quality Review rubric, known as OSCQR, to help them design more effective online courses.

[MUSIC]

John: Thanks for joining us for Tea for Teaching, an informal discussion of innovative and effective practices in teaching and learning.

Rebecca: This podcast series is hosted by John Kane, an economist...

John: ...and Rebecca Mushtare, a graphic designer.

Rebecca: Together, we run the Center for Excellence in Learning and Teaching at the State University of New York at Oswego.

[MUSIC]

John: Our guest today is Alexandra Pickett. Alex is the SUNY Online Director of Online Teaching and an adjunct professor in the Education Department at SUNY-Albany. Previously, she was the Director of the Open SUNY Center for Online Teaching, and prior to that the Associate Director of the SUNY Learning Network for over 12 years and has directly supported and coordinated the professional development of over 5000 Online SUNY faculty.

John: Welcome back, Alex.

Alex: Hey, John. Hi, Rebecca. Nice to see you again.

Rebecca: Good to see you too. Today's teas are:

Alex: As you may know, I only drink Darjeeling tea, always organic. And I just love my Darjeeling tea. It's delicious.

John: Ginger peach green tea.

Alex: Sounds good.

John: It's delicious. It's from the Republic of Tea.

Rebecca: I now have iced tea.

John: I had that earlier today, a few times.

Rebecca: Yeah, it's getting a little warm in my studio. It needs to be cold now. [LAUGHTER]

Alex: I know, it's warm right now. I'm getting warm too. Iced tea sounds good.

Rebecca: We invited you here today to discuss OSCQR, the SUNY Online Course Quality Review rubric designed by SUNY Online to support quality in online courses. Can you tell us a little bit about OSCQR?

Alex: Sure, I'd love to. So, OSCQR was developed, sort of with the advent of Open SUNY when we were developing the Open SUNY Plus programs and wanted a way to help campuses do a real systematic review of the courses in those Open SUNY Plus programs. Open SUNY launched in January of 2014. And so we began while we had the decision to develop OSCQR, and there were many reasons that went into that decision. We've been using rubrics and checklists pretty much from the first day, but over the years we evolved and ultimately bought into Quality Matters in order to have a branded solution for online course quality and course design tools. And so when we evolved into SUNY Online, we decided to develop OSCQR and kind of put aside Quality Matters for a variety of reasons. We needed to be able to use it in a way that was formative and Quality Matters started having more and more sort of restrictions on us. And the financial model changed. And so it just made more sense for us to develop our own rubric. And so in June 2014, we launched the first OSCQR interactive dashboard and rubric. Actually, that was June 14, when we started the design of it, and then in September 2014, we launched OSCQR 1.0. And we had 50 instructional design standards and 50 accessibility standards. And we used that rubric for the launch of the first Open SUNY Plus programs. And so we had cohorts of faculty in SUNY that were reviewing and refreshing online degree programs that were identified for participation in that Open SUNY Plus program. In October 2014, so that's like the next month, we launched Wave II of Open SUNY Plus and added programs and campuses to the Open SUNY Plus cohort of campuses. And so we launched them with OSCQR 2.0, which improved the interactive dashboard and rubric. And then we started winning some awards. We won the OLC Effective Practice award in November 2015. In June 2016, we launched the third edition. So we're currently in OSCQR 3.1. And so the third edition was actually launched in June 2016. And that edition consolidated the standards into 50 standards that integrated the accessibility and instructional design standards. So that's where we are today, we have a set of 50 instructional design course quality standards. We have won a number of additional awards after the first OLC Effective Practice award. We won a Newton award for innovation in 2016. We won the WCET WOW award for it in 2018. And we partnered with OLC in 2016 when they adopted OSCQR as their online course quality scorecard and we were thrilled that OLC wanted to adopt OSCQR, to give OSCQR a national home and take them under their wing. So, we're just thrilled with that partnership and with that umbrella and have continued to improve and evolve OSCQR for the benefit of all of us in terms of course quality review and refreshes, formative online course design, summative course reviews, in a variety of modes. The flexibility we were able to design into it, everything that was missing from previous quality checklists or rubrics that we had used. So we were able to really think about what we wanted the tool to do, what we needed it to do, why we were using it, who was using it, when they were using it. And we designed all of that in there in terms of flexibility.

Rebecca: Can you talk about some of the ways that OSCQR is different than some of these other tools. You mentioned, the formative feedback, and I know it's also Creative Commons licensed, Are there features that make it unique?

Alex: Yeah, I think that flexibility that I mentioned, was intentionally designed into the tools and it's actually a set of tools. It's not just one thing. And so, technically, there is a PDF version of the rubric that is kind of a standalone thing and that's the thing that OLC has adopted and distributes through their website. And that is intended as a tool that anyone can download. And so anyone, whether you're a member of OLC or not, you can go there and download the tool, the PDF, and they just ask for your name and your email address so that they can send you the PDF. It's low barrier, no commitment. So, it's just a PDF, an online faculty person can use that as a self-assessment tool. So, after you've taught your course, the first time, or if you've been teaching for a long time, you can take that checklist, that PDF, that rubric, and do a self assessment. Just reflect on your own design of your own course and answer the questions based on your course. And then you can either take the results of that to an instructional designer and work with them to improve your course or you can use the companion website that sort of is bundled in with OSCQR and mine it for ideas to improve your own online course and there's lots of information on that companion website to help you think through how to address each particular standard. So, that's one way to use OSCQR. I mentioned that it was a set of tools or like a collection of tools. So the PDF is one. There's an interactive online rubric. And there's an interactive online dashboard that can be used together or can be used independently. So, as an instructor, you can use the interactive online rubric, an instructional designer can use the standalone interactive rubric, and you can even use that in a variety of ways. So, an individual instructor could use the interactive version, instructional designers could use the interactive version summatively, as part of their course review process, to preflight a course to say "Yes, it's okay to go up online and be live." You could use it in sort of an initiative kind of a way where you have peers in biology review all of the biology online courses in the department or in a program. You could use it In a way where you have different experts. So, you could have the instructor, you could have a librarian, you could have a technologist, you could have a student, you could have an instructional designer, as a group, review the course with all their different lenses. And the interactive version of the rubric actually supports that model because it's essentially designed in a Google sheet. And so each reviewer has their own tab or their own sheet within that Google Sheet. There's tons of code behind the actual rubric and the way that it's designed, it actually will aggregate the ratings and the comments from each of the reviewers into an action plan. And that action plan then has a point of view based on what the inputs have been, and it will categorize the things that need improvement based on priority. So, it'll tell you these things are important, and these things are essential. And then it also will categorize based on amount of time to fix. So, there are things that might take half an hour to fix things that might take an hour or more to fix or things that might take more than two hours to fix. And so the purpose behind this is to help whoever is going to refresh the course to prioritize. So, it's a point of view. It's our point of view. Because it's an openly licensed open-source tool, if you have a different point of view, you can change that. So, it's entirely customizable and changeable by whoever is implementing it. Now, of course, the average bear is not going to be able to make those changes because there is code involved, but those tools, when you adopt them and want to adapt them... that's more at the campus level or the institutional level, so that they can customize it for the particular use. So, of course, there might be an instructor out there who has these Google coding skills who could do that, but it's more intended to be used as is if you're an individual instructor. Everything in it you can change. If you don't like the standards, you can change the standards. You can add standards. You can create different standards for different disciplines. So, for example, if you have a dental assisting program that has certification by the American Dental Association, they might have very specific criteria for their courses that might be different from your Psych 101 course. And so you can create different rubrics for different programs or disciplines or you can customize them to meet the needs of whatever courses you are reviewing and whatever model you're going to review. So, you can use those sort of team collaborative models, you can use a peer-review process, you can have an instructional designer conduct a formal review of an online course summatively before a course goes live. You can have an individual instructor self assess, and you can have instructors and instructional designers collaborate in a professional development activity, formatively. So, one of the differences with some of the other rubrics is that those rubrics are intended to be used summatively on courses that have been delivered a number of times by faculty who have some experience. OSCQR was intentionally developed to be used formatively with online faculty. So, as they design their course, they know what the standards are, so that by the time they're done, it's not like they're going to get a whole page of stuff that they have to change or fix. If they are following the standards, the review at the end (if there is one) is just going to be clean-up stuff. I would say that OSCQR is focused on instructional design, it says nothing about the teaching of the course. So, it is intentionally that way. It is focused on the instructional design of the course to assist and scaffold quality in the design of the course. It's not to say that we don't know what makes effective teaching, but we just haven't designed that aspect of the rubric yet. And also, there's some challenges and issues and sensitivity that we want to have when we're talking about the teaching of a course. But the reason I'm belaboring this point is that there is sometimes the tendency to forget that a course is both the design and the teaching that impact quality. And you can have a course that is stamped with Quality Matters and stamped with OSCQR and stamped with the Chico rubric and is gold, but then it's not taught in a way that is effective and so it's not a good course. So, you need both halves. And so OSCQR addresses the instructional design of the course. So that's one of the things I think that makes it unique is that it's designed to be flexible, to be used in a variety of models, and to be customizable and adaptable to the distinct uses, the distinct disciplines, the distinct campuses. Like you said, Rebecca it is openly licensed. The interactive rubric and the interactive dashboard are built in Google Sheets and are available to be customized if people want to for use. I talked a little bit about the interactive rubric. And I wanted to mention the interactive dashboard, which works with the interactive rubric and the dashboard also built in Google Sheets, also openly licensed, and also equally flexible and customizable. It's intended for larger scale online course quality initiatives, and typically at the institutional level. So, if you have a department or an institution where you're trying to do a larger course quality initiative, so you're trying to review all of the online courses on the campus or you're trying to review all of the online courses in a program, you might want to adopt the dashboard to facilitate that. So, as an instructional designer or manager of the process, a project manager, you might want to use the tool to generate a bunch of rubrics, associate them with specific courses, assign the people who are going to review them in whatever mode you're going to do the review. So, whether it's an individual instructional designer or a team of people, you can assign them from the rubric. And because these are Google Sheets, this stuff is automated, so people will be notified that they've been assigned the rubric. And then from the dashboard, you can coordinate that and view that. So, the dashboard gives you some tools that will let you know what percentage of the course review is complete. So you can track it all in one place. It gives you some tools to do some analytics so you can, across all of the courses that you're reviewing, you can see, for example, trends, and if everybody is doing very poorly on standard 3B, you can see that and then maybe address that with some professional development. And it gives you sort of quick access to your notes, a single place where you can track and link and generate the different rubrics that are necessary in whatever your initiative is. Whether you're doing general course quality reviews, or whether you're targeting a particular thing, like I've seen some campuses say, "Okay, this year, or in this group of years, we're going to target accessibility," for example, and so they will have an initiative that is at the foundation of the review activities. It might be that they want to improve instructor presence, teaching presence in the course, so they could potentially have a targeted focus for the reviews and have a multi-year plan for that. It could just be that you have 15 courses in a degree program that you want to have refreshed in time for the spring semester or next fall. And so you plan that out. And you can have rubrics generated for all of those courses in that program and you can track the progress of the reviews from the dashboard. So, that's another thing I think that makes OSCQR unique is that it's really taking the perspective of both the faculty, the instructional designer, and the campus and making it maximally flexible for the different use cases that different scenarios might bring. I think that was the intention behind the design. The other part of it, I think, that is unique is that we don't score faculty. This is not an evaluation of their course, we don't give you a passing grade or you don't get points. It really is, and always was intended as, a professional development tool to open conversations with faculty, between faculty, and with instructional designers on the best practices in online course design. And so the assumption is not that you will have nothing to fix in your course when you're done. Because, as I've mentioned before, online course design is iterative. It is an ongoing process that you are continuously improving. This tool assists in the continuous improvement of the design of the online course, assuming that it can always be better. Technology changes, understanding of how people teach well online changes, and so every time you teach online, you can review and improve the design of your course and your teaching practices. So, I think using it as a professional development tool allows all of us who are involved in an interested in online course quality to focus on the best practices and to focus on the conversation around best practices and instructional design, toward the continuous improvement rather than on evaluation of a course, or evaluation of an instructor and the design of their course. So, I think that's a fundamental difference. And for people who are used to having to score 80 percent in order to get the stamp of approval or whatever, it's a little weird. And I've seen people actually change OSCQR to have points. I would always argue against that. Althougdo whatever you want because it's openly licensed. To me, ih you know, you can t is much more important to think innovatively about the design of a course, to have faculty and instructional designers have positive and incremental progress toward quality. I think of quality kind of like a Socratic ideal, you are always striving for it, you don't hope or anticipate that your course is going to suck, [LAUGHTER] you want it to be of high quality, and OSCQR can help you do that. And they are research- based standards. We have organized them in a way that I think makes sense as an instructor or as an instructional designer in how you approach this. Another thing that I think is unique about OSCQR, it really is thinking about how you do a course review and what you look at first and then help people to focus in on the standards to really think about what's going on in the course, and then give some substantive feedback to the instructor or the instructional designer, whoever is going to be making the changes in the course, to be able to help move that course, in that particular standard, incrementally forward. The companion resource that I mentioned earlier is important in this process, because it addresses each standard individually and looks at what the standard means, examples and suggestions on how to improve each standard, some additional resources that are sort of background or additional resources that you can refer to. There's citations from the research that support each standard. And then there's the option and opportunity to leave a comment on a standard if you want to talk about a particular standard or have a question about it. And there's also the opportunity to make a suggestion for an improvement to the standard, or an addition of a suggestion or example for each standard. And we really invite and encourage folks to interact with the rubric in that way, to have influence on the standards and have developed and are in the process of developing additional standards based on community suggestions. So, I think those are some of the things that are unique about it.

John: You mentioned that the goal is to have courses iteratively improve, and you've talked a little bit about how the OSCQR revision process takes place. Could you tell us a little bit more about how it evolves and the process of evaluating the standards and making OSCQR better all the time.

Alex: Sure. We're in version 3.1 of OSCQR currently, and we are in the process of thinking through what the next iteration is going to be of OSCQR and have tons of ideas about how we can continuously improve, both from our communities who are using it, as well as from our internal plans, and we've always envisioned OSCQR to be something that iterates; we want to practice what we preach. And for example, we have a set of standards that address mobile learning that we have been working on for a while. And I think you were part of the FACT2 task group, John, that helped us work on these mobile standards. So, we've been working with entities, groups within SUNY, to think about things like accessibility, mobile standards, courses with labs, language courses, and thinking about standards that we might be able to add on to OSCQR to make it a more customized experience based on the type of course, not all courses are going to have mobile learning necessarily specifically highlighted. My daughter who's 18, just had her first year of college, I found her last year writing a paper on her phone, and I was like, "Are you kidding me?" and then I started talking to some researchers who were making some suggestions for OSCQR for the mobile standards. And there is ample evidence to suggest that she is not the only one. [LAUGHTER]

John: And I think that was especially true with the sudden pivot back in March when many people who chose not to engage in online learning because they didn't have the computer resources to do that effectively in their homes, ended up adopting their phone as a primary means of interaction. And that was a challenge for many people, because mobile platforms are really good for many things, but, perhaps, writing papers may not be their optimal use... or taking extended essay exams and so forth in a mobile device... may not be the best way or the most efficient way of doing that.

Alex: You know, if that's the only device that the student has from home, because they only have one computer and their parent is working at home and they have limited access, that might be their only device. So, yeah, it's very, very surprising some of the things that we learned as a result of the COVID pivot, and beyond, even in researching the standards, the pitfall of assumptions. It's really hard to see around assumptions because that's the nature of them, you don't know. It's only when they kind of hit you smack in the head that you realize. So, you were asking about how OSCQR has evolved. One of the things that we changed between version 2.1 and 3.0, was we collapsed the instructional design and the accessibility standards together. There were some redundancies and we wanted to integrate them, rather than have them be two separate processes, for ease of use. What we found was, it was too much work, and so they were doing one review or the other. By removing the redundancies and integrating the accessibility standards from the get go, we were able to get down to 50 standards, which was much more doable. And then the other thing that we did was we organized the standards into categories. And so we have a course overview and information category, and those are all the typical things that you would want to see at the start of a course review. And that sort of set the stage for the online course. So, all of your syllabus and information documents are in that area. And whenever I do a course review, those are the first things I look at. I want to see what the expectations are, what the assignments are, how the students are going to be evaluated, what the learning activities are, what the percentages of the grade things are, those kinds of things. And so that gives you a good overall snapshot of the course. And it's also super, super important to start the course off in these areas really well, so that students are not confused, so that expectations are crystal clear, so that things are findable. So, that's the first category. Tools and technology is the second category and you want to really focus in on what additional tools or what tools and technology students are going to be asked to use during a particular course. What skills are required? What prerequisites there might be? Accessibility figures in here a little bit. The third category is design and layout. And I think a lot of the accessibility standards are in this section, and it really talks about how you chunk a course, how you lay out the different components of the course. It talks very specifically about some of the accessibility things like font size, and flashing text colors, using tables, slideshows, and all of those things. It gives specific suggestions about all of those different ways of presenting content. The next category is the content and the activities, helping faculty and instructional designers think about activities that are learner centered, that are targeting Bloom's in the correct space, depending on the discipline and level of the course. Again, there's some accessibility standards in this one, and thinking about the variety of ways that you can engage students in an online environment. Then interaction is the next category, and that's more specifically about the design of the learning activities, the expectations for feedback, any kind of netiquette expectations you might have, how you develop community, how you develop a sense of presence, both from the instructor and the student perspective, how that's actually scaffolded in the design of the course, how you break the ice, how you answer questions, how you facilitate interaction, and any kind of collaborations, and so forth. The last category is assessment and feedback. And this has to do with your grading policies, the methods that you use to assess mastery or learning, giving students opportunities to self assess or to check their understanding, the grade book and how that is set up and pointed to by the instructor in the course, and then how you solicit feedback from the students in the course to help you to improve the design of the course... to understand what's working well from their perspective and what could be improved. So, that's kind of the overall sort of buckets of standards. Like I said, there's 50 of them. And they fall into each of those six standards. And so that was one of the things that we did when we moved to version 3.0. Version 3.1 that we're currently in, was one when we developed the companion resource that goes with it. And so if you go to OSCQR.SUNY.edu, you'll find sort of the other half of the coin through the rubric and the dashboard. It's just a simple website that has a page for each of the standards. And each of the pages, as I described, has information in a consistent way that addresses an explanation, and there's a little video on each of the standards that has people from our community talking about how they have implemented this particular standard, why it's important, and any thoughts they have about the particular standard... which I think is super cool, because it's folks from our own community, and citations, and all of those other things that I mentioned earlier, are consistently on each of the pages.

John: As we're moving into a fall semester during a pandemic, where most institutions in SUNY and many throughout the country are engaged in this magical thinking that we're all going to somehow go back and despite the fact that the virus is spreading, especially in college age groups right now, as people have started going back to parties and other things, many institutions are going to try to imagine that that problem will somehow go away by the start of the fall in August. In case that magic doesn't occur and we move online, how might the OSCQR rubric be helpful for those faculty who have to transition to online teaching? How might they use that to make their transition perhaps more effective for students?

Alex: Great question. OSCQR was designed way before COVID with not a glimmer of COVID anywhere near it. So, it was intended to be a tool used by faculty and instructional designers to support fully online instruction, and perhaps blended instruction, but targeting the online component of blended instruction. And so it really does not have anything to say about any synchronous online or any primarily synchronous online courses. But, I would say that any course that will be offered in the fall during this pivot that we're all doing, could be informed and influenced positively by faculty taking a look at the standards for the online components of the course. As I've said other times, this stuff is not necessarily intuitive and in fact, it's different from a face-to-face class. It's not better or worse, it's just different and you're use different tools. You have different options and different limitations in an online teaching and learning environment than you do in a face-to-face teaching environment. And so some of these things are not necessarily intuitive and may actually be a feeling of resistance on some of them because you don't either understand them or they just may not make sense because you haven't actually experienced it yet yourself, either as a student or as an instructor. And so I think anything that helps people understand the unique aspects of an online teaching and learning environment are going to help you better prepare. So, when new or novice online faculty are faced with moving all are some of their instruction or content online, these standards are going to help you understand that better and help you understand what are the things to think about, what are the things to target and how. And I think the rubric in conjunction with the companion website would be a really good tool to use formatively. So, as you are reconceptualizing, as you are thinking about what you're going to do in the fall, how you're going to do it, for the pieces that are going to be primarily online, the rubric can give you some signposts, some goal lines, some suggestions for how to do that as best as possible, given the nature of the online asynchronous teaching and learning and learning. And I think that by learning more, by looking through and understanding the standards and what they're suggesting, and what they're trying to address, that will deepen your understanding of how to present your content most effectively online, how to facilitate collaboration and interaction, either online asynchronously or even online synchronously, a little bit. And it'll certainly help you think through issues regarding providing asynchronous feedback and thinking about authentic online assessment and doing that asynchronously. So, I think anything that helps one in a formative way to understand what are the standards that exist, that are research based, that we understand to positively and significantly affect the experiences of faculty and the learning of students, will be a good tool to explore and to leverage and to use. These are open and available to anyone. It's a website. But, I would suggest taking advantage of instructional designers to help you and of any professional development that might be offered by your campus, by your instructional designer, by SUNY, in addition to looking at the freely available resources that are provided to help walk you through a process, to guide you through a process, and I would look around your campus to to see what faculty might be in your department, or even outside of your department, who have experience teaching online. We have faculty across the SUNY landscape who have been teaching online for 20 plus years with vast amounts of knowledge and information in every discipline conceivable, and who have already made all of the mistakes and who have already developed all of the stuff and understanding that are willing to share. And we have an amazing community of faculty and instructional designers and people who have expertise in online learning within SUNY. It's so unique because we are such a large system. And we have been doing this for a long time in some areas. And so I would really encourage folks to see what's going on on their campus and what resources and supports are available on your campus to help you. You are not alone. You don't have to reinvent the wheel and if anyone is sitting there in front of a blank course management shell thinking "What the heck am I supposed to do with this?" just know that you don't have to start there. I have publicly posted in my self-paced and self-serve resources area, downloadable templates that will quickstart you into any learning mode, any design of course, whether it's primarily synchronous, I have one that's using Zoom, one that's using Ultra Collaborate, I have one that is intended for a blended instruction. I have one that is fully online. We're working on getting Moodle, Canvas, and Brightspace templates up. The common cartridges are posted already. So if someone wants to take the common cartridge and put it into their own system they can. So, you don't have to start from scratch. We have ways to quick start you, all OSCQR infused, following our OSCQR standards. And that's my worst fear, that we have some lone instructor who is out there, just really struggling and having to recreate wheels, that there's no need to duplicate those wheels when there are tools and resources and people out here who can help you and guide you to find exactly what you need. I'm hoping that anyone who's listening to this and feeling a little overwhelmed can know that they're not alone and can know that there are places to turn and people who are willing to work with you, with your campus, with your instructional designer, to make sure that you have what you need in your hand to help you get past those beginning stages of staring at this blank shell and not knowing what you're going to do next. I wake up at night thinking about those faculty. I've heard stories from faculty that have put hours and hours and hours of work into stuff that has nothing to do with their discipline, who are leaving their husbands and their children and their life on the side because 100% of their time is focused on climbing the learning curve of the learning management system, because they're trying to do a good job and they're just struggling because there's so much to know, and so much to do, and so much stuff out there. I can imagine how confusing and overwhelming It would be and I've talked to some of these instructors. And so I just want them to know that they're not alone and that there are people who can help them and we can help point them to the resources that can get them started more quickly, be more efficient, more effective, and ultimately happier at the end of the day and more successful, and their students too. That's the goal, is to have everyone be able to do what we're being asked to do as well as possible without killing ourselves in the fall. [LAUGHTER]

John: We always end, as you know, with the question, what's next? So, what's next for OSCQR?

Alex: What's next for OSCQR? We're in a bit of a struggle right now because there are so many other competing priorities. But we do have next plans for OSCQR. Like I mentioned, the mobile standards are pretty much ready to go. We are thinking about the next set of standards and wanting to work with folks like the FACT2 task groups to help us inform and influence next types of standards. We're thinking about courses with labs, language courses, courses with synchronous components, now in COVID land. And so we always envisioned OSCQR as a tool that would continuously evolve, continuously change. In my dream, like I've had this tool designed for 20 years. In my dream, when you begin to generate your OSCQR rubric, you would be presented with a wizard that would ask you certain questions about the type of course or the nature of the program that you're about to review. And then you would select from a menu: will it have labs? will it have hands-on activities? will it have whatever... the different types of things? and then you make your selections and then it will generate a rubric customized on your input. That's a ways off. But right now, we're going to potentially work on getting the mobile standards in there. [LAUGHTER] One step at a time, and I also need some technical resources to help assist. So, we're working on developing that capacity. So yeah, stay tuned because OSCQR is a living, breathing being and is kind of a toddler, I would say, right now and we'll be growing up over time and being improved along with the rest of us. As we continue to learn more about how people teach and learn well online, we will continue to enhance and expand what OSCQR does and how it does it, all for the purpose of helping faculty and instructional designers address the issue of quality online. John, I wanted to mention that we have this amazing community in SUNY and what I'd like to do in the links for this podcast is ask folks to join our online networking community so that we can continue the conversation. We have the SUNY Online Teaching Fellows role that allows us to collect people's names and send information out to them periodically when we have new tools and resources and supports. And so I'd love to invite everyone to become a SUNY Online Fellow, and then to join the online networking community so that we can join the OSCQR user group if you want and continue to have conversations around online course quality and OSCQR if folks are interested.

John: And you mentioned that OSCQR is a toddler, but it's a toddler that has become pretty well known. I remember the early days when it was just under discussion to now it's being discussed internationally.

Alex:Yes, and there's research on it, too. Like I have a link on the OSCQR website for all of the times I found it in the media and all of the research that I've seen done with it. And if anyone has any additions to those lists, I'd love to have them added. So, yeah, it is internationally used. There are hundreds and hundreds of institutions outside of SUNY that are using it at the institutional level and at the individual level, both in the United States and outside of the United States. I think of OSCQR as affectionately as a toddler, but maybe it's more of a teenager. I don't really know... maybe that metaphor doesn't work. It's certainly is well established, I agree. ...well known and certainly when the OLC adopted it in 2016 really elevated the standing of that tool nationally. And so I am grateful for the OLC for giving us that recognition.

John: Well, thank you, Alex. It's been great talking to you again,

Alex: Anytime.

[MUSIC]

John: If you've enjoyed this podcast, please subscribe and leave a review on iTunes or your favorite podcast service. To continue the conversation, join us on our Tea for Teaching Facebook page.

Rebecca: You can find show notes, transcripts and other materials on teaforteaching.com. Music by Michael Gary Brewer. Editing assistance provided by Ryan Schirano.

[MUSIC]

# data[].attributes.attachment_link.description
John: Many faculty are finding themselves teaching a fully online course for the first time this fall. In this episode we discuss how faculty can use the research-based SUNY Online Course Quality Review rubric, known as OSCQR, to help them design more effective online courses.

# data[].attributes.body
Recommended!

# data[].attributes.attachment_link.readable_text
Theme 4 of the Economics Network symposium is focused on helping lecturers with the design of their assessment in a changing world. It has four parts. Part 1 is based on asynchronous material for you to look at in your own time, ideally ahead of the virtual seminar on September 16th. Part 2 is engagement with online activities from September 8th 2020 to September 16th 2020. Part 3 is a live virtual seminar on September 16th from 3-5pm. You can book your place for the seminar via the CTaLE website here. Booking will be open until September 9th 2020, 5pm. Part 4 outlines next steps as we draw the Economics Network Virtual Symposium 2020 to a close.

Thanks to contributors

Theme 4 was co-organised by Parama Chaudhury, Cloda Jenkins and Stefania Paredes-Fuentes. It would not have been possible without the contributions of the following: Madeleine Adamson, Matthew Aldrich, Fabio Arico, Tim Burnett, Antonio Cabrales, Steve Cook, Silvia dal Bianco, Carlos Cortinhas, Peter Dawson, Anna Goddeke, Simon Halliday, Christopher Jepsen, Maria Kozlovskaya, Valerie Lechene, Michael McCann, Doug McKee, Paul Middleditch, Mario Morroni, Ramin Nassehi, Ravshonbek Otojanov, Dimitra Petropoulou, Marion Prat, Steve Proud, Michael Reynolds, Riccardo Soliani, Frank Witte and Michael Wycherely. This has been the 'summer that never was' for most of us, with no clear break between 2019/20 and 2020/21. We are extra grateful in these circumstances for people making the time to create and share these materials. It has been a great lesson in the feasibility and value of collaborative working, online and largely asynchronously!

# data[].attributes.attachment_link.description
Theme 4 of the Economics Network symposium is focused on helping lecturers with the design of their assessment in a changing world. It has four parts. Part 1 is based on asynchronous material for you to look at in your own time, ideally ahead of the virtual seminar on September 16th.

# data[].attributes.headline
Hexagonal Thinking: A Colorful Tool for Thinking

# data[].attributes.subject
Hexagonal Thinking: A Colorful Tool for Thinking

# data[].attributes.attachment_link.readable_text
Sponsored by Teaching Channel and UL Xplorlabs

While hexagonal thinking is not new in the world of business and innovation, it's just making its way into the classroom. It's a method for considering the connections between ideas and finding the nuances in those connections. If you're looking for a fresh framework for discussion and critical thinking, this may be just the thing.

How it Works

When you place an idea on a hexagon, it has six sides where connections could be made to other ideas. When you place many ideas on many hexagons, the discussion about where to connect what will be different every time.

Hexagons for the Vietnam-War era
(click all images for a close-up view)

If you gave the same seven idea cards to seven people, do you think you'd get the same seven webs of interconnected hexagons? Would the people explain the connections in the same way?

Nope.

And really, that's the beauty of hexagonal thinking. It provides a springboard for a totally creative discussion. When you give a small group of students a deck of hexagons and ask them to connect them however they choose, every group will come up with a different web for different reasons. Along the way they'll hopefully question each other and dig deep into the concepts on the cards, arguing about which idea connects more to an important concept and which example deserves one of those precious six sides.

The Step-by-Step

Let's imagine you've got your students in small groups, and each group is staring at a beautiful deck of hexagons.

Now what?

I recommend you kick off this activity for the first time by giving directions something like this:
Once you have your set of hexagons, it's time for your group to begin making connections between them. Your conversations now will be about showing how and why you think the different ideas and options connect. Everyone will see things differently, and that's OK. Just keep talking until you find the connections that stick. As you discuss your ideas, use the text to find supporting evidence for why you think your connections are strong ones. Each hexagon can connect to up to six others. Arrange and rearrange until you feel you have the strongest hexagon web in place that you can. Then begin explaining your connections with connection arrows, writing in why you have created intersections between key hexagons. Everyone in the group will contribute differently, and that's OK. You need to have people listening and moving pieces to create the web, people debating, people asking questions. By the end of your discussion, you should have an interconnected web of concepts along with clearly explained connections. If you're working with paper hexagons, tape or glue them down to another sheet of paper to secure them, or take a photo of your finished web. If you're working digitally, submit your two slides: the finished hexagon web, followed by the slide with your explanations.
There's a lot of room here for students to participate in their own unique ways. For me, part of the point of the activity is for students to see that everyone can take on different roles that matter.

Two Examples

Let's take a look at what this might look like in your classroom. In this case, I'll share digital examples created in Google Slides, since you may well be leading your discussions and small group activities via online channels this year. But paper hexagons are equally wonderful when you can use them.

Example 1: English Language Arts

First let's take a look at a hexagonal thinking discussion for an English class, wrapping up a unit on 1984.

Here's the set-up, with the terms listed in individual (moveable) text boxes.

Now students can discuss the placement of their terms/hexagons. In this case, rather than shifting paper hexagons around, they will go into the Google Slide, then drag and drop their terms across the slide and into the web of hexagons as they decide where to place them.

Now that the students have made their connections as they wish, they need to explain their choices. This can take place in many ways:

Students could record arguments for one or two connections on a tool like Flipgrid,

Students could write down their analysis.

Groups could collaborate to write explanations for several of their connections or present them back to the class.

Take a peek at what the written explanations look like for the web pictured above. In this case, students are sharing five explanations, but you could adjust that to suit your needs. Some connections are always going to be more interesting than others - the ones that generate the most debate, the surprising ones, the ones where three key sections link in one place. Have students write or present about the key connections, rather than about every single one.

Example 2: Science

Now let's take a look at how this might look in a science class. Imagine you're looking back over a unit on global warming, and helping students review the big ideas.

Here's the set-up: For this version, students can move the hexagons as well as the terms, creating any number of different webs. Then they use the connection arrows to point to the intersections they are going to explain in detail.

Student groups discuss the possibilities, dragging the hexagons and terms into place as they go. The final product could look one hundred different ways.

Once they have their web created, they make their explanations.

Creating Your Hexagonal Thinking Decks

You can create decks of hexagons with the terms already inside (let your students help you cut them out!) or print blank hexagon sheets and invite student groups to brainstorm the concepts to put inside before their discussion.

No matter how you create them, you want the hexagon decks to be filled with the key ideas of your recent class work, as well as interesting ties across disciplines and to the modern world. Hexagonal thinking makes it easy to cross disciplines and bring relevant modern connections into the discussion.

Let's say you're creating a hexagon deck for Jason Reynolds' book, Long Way Down. You'll fill it with characters, themes and style elements, but also with figures from the news, current political movements, similar books, and background information about the author.

Now imagine you're creating a deck for a unit covering an artistic period. You can put in key terms and artists, but also important authors and political figures of the period, and artists and movements of today that connect back.

As for making the shapes themselves, there are lots of time-saving options, whether you want to use digital hexagons or paper ones.

Digital Hexagons

Get access to your free copy of the digital hexagonal thinking toolkit featured above right here.

Create your own digital hexagons with PowerPoint or Google Slides by following these instructions:

Paper Hexagons

Use this online hexagon generator, created by Pam Hook.

Print and laminate these free rainbow hexagons, then add concepts with dry erase markers for quick discussions.

Print blank sheets of hexagons (you can easily create these in any size using the shapes tool in PowerPoint) to give out to small groups of students, then ask them to generate key terms, ideas, connections to other disciplines, connections to the modern world, etc. on their hexagons and cut them out. Then have groups switch decks so they are discussing a fresh list of ideas.

Create your own hexagon templates with terms you can change up whenever you want to do a hexagonal thinking discussion. Cut them out while watching New Girl on Netflix, or make the first step of the discussion in class to cut out the hexagons while beginning to discuss how they will connect.

A fun addition to any paper deck is a set of connection arrows. Print a few bubble arrows (again, the PowerPoint shapes tool makes it easy) on paper for students to put in at key intersections. Have students fill them in with their explanation of the connection. Sticky notes will also work well.

Wondering how to grade this?

There are a few ways you could grade hexagonal thinking. You could have everyone write a short reflection at the end about their contributions, and grade that. You could use it as a springboard into a writing assignment, asking students to explain and argue for key connections from the discussion, then grade that. You could have students present about a key connection on Flipgrid. Easy.

But I want to suggest a slightly more complex option too. Recently I interviewed Sarah Fine, co-author of In Search of Deeper Learning. One of her findings, after years of study across American high schools, was that we rarely ask students to truly collaborate in our core subjects, so we don't really create a culture of interdependence among them. They don't get to use their individual gifts to contribute to a final product created by the team, although that is often what real-world projects look like. Instead, we favor what she calls "lockstep learning." Everyone must do the same thing, separately.

So maybe for hexagonal thinking, you grade the final group web. And you ask the kids to share about how their different strengths allowed them to contribute to the final product.

Maybe some kids are super brainstormers, coming up with lots of great terms and ideas for what to put on the hexagons. Maybe one or two students in the group are great at lettering and drawing. They could create beautiful visual representations of the terms and ideas. Maybe some students shine at leading discussion. Maybe some are natural at making spatial and visual connections, and can help move the hexagons around as they listen to the group. Maybe some are excellent listeners, and can be taking notes about the connections based on what they hear from others. Maybe some are good mediators, who can help find common ground and calm down heated debates when discussion gets intense.

If you go this route, I really think it's worth discussing in advance what project teams are like in the workplace, and how people contribute in different ways. Then, at the end, you might let kids write a short paragraph about how they contributed based on their own strengths and skills, and whether they were happy with the way the group dynamics turned out.

Variations and Additions

Individuals or Partners

This would certainly work as an individual assignment or a prompt for partners as well. But with a small group, you'll get more perspectives on where to place what, which can be nice. You might consider giving students the option to work alone, with a partner, or in a group.

Adding Visuals

If you and your students are fans of sketchnotes or one-pagers, you can always add a visual layer to your hexagons. Invite students to illustrate their concepts with representative imagery and add quotations from their texts to bring more meaning for each idea into the conversation.

This is certainly not necessary, but adds dimension to the discussion and the eventual display of the concept web.

Hexagonal One-Pagers for Long Way Down

Gallery Walking

Once your students have completed their webs, you can do a gallery walk to see each other's concept webs and check out the connections their peers have made (this can be done online through a collaborative Google slides presentation), or have each group in turn present briefly back to the class about some of their most interesting connection points. The hexagons that caused the most debate will probably be the most fun to share back, and everyone will get to see how others mixed and matched the same hexagons in unique ways.

When you bring hexagonal thinking into your classroom -whether that's online or in person-you bring a fresh spark of energy to discussion. Your students will see things in new ways as they seek to connect wide-ranging ideas. If you're finding it difficult to get kids talking this year as they transition into new scenarios, you're definitely not alone. Hexagonal thinking just might be the tool you're looking for to help your classes over this transition and back into their critical thinking zones.

Come back for more.
Join our mailing list and get weekly tips, tools, and inspiration that will make your teaching more effective and fun. You'll get access to our members-only library of free downloads, including 20 Ways to Cut Your Grading Time in Half, the e-booklet that has helped thousands of teachers save time on grading. Over 50,000 teachers have already joined-come on in.

# data[].attributes.attachment_link.title
Hexagonal Thinking: A Colorful Tool for Discussion

# data[].attributes.attachment_link.description
If you've been looking for a fresh way to get students thinking outside the box, this may be just what you need.

# data[].attributes.body
CTALE

# data[].attributes.attachment_link.title
Moving to Adaptable Learning Design in Economics

# data[].attributes.attachment_link.description
This guidance was written by Professor Parama Chaudhury and Dr. Cloda Jenkins of UCL's Centre for Teaching and Learning Economics (CTaLE) and provides advice for instructors looking to move to an adap

# data[].attributes.attachment_link.readable_text
September 2020 | Volume 78 | Number 1
Grade Expectations Pages 67-70

Michelle Vanhala

My student Jacob loved to skateboard. His evenings were spent at the local park with friends, perfecting his ollies and definitely not doing homework, as I, his physics teacher, quickly came to realize. Despite his refusal to work outside of class, his careful attention during my lectures and strong math background contributed to high quiz and test scores. Based on our interactions, it seemed obvious to me that Jacob "got it" on most physics concepts. He had a C+ in my class.

Whitney was a perfectionist. She color-coded her notes and meticulously recorded even the most minute details into her notebook. She realized that in order to get into the University of Michigan, where her parents worked, she needed all As in her classes. She always turned her completed work in on time, and whenever her grade dropped, she stayed after class to request extra credit opportunities. Despite her perfect homework score, patterns in her assessments revealed some persisting misconceptions. Whitney had an A- in my class.

It was 2014, and I was a first-year chemistry and physics teacher at a rural, public high school in southeastern Michigan. On most days, I provided direct instruction while students filled in guided notes and then completed practice problems. Any unfinished problems were done as homework, due at the start of class the next day. My assessments were mostly multiple choice so that I could quickly run bubble sheets through the Scantron machine to get a number to enter online. Eighty percent of each student's grade came from these types of assessments, while homework and classwork accounted for 20 percent.

In meeting with students and parents, I saw frustrating patterns: There were many students (like Whitney) who had high averages on homework and classwork, but disappointing quiz and test scores. "My daughter gets test anxiety. She just doesn't test well," one parent rationalized. "How can my son study better for the tests? Is there a study guide that shows what will be on the test?" another asked. She may indeed have had some test anxiety and he certainly may have benefitted from more studying, but I struggled to articulate the real problem, which was that these students just didn't understand the content well, despite near perfect homework grades.

On the flip side, there were students like Jacob who aced their tests, but failed to complete or turn in homework. Their low homework scores brought their grades down significantly. "Is there any extra credit?" was the common question I got from those students and their parents. A last-ditch attempt to earn credit often turned into a negotiation: "What if he can turn in all of these missing homework assignments this week? Do you think that could bring him up to a B for the class?"

In the years since 2014, I've attended multiple professional development sessions and workshops centered around issues of grading. I've heard similar frustrations echoed by teachers across the country. The blame is often placed on the students rather than teachers acknowledging the issues inherent in the system that promote a grade-centered mindset: "Students need to take responsibility and ownership of how their choices affect their grade," I've heard.

While there is truth in these assertions, I've often wondered: To what degree are the grades I assign reflective of what my students actually know? Am I just assessing compliance? In what ways is our educational system set up to perpetuate this grade-centric mindset, and how can we as teachers leverage meaningful change? How can we cultivate responsibility and ownership of not just grades, but of the learning that the grade should reflect?

The "No Grade Plan"

I don't have all of the answers, but these questions make me think back to a teacher I had my first year of college, Dr. Phame Camarena.

Dr. Camarena, or Phame, as he asked us to call him, organized his human growth and development class to be a balance of direct instruction, activities, discussions, guest speakers, and group projects, all designed to introduce the foundational theories of sociology in the context of education. The capstone project for the course was known affectionately as "The PDP": The Personal Development Project. While teaching us about the theories behind childhood, adolescence, and adulthood development, Phame wanted us to experience these concepts in action. He challenged us to complete 10 activities that stretched our comfort zone and to reflect on these experiences in a summative portfolio assessment. One student became a vegetarian for a month as part of this project, someone else spent a week walking to class without shoes, and some of us even went to a local dance club for the first time in our lives.

One activity that Phame presented to us that fulfilled a requirement for this assignment was the "No Grade Plan." "This is for those of you who are perfectionists," he explained. "It's for the students who check their grades online every day and obsess about anything less than an A." The student he was describing was 100-percent me, and I knew that the "No Grade Plan" would be a significant personal challenge.

The handful of students who opted in to the "No Grade Plan," which included me, received detailed feedback from Phame. Every assignment for the course was returned with up to a page of responses and questions. The assignments did not, however, have a final score or grade on them. Phame kept that numerical grade secret until the final report card.

At first, I was fine with this. "I am an all As student after all, so what do I need to worry about?" was my initial reaction. My anxiety grew, however, after skimming through Phame's helpful but lengthy feedback on my first essay. I remember attempting to tally up a final score: "Three questions and one positive comment ... . My classmates got a score out of 10 points on this assignment, so I probably got 8 out of 10." I reasoned. "Is that a B-?"

At a certain point, I began to come to terms with the realization that there was no way for me to quantify the feedback I was receiving. My attention turned instead to the content of the comments, and I worked with each successive assignment to address Phame's questions. With each draft, I began to think deeper and more critically, organizing my thoughts more coherently based on his feedback. By the end of the semester, I was confident that I had not only learned the class content thoroughly, but that I had also grown in my communication and writing abilities. I couldn't predict my exact grade, but I was confident that I would do well enough. I also appreciated that there was something unquantifiable about my learning and growth that semester.

Grading for Learning

I knew that a "No Grade Plan" would never fly at my public school, where I was required to enter a certain number of grades by a certain time in the semester. So how could I make grading in my own classroom meaningful and reflective of learning?

At a teaching conference in the summer of 2015, I attended a session on standards-based grading, and the first seed was planted. The expectation that students should have a "Bill of Rights" in regard to grading was an idea that I picked up from this session. I took some time to think about my own teaching philosophy and drafted my own "Student Bill of Rights," which included the following three principles:

1. Grades should be an objective reflection of what a student actually knows and can demonstrate, not a reflection of behavioral, personal, or socioeconomic characteristics.

2. Grade reporting should communicate useful information. Grades should be a record of an individual's academic strengths and weaknesses, able to be used for improvement.

3. Mistakes are an opportunity to learn, and everyone learns at different paces and in different ways. Students should therefore be allowed multiple opportunities to practice and demonstrate learning of clearly communicated learning objectives.

From there, I thought about what this could look like in my classroom and began the work of drafting learning objectives for each unit. In later years, these learning objectives morphed from "I can" statements into questions that students were prompted to answer at the end of a lesson. Every single task, assignment, and assessment clearly communicated the learning goal at the top so that students understood the assignment's purpose.

I stopped grading homework and classwork entirely, which I described to students as "practice" toward mastering each learning goal. Grades were calculated based only on summative assessments completed in class, which varied in format but were all designed to be an objective, individual demonstration of what students understood regarding that one particular learning goal. On practice problems (no longer referred to as "homework" or "classwork"), I gave detailed feedback so that learners could improve. But these problems were not graded.

In my online gradebook, I stopped categorizing assignments as "Homework #1" or "Quiz #5" and instead began writing out the learning goal in its entirety as the header. Students received one grade per learning goal, a grade that came from their performance on the summative assessment. In this way, my printed grade reports became a list of what each student did and did not understand. A student could clearly see that he had an A on "How do I balance chemical equations?" but a failing grade on "How can I calculate the reactants or products based on mole ratios?"

But a failing grade for any learning goal was not permanent. It didn't seem fair to me to say, "Hey, you all must know how to calculate reactants and products based on mole ratio by Friday, March 15th, 2019, at 10:30 a.m." I encouraged students to learn from the mistakes that they made and retake summative assessments. I made multiple versions of assessments for each learning goal. If a student showed me that she understood a concept at a later date (that is, up until my principal told me that my grades were due), then I wanted the gradebook to reflect that.

This transition wasn't easy. It took time to articulate the objectives, align materials for each, and create multiple versions of each assessment. I then had to grade the assessments and reassessments. But that time was worth it in light of the changes I began to see in my students' attitudes.

Learning became less about the grade and more about the learning goals. The question, "What can I do to boost my grade?" became, for instance, "Ms. Vanhala, I still don't understand mole ratios. Can we meet after school to go over my mistakes on this assessment? And then I'll review and schedule a retake for next week." Students were able to see explicitly how their grades connected to what they knew (or didn't know), and they weren't permanently penalized for a mistake, be it academic or social. A value was placed on learning from mistakes.

While the initial transition took a lot of work, having a grading policy that finally aligned with my educational philosophy was refreshing. After the framework was in place for one unit, modifying, adjusting, and building on the foundation became easier. I now spend less time on grading, but the feedback that students receive is more meaningful. As a young teacher who had already been feeling burned out on grading, I feel that this new method will be more sustainable in the long term.

It's not a perfect system. The creation and modification of assessments that are valid, reliable, and multidimensional rather than just multiple choice is a work-in-progress that never feels 100 percent "done." I also wonder about how well this system prepares my students for college classes, some of which may be graded in a more traditional manner. But I would rather students leave my class at the end of the semester having learned and retained a solid foundation of content than with a belated understanding of the importance of deadlines. The growth mindset and persistence I've seen in my students show that, although my grading system may still be in development, the work to emphasize learning and make grading meaningful is worthwhile.

Michelle Vanhala is an environmental science teacher at Washtenaw Technical Middle College in Ann Arbor, Michigan. She was recognized as a 2014 Knowles Teaching Fellow and as the 2020 Michigan Science Teacher of Promise. Follow her on Twitter.

# data[].attributes.attachment_link.description
September 2020 | Volume 78 | Number 1 Grade Expectations Pages 67-70 Michelle Vanhala My student Jacob loved to skateboard. His evenings were spent at the local park with friends, perfecting his ollies and definitely not doing homework, as I, his physics teacher, quickly came to realize.

# data[].attributes.body
A Lesson Learned on Grading in a Pandemic. | THE TEMPERED RADICAL

# data[].attributes.attachment_link.readable_text
If you follow me on Twitter, you know that I've been talking a LOT about grading practices in the pandemic. That all started with this post:

Note to educators: Taking points off for late work and/or not allowing kids to rework tasks right now is bad practice. Kids don't have the same supports that they do when working with you at school. To expect them to be perfect at this right now just isn't reflective of reality.
- Bill Ferriter (@plugusin) September 23, 2020

The responses to that message have been all over the board - but the ones that leave me the most riled have argued that students have plenty of time to get their work done, that teachers shouldn't be expected to go looking around for late work from students, and that parents who aren't helping their students to complete their work are just "lackadaisical."

Seriously. Scroll through the responses.

Some of them are appalling. Not only do they show a lack of empathy and understanding during the middle of a pandemic, they show a lack of understanding of sound instruction. It was disheartening to me.
Lemme tell you what I've done this quarter:

1. I've told my students that I will take late work all the way until the end of the quarter, as long as they email me when it is done so I know to go and give them a grade.

2. I've also told my students that they can raise any score that they want to raise - and given them a list of things that they can do to add points to assignments.

3. Finally, now that we are at the end of the quarter, I told my students that I would exempt zeros on practice tasks and give them the average that they earned only on the quizzes that we have taken this quarter.

The results have been pretty darn remarkable. Here's what I've noticed:

1. Students who have been behind all quarter have been emailing me all week with completed work - even though they know that I am exempting zeros.

2. Students who have an academic record that is full of Cs and Ds are doing TONS of extra work, raising their scores on darn near every assignment in my class.

3. I've had dozens of kids reach out asking about ways to improve their average.

4. At this point, the only students with significant amounts of missing work have REALLY challenging learning circumstances at home. They deserve to have their work exempted until I can find a better way to help them succeed in spite of their challenges at home.

Here are some statistics to back up these observations:

1. I teach 164 students this year. If you remove the students with the most challenging home circumstances, I have 18 missing assignments. That's out of 1,476 possible papers.

2. That means less than one percent of the work that I assigned is missing.

3. I have also received and graded 78 extra credit tasks and/or reworks over the course of the quarter. That's additional work that no one was required to do.

To be honest, I was surprised by my students' reaction to my grading policy.

I didn't expect students who had zeros exempted to work so hard to get those tasks done. After all, the zeros didn't count against their average. I also didn't expect so many students who have traditionally struggled in schools to do TONS of extra tasks in order to raise their grades, given that most of them had passing grades already.

So, what do I attribute that reaction to?
I think my students recognize that they CAN be successful in my room - and so they've decided to lean in and take advantage of the opportunity.
See, that's the thing teachers forget: EVERY kid wants to succeed. But some have learned after years of struggling that it's not even worth trying.

Once they figured out that I was serious about helping them to earn a grade that they could be proud of, they were willing to take care of their missing work and finish a few extra credit tasks while they were at it.

The best part of all of this is that success is like a snowball for struggling kids. Once they experience it, they become more confident in themselves as learners. That confidence will result in those same students taking additional risk in my room moving forward - whether that means answering questions in our live lessons or attempting tasks that they would have otherwise avoided.

Did my grading choices require extra work from me?

Sure. I was getting lots of emails from my students about missing work, extra credit and reworks that I had to look at each day.

But I bet that will taper off over the course of the school year, y'all. Now that my students realize that they can be successful in my room, I bet they are going to do more of their work the first time around.

And if it doesn't taper off, I don't care. I'll just give fewer assignments to create more time for me to help students succeed in my room.

In the end, my goal is to leave every kid convinced that they are a capable and competent learner in my room - and that can only start when they are invested in the work that we are doing together.
Related Radical Reads:

# data[].attributes.attachment_link.title
A Lesson Learned on Grading in a Pandemic.

# data[].attributes.attachment_link.description
If you follow me on Twitter, you know that I've been talking a LOT about grading practices in the pandemic.