Reflecting on reliability by Chartered College of Teaching https://padlet.com/charteredcollege/reliableclead Share your reflections en-us 2014-01-08 18:23:48 UTC 2022-08-21 10:36:44 UTC hello@padlet.com Triangulation https://padlet.com/charteredcollege/reliableclead/wish/1663818336 I believe the only way to increase reliability is to use triangulation, rather than single piece of assessment. Situations in which reliability is not essential is when you are using the assessment as a teaching tool e.g. to help students identify gaps in their own revision/knowledge-the Evaluation is self driven. ]]> 2021-07-29 05:44:25 UTC https://padlet.com/charteredcollege/reliableclead/wish/1663818336 Aggregation: https://padlet.com/charteredcollege/reliableclead/wish/1669000240 If tests are such high stakes then multiple sources and pieces should be sought. Triangulation, as already suggested would support this. Of course - keeping the construct consistent.]]> 2021-08-04 14:58:24 UTC https://padlet.com/charteredcollege/reliableclead/wish/1669000240 Reliability enhancements https://padlet.com/charteredcollege/reliableclead/wish/1675664155 As the article suggests, the higher the stakes for the students, the greater the reliability should be. Thus factors negatively affecting reliability should be reduced for higher stakes testing. An example of this might be the sitting of mocks in full exams conditions to reduce environmental factors that influence reliability.]]> 2021-08-11 15:58:45 UTC https://padlet.com/charteredcollege/reliableclead/wish/1675664155 Formative use of summative assessment https://padlet.com/charteredcollege/reliableclead/wish/1679336730 I agree that the triangulation of different assessments is crucial, as well as the careful planning of internal assessments to reduce error and increase reliability. Several factors are at play here such as schools being clear about their assessment policy overall, but acknowledging that assessments may need to be different in different subject areas.
In addition, assessments being fit for purpose so sometimes low stakes activities, frequently repeated. Accurate feedback to students is imperative and when used effectively this helps to consolidate their knowledge base, and build their confidence in that subject area.]]> 2021-08-15 07:18:10 UTC https://padlet.com/charteredcollege/reliableclead/wish/1679336730 Reliability https://padlet.com/charteredcollege/reliableclead/wish/1804189166 I think I have at times had high levels of intra-rater variation.  I am trying to get round it by marking more demanding assessment questions first, to avoid fatigue and inconsistency.

I am bias too.  If I am unsure whether to give a mark I do look at the students name and commit a cognitive shortcut of deciding what they might have meant. That needs to stop.

Triangulation sounds like it would provide more reliable results and I can see it being good for assessing skills, but could we  assess all aspects of subject knowledge using it, the quantity of data generated would be huge?

If assessing subject knowledge is the aim, we must start by ensuring that the reading age does not exceed the age of the students sitting the exam (apparently maths GCSE has a reading age of 17!?)]]> 2021-10-09 15:02:47 UTC https://padlet.com/charteredcollege/reliableclead/wish/1804189166 Reliability of tests https://padlet.com/charteredcollege/reliableclead/wish/1847369496 Where the assessment is determining setting I think that we need to really work harder as teams to maximise reliability particularly as we know what the impact on self esteem and mindset there is when moving ‘down’ a set.  
Some of my assessments are to test the quality of my communication as well as what students have learnt so I am not as bothered about reliability if I am using the test to see where all students understood or struggled with content. 

I am interested in inter-rater reliability as I think in a busy school year this suffers, school leaders need to provide the opportunity to discuss and respond to this.



]]> 2021-10-27 06:35:57 UTC https://padlet.com/charteredcollege/reliableclead/wish/1847369496 Reliability and TAGS https://padlet.com/charteredcollege/reliableclead/wish/1852249819 One question in the back of my mind this year was how reliable were TAG grades across the country in my subject area. With the one constant of a series of examination papers taken away, it almost felt that the reliability had been taken away. I feel like with such high-stakes assessments, reliability in TAGs should have been more thought through centrally and more prescribed to centres to avoid variations. Have I measured my students too harshly or too lightly through the process we followed?


]]> 2021-10-28 19:16:20 UTC https://padlet.com/charteredcollege/reliableclead/wish/1852249819 https://padlet.com/charteredcollege/reliableclead/wish/1858467166 Yes teachers should of course aim to maximise the reliability of their assessments, but they should be given time to achieve this. We are given no time to moderate end of unit tests at key stage three for example, so therefore moderation rarely happens. ]]> 2021-11-01 11:21:44 UTC https://padlet.com/charteredcollege/reliableclead/wish/1858467166 Is reliability really worth worrying about? https://padlet.com/charteredcollege/reliableclead/wish/1871431569 I would be tempted to say that the only situation where we want to enhance reliability is for mock GCSE and A-level exams. For most other situations I would think reliability is automatically enhanced by repeated assessments and aggregated data. For example, setting in my school happens based on performance in english, maths and science (multiple tests), which I would argue makes the decision fairly reliable.]]> 2021-11-06 11:24:51 UTC https://padlet.com/charteredcollege/reliableclead/wish/1871431569 Reliability https://padlet.com/charteredcollege/reliableclead/wish/1952350404 In my dept we try to ensure consistency in end of year exams by having one staff member mark each section. There is not usually time for moderation, so intra-rater reliability is usually the problem. 
For class tests this is a similar problem. 

I'm not usually concerned with reliability of formative tests as the focus is on improving student performance in particular skills rather than simply in the outcome. 

Even then, there is usually some limited discussion about marking bands and some very limited moderation.

Improving reliability in the summative tests are most important.]]> 2021-12-16 17:55:06 UTC https://padlet.com/charteredcollege/reliableclead/wish/1952350404 Reliability - Science https://padlet.com/charteredcollege/reliableclead/wish/1953459829 Exams - sample cross moderation makes a MASSIVE difference to reliability of whole cohort data.  E.g. A new teacher may be marking incorrectly and this needs to be spotted. But also experienced teachers make mistakes (sometimes ingrained misconceptions) and moderation helps iron these out. 
Tests - Lower stakes and less time to moderate therefore less likely to happen!  ]]> 2021-12-17 10:11:23 UTC https://padlet.com/charteredcollege/reliableclead/wish/1953459829 Reliability https://padlet.com/charteredcollege/reliableclead/wish/1954695088 This module has made me reflect on how little thought most teachers give to the reliability of assessments. In a primary school, only a small proportion of our assessments have high stakes (for the children, at least). 
I have been concerned about the reliability of our writing assessment - as it is subjective. This year we are taking steps to make this more reliable. We are doing more in-school and more local authority moderation. 
The end of year judgements need to be reliable as they contribute to performance related pay and impact on the next teacher’s progress data. I don’t feel that our SLT appreciate the implications of this.]]> 2021-12-18 10:17:55 UTC https://padlet.com/charteredcollege/reliableclead/wish/1954695088 Reliability https://padlet.com/charteredcollege/reliableclead/wish/2055457548 This is really interesting to consider. Assessments which are used for the purposes of big decisions (e.g. setting) should aim to be maximally reliable. I am torn between thinking that such decisions should be made on the basis of multiple assessments, and then wondering whether that just increases the opportunities for reliability to be compromised? 
Ideally, I think all assessments should be used to help teachers respond to student needs. If this is consistently the case, then perhaps drawing evidence from multiple sources does mean that the reliability of any one particular assessment score becomes less important - it is only part of the bigger picture used to make an inference about learning.]]> 2022-02-18 15:35:23 UTC https://padlet.com/charteredcollege/reliableclead/wish/2055457548 Reflections https://padlet.com/charteredcollege/reliableclead/wish/2115781670 The reliability of end of topic tests and more importantly the mock exam data - this can often be used to set the tier for Science exam entry. This decision should not be on those results alone, they should have more of an importance on the students background as that will play a bigger role in their performance in the exams. ]]> 2022-03-27 18:04:46 UTC https://padlet.com/charteredcollege/reliableclead/wish/2115781670