Monitoring examiner reliability- Ian Salinas by IAN JOSHUA RESENDIZ SALINAS https://padlet.com/ianresendiz_g/sawed5k4hh60t7n Hecho con la mejor de las intenciones en-us 2021-07-05 05:03:21 UTC 2025-11-02 02:45:06 UTC hello@padlet.com The importance of monitoring examiner reability ianresendiz_g https://padlet.com/ianresendiz_g/sawed5k4hh60t7n/wish/1636877252 Intra- rather reability:
It is usually measured by means of a correlation coefficient or through some  form of analysis of variance.
Eg: The marks that four different examiners  have given to the same set of written scripts can be contrasted.
If the different examiners have given the same marks each time,  the means and standard deviations will be identical.  Analysis of variance is a way of assesing how significant any differences might be.
(See any introductory statistic textbook for more details).

Inter- rather reability:
It refers to the degree os fimilarity between different examiners: can  two  or more examiners, without influencing one another, give the same marks to the same set of scripts or oral performances. 
It is essential that each examiner try to match the "standard" all the time. This standard is set either by the Chief Examiner or the standardising committee.
This reliability is also measured by a correlation coefficent or by some form of analysis of variance. ]]> 2021-07-05 05:23:12 UTC https://padlet.com/ianresendiz_g/sawed5k4hh60t7n/wish/1636877252 Central marking ianresendiz_g https://padlet.com/ianresendiz_g/sawed5k4hh60t7n/wish/1636894275 There are three ways of monitoring:

Sampling by the Chief Examiner or Team Leader:
When marking take place centrally, examiners are usually divided into teams: each team has a coordinator and a maximum of 10 or 12 examiners. If the test is a small one, there may only be one team, coordinated by the Chief Examiner.
If there are more candidates, there may be several teams, each one coordinated by a Team Leader.

The examiners shuld be instructed to mark as they were trained to mark, and should be especially careful not to write on any candidate's script.
The sampling procedure should being soon after the marking begins.
Each examiner is expected to mark a certain number of scripts on the first day of marking.
 The TL collects a percentage of marked scripts from the examiner, and reads through them again in order to give an independent mark.
TL needs to discuss with the examiner, since the aim of the discussion is to narrow the differences in opinion between the two párties.  Sometimes the examiner will be able to convince the TL to reconsider, but in instructions where ampling takes place it is generally assumed that the TL will know best and in the end the examiner will have to reconsider.

Using reliability scripts

The second method of monitoring marking is to ask each examiner independently to mark the same packet of reliability scripts.
These scripts will have been chosen by the Chief Examiner to represent different points on the ratinf scale and different problems that examiners are faced with.
There are two way in wich the Team Leader can determine wheter the examiners are marking in the way they are supposed to be marking.
The quickest way, known as "eyeballing": is to place each examiner's socres side by side with the committees's scores.

"Eyeballing", this will reveal only the most conspicuous of problems. A more informative method is to correlate each examiner's marks with the standardisting committess's marks, and to comparemeans and standard deviations, TRhe correlation will indicate whether the examiner has ranked the reliability scripts in the same order as the committee (a reasonable correlation to aim for would be).

Routine double- marking.

It is to require routine double-marking for ebery part of the examn which requires a subjective judgment. This means that every piece of writing is marked by two different examiners, each working indeendently. 

]]> 2021-07-05 05:37:39 UTC https://padlet.com/ianresendiz_g/sawed5k4hh60t7n/wish/1636894275 Alternatives when marking is carried out elsewhere ianresendiz_g https://padlet.com/ianresendiz_g/sawed5k4hh60t7n/wish/1637011611 Marking in examiners' homes. 

The procedure where TL analyses a sample of he examiners' scripts mus be modified. If examiners are marking at home they may not be in a position to guardantee to mark a certain number of scripts a day. 
The TL access to scripts that the examiners have marked at all times of the day and under varying conditions such as If the examiners choose their own sample, they may send scripts which they have marked when their judgement is fresher, or scripts which they have spent more time on and considered more carefully.]]> 2021-07-05 07:27:58 UTC https://padlet.com/ianresendiz_g/sawed5k4hh60t7n/wish/1637011611 Monitoring the marking of writing. ianresendiz_g https://padlet.com/ianresendiz_g/sawed5k4hh60t7n/wish/1637518052 The LCCI ,  in their Handbook of Duties of Examiners and Moderators for Business Studies Examinations they have a page the call "Standard Re-scrutiny Procedures". The procedures are:

]]> 2021-07-05 16:07:07 UTC https://padlet.com/ianresendiz_g/sawed5k4hh60t7n/wish/1637518052 ianresendiz_g https://padlet.com/ianresendiz_g/sawed5k4hh60t7n/wish/1637526911 2021-07-05 16:16:35 UTC https://padlet.com/ianresendiz_g/sawed5k4hh60t7n/wish/1637526911 Monitoring the markinf of speaking ianresendiz_g https://padlet.com/ianresendiz_g/sawed5k4hh60t7n/wish/1637528824 Oxford-ARELS published a short description of how it monitors the marking of speakin, in a booklet entitled Oxford-ARELS Examinations in English as a Foreing Language.

The description is as follows:

The candidates' tapes are all matked in the UK by qualified examiners. At the two higher levels, independent marking by two examiners is mandatory; and If they disagree by more than a small amount the senior examiners is brought in to give a third, and definitive judgement.

At the Preliminary level all tapes which, at first marking, are close to a grade boundary are double. amrked; and If there is more than a minor disafreement, a third marking is called for.]]> 2021-07-05 16:18:44 UTC https://padlet.com/ianresendiz_g/sawed5k4hh60t7n/wish/1637528824 Discussion ianresendiz_g https://padlet.com/ianresendiz_g/sawed5k4hh60t7n/wish/1637534167 The most common doing this  in examination boards is by sampling the examiners' makrs and asking for adjustments to be made if the marking is not satisfactory, In tests with large populations the sampling is done b y TL, whose own judgments are often not questioned. 
The TL marking is sampled ny the Chief Examiner whose judgement is final.
The descriptions they gave were useful, but there are several questions that need clarification:

1- Why do so many boards use sampling rather than routine doble-marking?

2.- Why do some boards sample only at the beginning of marking, and not all the way through?

3.- Why do some boards ask the examiner to choose the scripts which are to be sampled, instead of conducting random sampling?

4.-Do all boards ask examiners who have been "out of line" to send adjusted scripts back for further sampling?


Clarificartion of these points would be valuable for institutions that are settinf up new tests and wish to adopt practical means of monitoring without sacrificing too much reliability.
]]> 2021-07-05 16:24:34 UTC https://padlet.com/ianresendiz_g/sawed5k4hh60t7n/wish/1637534167