# data[].attributes.headline
La conception des questions à choix multiple

# data[].attributes.subject
La conception des questions à choix multiple

# data[].attributes.body
https://www.iml.unibe.ch/attachment/56/download/Manuel_QCM_fr.pdf

# data[].attributes.headline
Item analysis of multiple choice questions: A quality assurance test for an assessment tool

# data[].attributes.subject
Item analysis of multiple choice questions: A quality assurance test for an assessment tool

# data[].attributes.attachment_link.readable_text
Abstract

Keywords: MCQs, Item analysis, Difficulty index, Discrimination index, Distractor effectiveness

Introduction

Competency-based assessment centres on performance of the students, that is, whether they demonstrate competence. Assessment plays an important role in helping to interpret the magnitude of a student's ability and their own learning progress.,

The key features of any assessment are content and construct validity (a method assesses what it intends to assess), reliability (how well a score represents an individual's ability) and objectivity (assessment with single correct answer) which can be recalled based on higher-order thinking skills and problem solving. All methods of assessment have some strengths and shortcomings.

Multiple choice questions (MCQs) are commonly used in assessments at undergraduate and postgraduate medical examinations because they are efficient, reliable and can be conveniently standardized. The quality of MCQs is important because of its effect on the students' overall competency level during their assessment. Well-constructed MCQs allow the assessment of higher-order cognitive skills such as interpretation, analytical and critical thinking, application or synthesis in the framework of Bloom's taxonomy and Miller's pyramid. Framing the MCQs is a challenging task. A meticulously built up MCQ question bank after thorough item analysis is a handy tool for any academic institute for conducting the assessments.

The item analysis of an assessment tool provides input about validity and reliability of the item. The MCQ item analysis consists of the difficulty index (DIF I) (percentage of students that correctly answered the item), discrimination index (DI) (distinguish between high achievers and non-achievers), distractor effectiveness (DE) (whether well the items are well constructed) and internal consistency reliability (how well the item are correlated to one another). Each item is evaluated for these indices because if an item is flawed, then it can become a distractor and the assessment can fail.

We have undertaken the project of item analysis of MCQ tests conducted for first-year Bachelor of Medicine and Bachelor of Surgery (MBBS) students in physiology. This item analysis will help to retain quality MCQs, discard or reframe items which have not been well framed. This will help to build a MCQ question bank comprising 'ideal' MCQs. This exercise will also help the faculty to construct good MCQs in future.

Material and methods

Three internal assessments of 150 first-year MBBS students were conducted which included 90 MCQs. Each MCQ consisted of a stem and 04 options with only one correct response and three distractors. The correct response was awarded one mark and the wrong response given zero mark. There was no negative marking. The upper 27% (41) students were considered high achievers (H) and lower 27% (41) as low achievers (L). Each item was analysed for four indices, that is, DIF I, DI, DE and internal consistency reliability.

DIF I/facility value/P value

It is the percentage of students in the high and low achievers' group who answered the item correctly. It ranges between 0% and 100%. The criteria for classification of the DIF I are as follows: DIF I <30 (too difficult), DIF I between 30 and 70% (good/acceptable/average), DIF I >70% (too easy) and DIF I between 50 and 60% (excellent/ideal)., , It was calculated using the formula DIF I or P = ( H + L) × 100/ N, where H = number of students answering the item correctly in the high achieving group, L = number of students answering the item correctly in the low achieving group and N = total number of students in the two groups (including non-responders).

DI or d value

The DI is the ability of an item to differentiate between students of higher and lower abilities and ranges between 0 and 1. The criteria for classification of the DI are as follows: DI ≤ 0.20 (poor), 0.21-0.24 (acceptable), DI 0.25-0.34 (good) and DI ≥ 0.35 (excellent). It was calculated using the formula DI = 2 × (H-L)/N, where the symbols H, L and N represent the same values as those mentioned earlier.

Distractor Effectiveness

DE is determined for each item based on the number of non-functional distractors (NFDs) (option selected by <5% of students) in it. The items were categorized on the basis of numbers of NFDs in MCQs, that is, an item having three NFDs, two NFDs, one NFD and zero NFD, the DE would be 0% (poor), 33.3% (moderate), 66.6% (good) and 100% (excellent), respectively., ,

Internal consistency reliability

The SEM is directly related to the reliability of the test. It is an index of the amount of variability in an individual student's performance due to random measurement error. The standard deviation of the distribution is called the SEM and reflects the amount of change in the student's score which could be expected from one test administration to another.

Cronbach alpha is a measure of internal consistency, that is, how closely related a set of items are as a group. It is a measure of reliability. The value of alpha between 0.8 and 0.9 falls in good range. KR-20 is a measure of reliability for a test with binary variables (i.e. answers that are right or wrong). Reliability refers to how consistent the results from the test are or how well the test is actually measuring what you want it to measure. The scores for KR-20 range from 0 to 1, where 0 is no reliability and 1 is perfect reliability. The closer the score is to 1, the more reliable the test. Just what constitutes an 'acceptable' KR-20 score depends on the type of test. In general, a score of more than 0.5 is usually considered reasonable.

Skewness and kurtosis values were measured. Skewness is a measure of lack of symmetry. Kurtosis measures if data are heavy tailed or light tailed relative to normal distribution. The values of skewness and kurtosis between −2 and +2 are considered as acceptable to prove normal univariate distribution.

Statistical analysis

Statistical analysis was performed by using MS Excel 2010 and SPSS, version 20.0 (IBM, Armonk, NY, United States of America).

Results

The results of study showed that the scores of 150 students ranged from 30 to 91%. The test score of class ranged from 27 to 82 (total score 90) with a mean test score of 47.35 ± 11.6 (mean ± SD). The median was 44, and the interquartile range was 20. The skewness and kurtosis values were 0.59 and 0.69, respectively. The reliability measure of test items by KR-20 was 0.71, with a Cronbach alpha of 0.85 which is in the range of good. The SEM was 1.22 ().

Results of the DIF I showed that of 90 MCQs, 74 (82%) MCQs had good/acceptable levels of difficulty (DIF I 30-70%), whereas seven (8%) were too difficult (DIF I <30%) and nine (10%) were too easy (DIF I >70%). Among all MCQs, 34 (38%) MCQs had excellent/ideal levels of difficulty (DIF I 50-60%) ().

Of total 90 MCQs, 18 (20%) MCQ items had a poor DI (≤0.20), 11 (12%) items had an acceptable DI (0.21-0.24), 32 (36%) showed a good DI (0.25-0.34) and 29 (32%) items showed an excellent DI (≥0.35) ().

Table 3

Discrimination index (DI)

Interpretation

Items (%)

Discrimination index (mean ± SD)

Pearson correlation of the DIF I and DI was also analysed for each of the MCQ items. We found a weak correlation between the DIF I and DI (r = 0.140, p < .0001).

Of the total 270 distractors for 90 items, 198 (73%) were functional distractors (FDs) and 72 (27%) were NFDs. Of all items, two (2%) items had poor DE (0%), eight (9%) had moderate DE (33.3%), 50 (56%) had good DE (66.6%) and remaining 30 (33%) had excellent DE (100%) ().

Our study showed that 18 (20%) MCQs fulfilled all the three criterias (a good/acceptable level of the DIF I [30-70%] with a high DI [≥0.25] and 100% DE.) which should be present for an ideal MCQ.

The mean values of the DI, DIF I and DE are given in .

Discussion

Item analysis is a relatively simple, valuable procedure that provides a method for analysing observation, interpretation of the knowledge achieved by the students and information regarding the quality of test items., In this study, we have performed item analysis of single best response type MCQ as it is seen as an efficient tool of assessment of the student's level of learning in academics., , The efficiency of MCQ assessment is solely based upon the quality of test MCQs.

In this study, Skewness and Kurtosis values (0.59 and 0.69) represent the moderately skewed data with tail on the right side of the distribution and data are lighttailed (lack of outliers) i.e distribution of data is approximately normal. In this study, of total 90 items, the majority, that is, 74 (82%) had a good/acceptable level of DIF I (30-70%) with a mean DIF I of 55.32 ± 7.4 (mean ± SD), whereas seven (8%) were too difficult (DIF I <30%) and nine (10%) were too easy (DIF I >70%). Karelia et al. reported a range of DIF I mean ± SD between 47.17 ± 19.79 and 58.8 ± 19.33, which supports our study findings. A similar study by Rao et al. reported that 85% had an acceptable level of DIF I (30-70%) with a mean value of 50.16 ± 16.15, 5% had easy and 10% items were difficult. Another study by Patel et al. showed 80% of items had an acceptable level of DIF I. A similar study showed that 70% of items had an acceptable range of DIF I, 20% were too easy and 10% were too difficult. Mahjabeen et al. reported that 81% of MCQs were in an acceptable group. Another study of item analysis of MCQs showed that 80%, 7% and 13% of MCQs were acceptable, too easy and too difficult, respectively.

In our study, 72 (80%) items had acceptable to excellent discriminating power (DI > 0.20) and 18 (20%) had poor discriminating power (DI ≤ 0.20). The majority of items 61 (68%) had good to excellent DI (≥0.25) which suggests that these items are good to excellent in discriminating or differentiating the ability of the students with higher scores and those with lower scores. The DI values of the present study are comparable with studies on item analysis by Date et al. and others as similar findings with 78% items having acceptable to excellent discriminating power (DI > 0.20), 65-70% having good to excellent (DI ≥ 0.25) and 24% having poor discriminating power (DI ≤ 0.20), were reported. Item DIs should be interpreted in the context of the type of test which is being analysed. The values of DI tend to be lower for tests measuring a wide range of content areas than for more homogeneous tests. Items with low DIs are often ambiguously worded and should be reexamined. Higher value of the DI indicates higher efficiency of an MCQ item.

The present study shows weak correlation between the DIF I and DI (r = 0.140, p < .0001), indicating that with increasing the DIF I, the ability to discriminate between the high and low scorers decreased and fewer easy questions were used in MCQ tests. The weak correlation between the DIF I and DI also indicates that relationship between the DIF I and DI of 90 MCQ items is dome shaped rather than linear. Several studies also showed poor correlation between the DIF I and DI.,

Our study showed 73% were FDs and 27% were NFDs. Of all items, 2% items had poor DE (DE = 0%), 9% had moderate DE (DE = 33.3%), 56% had good DE (DE = 66.6%) and remaining 33% had excellent DE (DE = 100%). Our study showed that, in 33% MCQs, all three wrong options fully distracted the student's attention. Our findings are in agreement with the study by Date et al who reported that 70% items were FDs, 30% items were NFDs and other studies showed similar findings as 76-83% items were FDs and 17-24% items were NFDs., Another study by Mahjabeen et al showed that 72% items were FDs and only 28% items were NFDs. Framing possible distractors and reducing the NFDs is an important aspect for good quality MCQs. More NFD makes MCQ easy and vice versa with more functioning distractors making it difficult. A non-functional distractor can reduce the discrimination power of a MCQ. More training and effort are required for construction of possible options of MCQ items to improve the validity and reliability of the tests. The comparison of various item analysis indices between various studies is shown in ''.

For an ideal MCQ, the level of the DIF I should have a good/acceptable level of DIF I (30-70%) with high DI (≥0.25) and 100% DE. The present study showed that a total 20% MCQs fulfilled all the three criteria of an ideal MCQ. Other studies reported that 10-15% MCQs satisfied all the three criteria of ideal MCQs. Our study showed mean and standard deviation for DIF I (good), DI (excellent) and DE (moderate) which suggested that the majority of MCQs were in the category of good MCQs. Our results are comparable to a study by Ingale et al. that analysed 30 MCQs.

In this study, the results showed that the overall validity of the test was good with a Cronbach alpha of 0.85, KR 20 = 0.71 and SEM 1.22. Based on the assumption that any test score contains an error, SEM is used to estimate a band or interval within which a person's true score would fall, that is, the score if there were no error of measurement. The smaller the SEM, the narrower is the interval. Narrow intervals are more precise, containing less error, than larger intervals.

Conclusion and recommendations

Item analysis is a valuable procedure which should be regularly performed after the assessment providing information regarding the reliability and validity of an item/test by calculating the DIF I, DI and DE and their interrelationship. An ideal single best response item (MCQ) with four options will be the one which has average difficulty (DIF I 30-70%), high discrimination (DI ≥ 0.25) and maximum DE (100%) with three FDs.

Items analysed in this study were neither too easy nor too difficult which is acceptable. Therefore, items were acceptably difficult and were good at differentiating students with higher and lower abilities. The values of the DIF I and DI indicated that not too easy and too difficult questions were part of the examination.

Our study helped teachers identify good and ideal MCQs which can be part of the question bank for future and those MCQs which needed revision. We recommend that item analysis should be performed for all MCQ-based assessments to determine validity and reliability of the assessment.

Disclosure of competing interest

The authors have none to declare.

References

# data[].attributes.attachment_link.title
Item analysis of multiple choice questions: A quality assurance test for an assessment tool

# data[].attributes.attachment_link.description
The item analysis of multiple choice questions (MCQs) is an essential tool that can provide input on validity and reliability of items. It helps to identify items which can be revised or discarded, thus building a quality MCQ bank.The study focussed on ...

# data[].attributes.headline
Item analysis as tool to validate multiple choice question bank in pharmacology 

# data[].attributes.subject
Item analysis as tool to validate multiple choice question bank in pharmacology 

# data[].attributes.headline
Evaluation of multiple choice questions using item analysis tool

# data[].attributes.subject
Evaluation of multiple choice questions using item analysis tool

# data[].attributes.headline
Item analysis of multiple choice questions: Assessing an assessment tool in medical students

# data[].attributes.subject
Item analysis of multiple choice questions: Assessing an assessment tool in medical students

# data[].attributes.attachment_link.readable_text
ORIGINAL ARTICLE

Year : 2016 | Volume : 2 | Issue : 4 | Page : 201-204

Item analysis of multiple choice questions: Assessing an assessment tool in medical students Department of Pathology, K. S. Hegde Medical Academy, Deralakatte, Mangalore, Karnataka, India

Chandrika Rao, HL Kishan Prasad, K Sajitha,
Harish Permi, Jayaprakash Shetty

Date of Web Publication

2-Sep-2016

Correspondence Address:
Dr. Chandrika Rao
Department of Pathology, K. S. Hegde Medical Academy, Deralakatte, Mangalore - 575 018, Karnataka
India

Source of Support: None, Conflict of Interest: None

Check

DOI: 10.4103/2395-2296.189670

Aim: Assessment is a very important component of the medical course curriculum. Item analysis is the process of collecting, summarizing, and using information from student's responses to assess the quality of multiple-choice questions (MCQs). Difficulty index (P) and discrimination index (D) are the parameters used to evaluate the standard of MCQs. The aim of the study was to assess quality of MCQs. Materials and Methods: The study was conducted in the Department of Pathology. One hundred and twenty, 2 nd year MBBS students took the MCQs test comprising 40 questions. There was no negative marking and evaluation was done out of 40 marks, and 50% score was the passing mark. Postvalidation of the paper was done by item analysis. Each item was analyzed for difficulty index, discrimination index, and distractor effectiveness. The relationship between them for each item was determined by Pearson correlation analysis using SPSS 20.0. Results: Difficulty index of 34 (85%) items was in the acceptable range ( P = 30-70%), 2 (5%) item was too easy ( P >70%), and 4 (10%) items were too difficult ( P <30%). Discrimination index of 24 (60%) items was excellent ( D >0.4), 4 (10%) items were good ( D =0.3-0.39), 6 (15%) items were acceptable ( D =0.2-0.29), and 6 (15%) items were poor ( D < 0-0.19). A total 40 items had 120 distractors. Amongst these, 6 (5%) were nonfunctional distracters, 114 (95%) were functional distracters. The discrimination index exhibited positive correlation with difficulty index ( r = 0.563, P = 0.010, significant at 0.01 level [two-tailed]). The maximum discrimination ( D = 0.5-0.6) was observed in acceptable range ( P = 30-70%). Conclusion: In this study, the majority of items fulfilled the criteria of acceptable difficulty and good discrimination. Moderately easy/difficult had the maximal discriminative ability. Very difficult item displayed poor discrimination, but the very easy item had high discrimination index, indicating a faulty item, or incorrect keys. The results of this study would initiate a change in the way MCQ test items are selected for any examination, and there should be proper assessment strategy as part of the curriculum development.

Keywords: Difficulty index, discrimination index, distractor effectiveness, item analysis, multiple choice questions

How to cite this article:
Rao C, Kishan Prasad H L, Sajitha K, Permi H, Shetty J. Item analysis of multiple choice questions: Assessing an assessment tool in medical students. Int J Educ Psychol Res 2016;2:201-4

How to cite this URL:
Rao C, Kishan Prasad H L, Sajitha K, Permi H, Shetty J. Item analysis of multiple choice questions: Assessing an assessment tool in medical students. Int J Educ Psychol Res [serial online] 2016 [cited 2022 Dec 14];2:201-4. Available from: https://www.ijeprjournal.org/text.asp?2016/2/4/201/189670

Multiple choice questions (MCQs) are frequently used to assess students in different educational streams for objectivity and wide reach of coverage in less time. MCQs are used mostly for comprehensive assessment at the end of academic sessions and provide feedback to the teachers on their educational actions. Designing MCQ is a complex and time-consuming process in a multidisciplinary, integrated curriculum. MCQ needs to be tested for the standard or quality.[1] Item analysis examines the student responses to individual test items (MCQ) to assess the quality of those items and test as a whole. Item analysis assesses the assessment tool for the benefit of both student and teacher.

The aim of the study was to analyze the quality of MCQ's. To investigate the relationship of items having good difficulty and discrimination indices with their distractor efficiency and to find out the correlation between difficulty index (P) and discrimination index (D).

The study was conducted in the Department of Pathology as part of the assessment. Total 120, second year MBBS students took the MCQ's test comprising 40 questions with a single best response. There was no negative marking and time allotted was half an hour. Prevalidation of the paper was done by the head of the department. The evaluation was done out of 40 marks, and 50% score was the passing mark.

Postvalidation of the paper was done by item analysis. The scores of all the students were arranged in order of merit. The upper one-third students were considered high achievers and lower one-third as low achievers. Each item was analyzed for:[2],[3]

Difficulty index or P value using formula P = H + L/N ×100

H = Number of students answering the item correctly in the high achieving group

L = Number of students answering the item correctly in the low achieving group

N = Total number of students in the two groups (including nonresponders)

Discrimination index (D) or d value using formula, d = H - L ×2/N

Where the symbols H, L, and N represent the same values as mentioned above

Distractor effectiveness (DE) or functionality.

Interpretation:

Difficulty index (P) if

P< 30% Difficult

P = 30-70% Acceptable

P > 70% Easy

Discrimination index (D) if

D = Negative. Defective item/wrong key

D = 0-0.19 Poor discrimination

D between 0.2 and 0.29 Acceptable discrimination

D between 0.3 and 0.39 Good discrimination

D > 0.4 Excellent discrimination.

An item contains a stem and four options including one correct (key) and three incorrect (distractor) alternatives. Nonfunctional distractor (NFD) in an item is the option, other than the key selected by <5% of students and functional or effective distractor is the option selected by 5% or more students. On the basis of NFDs in an item, DE ranges from 0% to 100%. If an item contains three or two or one or nil NFDs, then DE would be 0, 33.3%, 66.6%, and 100%, respectively.

Statistical analysis

The data are reported as a percentage and mean plus or minus standard deviation (SD) of n items. The relationship between the difficulty index and discrimination index values for all items was determined using Pearson correlation analysis and using SPSS 20.0 (IBM, Armonk, NY, United States of America). P< 0.05 was considered to indicate statistical significance.

A total of 120 students gave the test consisting of 40 MCQs. As seen in [Table 1], mean difficulty index (P) was 50.16 ± 16.15 while mean discrimination index (D) was 0.34 ± 0.17. The distribution between difficulty indices (range 23.7-75.0) and discrimination indices (range 0-0.66) in all forty MCQ items were analyzed.

Table 1: Comparison of difficulty index. Discrimination index and distractor effectiveness of the MCQ item

Click here to view

A total of 40 items had 120 distractors. Amongst these, 6 (5%) were NFDs, 114 (95%) were functional distractors. Mean distractor efficiency was 89.99 ± 24.42 and distribution range from 0% to 100% [Table 1].

[Figure 1] shows, out of a total 40 items, difficulty indices of 5% MCQ items were easy ( P > 70%), about 10% were difficult ( P < 30%) and the remaining 85% of the items were within an acceptable range (30-70%).

The discrimination indices for 40 items showed 15% of the items with poor discrimination power (0-0.19), and 60% of the items exhibited excellent discrimination (>0.4). The remaining 25% were acceptable and good, out of which 15% of the items had an acceptable range (0.2-0.29) and 10% of the items showed good discrimination (0.3-0.39) [Figure 2].

The discrimination index correlated positively with the difficulty index ( r =0.563, P = 0.010, significant at 0.01 level [two-tailed]). The maximum discrimination ( D = 0.5-0.6) was observed in acceptable range ( P = 30-70%).

The effective measurement of knowledge acquired is an important component of medical education. MCQ form useful assessment tool in measuring factual recall and if carefully constructed can test higher order of thinking skills which is very important for a medical graduate.[4] The method of assessment should be regularly evaluated. Developing an appropriate assessment strategy is a key part in curriculum development. It is important to evaluate MCQ items to see how effective they are in assessing the knowledge of students.[4]

Postexamination analysis of the MCQs helps to assess the quality of individual test items and test as a whole.[1] Poor items can be modified or removed from the store of questions.

Previous studies have proposed the mean of difficulty index as 39.4 ± 21.4%,[3] 52.53 ± 20.59.[4] Karelia et al. showed a range of mean ± SD between 47.17 ± 19.79 and 58.8 ± 19.33 in a study conducted over a period of 5 years.[5] They also showed 61% items in acceptable range ( P = 30-70%), 24% items ( P >70%), and 15% items ( P< 30%). Other study by Patel and Mahajan showed 80% of items in the acceptable range.[6] Our findings corresponded with this study having a mean of difficulty index as 75.0 ± 23.7. The P value of 34 (85%) items was in acceptable range, two items (5%) easy, and 4 (10%) items difficult.

Higher the difficulty index lower is the difficulty of the question. The difficulty index and discrimination index are reciprocally related.[1] Questions with high P value are considered to be good discriminators.[7] The value of discrimination index normally ranges between 0 and 1. Any discrimination index of 0.2 or higher is acceptable, and the test item would be able to differentiate between weak and good students. In this, it shows that 75% had discrimination index of more than 0.2. Out of 75%, 65% showed mean discrimination index of equal to or more than 0.4, indicating that these MCQ item were excellent test items for differentiating between poor and good performers. There were no items with negative discrimination index. Some studies have shown negative discrimination index in 20%.[8] Items with negative discrimination index decrease the validity of the test and should be removed from the collection of questions.

Earlier studies have revealed 29% items with discrimination index >0.4, 46% items with discrimination index between 0.2 0.39 and 21% items with discrimination index <0.19.[7] A positive correlation was noted in difficulty and discrimination indices. The Same observation was reported by Pande et al., 2013[4] and Si-Mui Sim and Rasaiah 2006[9] in their studies. Mitra et al., 2009[10] showed that the discrimination index correlated poorly switch difficulty index ( r = −0.325). The negative correlation signifies with increasing difficulty index values; there was a decrease in the discrimination index indicating that low performers were more likely to get the correct answer. In the present study, moderately easy/difficult (acceptable range) items had the maximal discriminative ability. Very difficult item displayed poor discrimination, but the very easy item had high discrimination index, indicating a faulty item, or incorrect keys.

The distraction effect of items in our study was 89.99%. The number of an NFDs also affect the discrimination power of an item. It is seen that reducing the number of distractors from four to three decreases the difficulty index while increasing the discrimination index and reliability. Hingorjo [11] observed that items having one NFD had excellent discrimination ability. ( D = 0.427) As compared to items with all four functioning distractors ( D = 0.351). This compares well with other studies favoring better discrimination by three distractors as compared to four.[11]

It was also observed that item having good difficulty index ( P = 30-70) and good/excellent D ( D > 0.24), considered to be ideal question, had DE of 85.15% which is close to items having one NFD.

Item analysis is a simple yet valuable procedure performed after the examination providing information regarding the reliability and validity of an item/test by calculating difficulty index, discrimination index, distractor efficiency, and their interrelationship. An ideal item (MCQ) will be the one which has average difficulty index between 31% and 60%, high discrimination ( D > 0.25), and maximum distractor efficiency (100%) with three functional distractors. Items analyzed in the study were neither too easy nor too difficult (mean difficulty index = 50.16%), and overall discrimination index was 0.34, which is acceptable. In this study, the majority of items fulfilled the criteria of acceptable difficulty and good discrimination. Easy items with poor discrimination index will be reviewed and reconstructed. The results of this study should initiate a change in the way MCQ test items are selected for any examination, and there should be proper assessment strategy as part of the curriculum development. Much more of these kinds of analysis should be carried out after each examination to identify the areas of potential weakness in the one best answer type of MCQ tests to improve the standard of assessment.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.

[Figure 1], [Figure 2]

[Table 1]

# data[].attributes.attachment_link.keywords[]
international journal of educational and psychological researches

# data[].attributes.attachment_link.title
Item analysis of multiple choice questions: Assessing an assessment tool in medical students Rao C, Kishan Prasad H L, Sajitha K, Permi H, Shetty J - Int J Educ Psychol Res

# data[].attributes.attachment_link.description
ORIGINAL ARTICLE Year : 2016 | Volume : 2 | Issue : 4 | Page : 201-204 Item analysis of multiple choice questions: Assessing an assessment tool in medical students Department of Pathology, K. S. Hegde Medical Academy, Deralakatte, Mangalore, Karnataka, India Chandrika Rao, HL Kishan Prasad, K Sajitha, Harish Permi, Jayaprakash Shetty Date of Web Publication 2-Sep-2016 Correspondence Address:Dr. Chandrika RaoDepartment of Pathology, K.

# data[].attributes.attachment_link.readable_text
Item analysis is a process which examines student responses to individual test items (questions) in order to assess the quality of those items and of the test as a whole. Item analysis is especially valuable in improving items which will be used again in later tests, but it can also be used to eliminate ambiguous or misleading items in a single test administration. In addition, item analysis is valuable for increasing instructors' skills in test construction, and identifying specific areas of course content which need greater emphasis or clarity. Separate item analyses can be requested for each raw score ¹ created during a given ScorePak® run.

Sample Item Analysis (30K PDF)

A basic assumption made by ScorePak® is that the test under analysis is composed of items measuring a single subject area or underlying ability. The quality of the test as a whole is assessed by estimating its "internal consistency." The quality of individual items is assessed by comparing students' item responses to their total test scores.

Following is a description of the various statistics provided on a ScorePak® item analysis report. This report has two parts. The first part assesses the items which made up the exam. The second part shows statistics summarizing the performance of the test as a whole.

Item Statistics

Item statistics are used to assess the performance of individual test items on the assumption that the overall quality of a test derives from the quality of its items. The ScorePak® item analysis report provides the following item information:

Item Number

This is the question number taken from the student answer sheet, and the ScorePak® Key Sheet. Up to 150 items can be scored on the Standard Answer Sheet.

Mean and Standard Deviation

The mean is the "average" student response to an item. It is computed by adding up the number of points earned by all students on the item, and dividing that total by the number of students.

The standard deviation, or S.D., is a measure of the dispersion of student scores on that item. That is, it indicates how "spread out" the responses were. The item standard deviation is most meaningful when comparing items which have more than one correct alternative and when scale scoring is used. For this reason it is not typically used to evaluate classroom tests.

Item Difficulty

For items with one correct alternative worth a single point, the item difficulty is simply the percentage of students who answer an item correctly. In this case, it is also equal to the item mean. The item difficulty index ranges from 0 to 100; the higher the value, the easier the question. When an alternative is worth other than a single point, or when there is more than one correct alternative per question, the item difficulty is the average score on that item divided by the highest number of points for any one alternative. Item difficulty is relevant for determining whether students have learned the concept being tested. It also plays an important role in the ability of an item to discriminate between students who know the tested material and those who do not. The item will have low discrimination if it is so difficult that almost everyone gets it wrong or guesses, or so easy that almost everyone gets it right.

To maximize item discrimination, desirable difficulty levels are slightly higher than midway between chance and perfect scores for the item. (The chance score for five-option questions, for example, is 20 because one-fifth of the students responding to the question could be expected to choose the correct option by guessing.) Ideal difficulty levels for multiple-choice items in terms of discrimination potential are:
(From Lord, F.M. "The Relationship of the Reliability of Multiple-Choice Test to the Distribution of Item Difficulties," Psychometrika, 1952, 18, 181-194.)
ScorePak® arbitrarily classifies item difficulty as "easy" if the index is 85% or above; "moderate" if it is between 51 and 84%; and "hard" if it is 50% or below.

Item Discrimination

Item discrimination refers to the ability of an item to differentiate among students on the basis of how well they know the material being tested. Various hand calculation procedures have traditionally been used to compare item responses to total test scores using high and low scoring groups of students. Computerized analyses provide more accurate assessment of the discrimination power of items because they take into account responses of all students rather than just high and low scoring groups.

The item discrimination index provided by ScorePak® is a Pearson Product Moment correlation ² between student responses to a particular item and total scores on all other items on the test. This index is the equivalent of a point-biserial coefficient in this application. It provides an estimate of the degree to which an individual item is measuring the same thing as the rest of the items.

Because the discrimination index reflects the degree to which an item and the test as a whole are measuring a unitary ability or attribute, values of the coefficient will tend to be lower for tests measuring a wide range of content areas than for more homogeneous tests. Item discrimination indices must always be interpreted in the context of the type of test which is being analyzed. Items with low discrimination indices are often ambiguously worded and should be examined. Items with negative indices should be examined to determine why a negative value was obtained. For example, a negative value may indicate that the item was mis-keyed, so that students who knew the material tended to choose an unkeyed, but correct, response option.

Tests with high internal consistency consist of items with mostly positive relationships with total test score. In practice, values of the discrimination index will seldom exceed .50 because of the differing shapes of item and total score distributions. ScorePak® classifies item discrimination as "good" if the index is above .30; "fair" if it is between .10 and.30; and "poor" if it is below .10.

Alternate Weight

This column shows the number of points given for each response alternative. For most tests, there will be one correct answer which will be given one point, but ScorePak® allows multiple correct alternatives, each of which may be assigned a different weight.

Means

The mean total test score (minus that item) is shown for students who selected each of the possible response alternatives. This information should be looked at in conjunction with the discrimination index; higher total test scores should be obtained by students choosing the correct, or most highly weighted alternative. Incorrect alternatives with relatively high means should be examined to determine why "better" students chose that particular alternative.

Frequencies and Distribution

The number and percentage of students who choose each alternative are reported. The bar graph on the right shows the percentage choosing each response; each "#" represents approximately 2.5%. Frequently chosen wrong alternatives may indicate common misconceptions among the students.

Difficulty and Discrimination Distributions

At the end of the Item Analysis report, test items are listed according their degrees of difficulty (easy, medium, hard) and discrimination (good, fair, poor). These distributions provide a quick overview of the test, and can be used to identify items which are not performing well and which can perhaps be improved or discarded.

Test Statistics

Two statistics are provided to evaluate the performance of the test as a whole.

Reliability Coefficient

The reliability of a test refers to the extent to which the test is likely to produce consistent scores. The particular reliability coefficient computed by ScorePak® reflects three characteristics of the test:

Intercorrelations among the items - the greater the relative number of positive relationships, and the stronger those relationships are, the greater the reliability. Item discrimination indices and the test's reliability coefficient are related in this regard.

Test length - a test with more items will have a higher reliability, all other things being equal.

Test content - generally, the more diverse the subject matter tested and the testing techniques used, the lower the reliability.

Reliability coefficients theoretically range in value from zero (no reliability) to 1.00 (perfect reliability). In practice, their approximate range is from .50 to .90 for about 95% of the classroom tests scored by ScorePak®. High reliability means that the questions of a test tended to "pull together." Students who answered a given question correctly were more likely to answer other questions correctly. If a parallel test were developed by using similar items, the relative scores of students would show little change. Low reliability means that the questions tended to be unrelated to each other in terms of who answered them correctly. The resulting test scores reflect peculiarities of the items or the testing situation more than students' knowledge of the subject matter.

As with many statistics, it is dangerous to interpret the magnitude of a reliability coefficient out of context. High reliability should be demanded in situations in which a single test score is used to make major decisions, such as professional licensure examinations. Because classroom examinations are typically combined with other scores to determine grades, the standards for a single test need not be as stringent. The following general guidelines can be used to interpret reliability coefficients for classroom exams:

Reliability

Interpretation

.90 and above

Excellent reliability; at the level of the best standardized tests

.80 - .90

Very good for a classroom test

.70 - .80

Good for a classroom test; in the range of most. There are probably a few items which could be improved.

.60 - .70

Somewhat low. This test needs to be supplemented by other measures (e.g., more tests) to determine grades. There are probably some items which could be improved.

.50 - .60

Suggests need for revision of test, unless it is quite short (ten or fewer items). The test definitely needs to be supplemented by other measures (e.g., more tests) for grading.

.50 or below

Questionable reliability. This test should not contribute heavily to the course grade, and it needs revision.

The measure of reliability used by ScorePak® is Cronbach's Alpha. This is the general form of the more commonly reported KR-20 and can be applied to tests composed of items with different numbers of points given for different response alternatives. When coefficient alpha is applied to tests in which each item has only one correct answer and all correct answers are worth the same number of points, the resulting coefficient is identical to KR-20.
(Further discussion of test reliability can be found in J. C. Nunnally, Psychometric Theory. New York: McGraw-Hill, 1967, pp. 172-235, see especially formulas 6-26, p. 196.)
Standard Error of Measurement

The standard error of measurement is directly related to the reliability of the test. It is an index of the amount of variability in an individual student's performance due to random measurement error. If it were possible to administer an infinite number of parallel tests, a student's score would be expected to change from one administration to the next due to a number of factors. For each student, the scores would form a "normal" (bell-shaped) distribution. The mean of the distribution is assumed to be the student's "true score," and reflects what he or she "really" knows about the subject. The standard deviation of the distribution is called the standard error of measurement and reflects the amount of change in the student's score which could be expected from one test administration to another.

Whereas the reliability of a test always varies between 0.00 and 1.00, the standard error of measurement is expressed in the same scale as the test scores. For example, multiplying all test scores by a constant will multiply the standard error of measurement by that same constant, but will leave the reliability coefficient unchanged.

A general rule of thumb to predict the amount of change which can be expected in individual test scores is to multiply the standard error of measurement by 1.5. Only rarely would one expect a student's score to increase or decrease by more than that amount between two such similar tests. The smaller the standard error of measurement, the more accurate the measurement provided by the test.
(Further discussion of the standard error of measurement can be found in J. C. Nunnally, Psychometric Theory. New York: McGraw-Hill, 1967, pp.172-235, see especially formulas 6-34, p. 201.)
A Caution in Interpreting Item Analysis Results

Each of the various item statistics provided by ScorePak® provides information which can be used to improve individual test items and to increase the quality of the test as a whole. Such statistics must always be interpreted in the context of the type of test given and the individuals being tested. W. A. Mehrens and I. J. Lehmann provide the following set of cautions in using item analysis results (Measurement and Evaluation in Education and Psychology. New York: Holt, Rinehart and Winston, 1973, 333-334):

Item analysis data are not synonymous with item validity. An external criterion is required to accurately judge the validity of test items. By using the internal criterion of total test score, item analyses reflect internal consistency of items rather than validity.

The discrimination index is not always a measure of item quality. There is a variety of reasons an item may have low discriminating power:(a) extremely difficult or easy items will have low ability to discriminate but such items are often needed to adequately sample course content and objectives;(b) an item may show low discrimination if the test measures many different content areas and cognitive skills. For example, if the majority of the test measures "knowledge of facts," then an item assessing "ability to apply principles" may have a low correlation with total test score, yet both types of items are needed to measure attainment of course objectives.

Item analysis data are tentative. Such data are influenced by the type and number of students being tested, instructional procedures employed, and chance errors. If repeated use of items is possible, statistics should be recorded for each administration of each item.

¹ Raw scores are those scores which are computed by scoring answer sheets against a ScorePak® Key Sheet. Raw score names are EXAM1 through EXAM9, QUIZ1 through QUIZ9, MIDTRM1 through MIDTRM3, and FINAL. ScorePak® cannot analyze scores taken from the bonus section of student answer sheets or computed from other scores, because such scores are not derived from individual items which can be accessed by ScorePak®. Furthermore, separate analyses must be requested for different versions of the same exam. Return to the text. (anchor near note 1 in text)

² A correlation is a statistic which indexes the degree of linear relationship between two variables. If the value of one variable is related to the value of another, they are said to be "correlated." In positive relationships, the value of one variable tends to be high when the value of the other is high, and low when the other is low. In negative relationships, the value of one variable tends to be high when the other is low, and vice versa. The possible values of correlation coefficients range from -1.00 to 1.00. The strength of the relationship is shown by the absolute value of the coefficient (that is, how large the number is whether it is positive or negative). The sign indicates the direction of the relationship (whether positive or negative). Return to the text.

# data[].attributes.attachment_link.description
Item analysis is a process which examines student responses to individual test items (questions) in order to assess the quality of those items and of the test as a whole. Item analysis is especially valuable in improving items which will be used again in later tests, but it can also be used to eliminate ambiguous or misleading items in a single test administration.

# data[].attributes.headline
The relation of the reliability of multiple-choice tests to the distribution of item difficulties

# data[].attributes.subject
The relation of the reliability of multiple-choice tests to the distribution of item difficulties

# data[].attributes.attachment_link.readable_text
Under certain assumptions an expression, in terms of item difficulties and intercorrelations, is derived for the curvilinear correlation of test score on the "ability underlying the test," this ability being defined as the common factor of the item tetrachoric intercorrelations corrected for guessing. It is shown that this curvilinear correlation is equal to the square root of the test reliability. Numerical values for these curvilinear correlations are presented for a number of hypothetical tests, defined in terms of their item parameters. These numerical results indicate that the reliability and the curvilinear correlation will be maximized by (1) minimizing the variability of item difficulty and (2) making the level of item difficulty somewhat easier than the halfway point between a chance percentage of correct answers and 100 per cent correct answers.

# data[].attributes.attachment_link.title
The relation of the reliability of multiple-choice tests to the distribution of item difficulties - Psychometrika

# data[].attributes.attachment_link.description
Under certain assumptions an expression, in terms of item difficulties and intercorrelations, is derived for the curvilinear correlation of test score on the "ability underlying the test," this ability being defined as the common factor of the item tetrachoric intercorrelations corrected for guessing.

# data[].attributes.headline
Analysis of One-Best MCQs: the Difficulty Index, DiscriminationIndex and Distractor Efficiency

# data[].attributes.subject
Analysis of One-Best MCQs: the Difficulty Index, DiscriminationIndex and Distractor Efficiency

# data[].attributes.headline
Design, format, validity and reliability of multiple choice questions for use in nursing research and education

# data[].attributes.subject
Design, format, validity and reliability of multiple choice questions for use in nursing research and education

# data[].attributes.attachment_link.title
Design, format, validity and reliability of multiple choice questions for use in nursing research and education

# data[].attributes.attachment_link.description
Multiple choice questions are used extensively in nursing research and education and play a fundamental role in the design of research studies or educ…

# data[].attributes.headline
Developing, Analyzing, and Using Distractors for Multiple-Choice Tests in Education: A Comprehensive Review

# data[].attributes.subject
Developing, Analyzing, and Using Distractors for Multiple-Choice Tests in Education: A Comprehensive Review

# data[].attributes.headline
Analyse de la validité du contenu et de la qualité formelle desquestions isolées des épreuves classantes nationales informatisées(ECNi) de l’année universitaire 2015–2016

# data[].attributes.subject
Analyse de la validité du contenu et de la qualité formelle desquestions isolées des épreuves classantes nationales informatisées(ECNi) de l’année universitaire 2015–2016

# data[].attributes.headline
Content and Concurrent Validity of the Motivation for Change Questionnaire

# data[].attributes.subject
Content and Concurrent Validity of the Motivation for Change Questionnaire

# data[].attributes.attachment_link.readable_text
Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

References

1. Ekberg K. An epidemiological approach to disorders in the neck and shoulders. PhD thesis, Department of Occupational and Environmental Medicine, Faculty of Health Science, Linköping University, Linköping, Sweden, 1994.

2. Meerding WJ, Bonneux L, Polder JJ, Koopmanschap MA, van der Maas PJ. Demographic and epistemological determinants of health costs in Netherlands: cost of illness study. Br Med J 1998;317:111-5.

3. Gard G, Rivano-Fisher M. Motivation for change. Development and reliability test of a new instrument. World congress of physical therapy, Barcelona, 12-15 may 2003.

4. Marnetoft SU. Vocational rehabilitation of unemployed sick-listed people in a Swedish rural area. PhD thesis, Department of Public Health, Stockholm, Sweden, 2000.

5. Grahn B, Stigmar K, Ekdahl C. Motivation for change and personal resources in patients with prolonged musculoskeletal disorders. J Bodywork Movement Ther 2001;5:160-72.

Article Google Scholar

6. Swedish Social Insurance Agency 2006: 6. The swedish social insurance agency and vocational rehabilitation-regulations in practice. Stockholm, Sweden, 2006 (In Swedish).

Download references

# data[].attributes.attachment_link.title
Content and Concurrent Validity of the Motivation for Change Questionnaire - Journal of Occupational Rehabilitation

# data[].attributes.attachment_link.description
Introduction Musculoskeletal disorders (MSD) are nowadays seen within a biopsychosocial framework, including salutogenic factors, motivation factors, and coping ability. Such a framework recognizes the importance of motivational factors in health promotion and in rehabilitation. The Motivation for Change Questionnaire (MCQ) has been developed to measure the strength of individuals' motivation for change in life, MCQ part 1, and work situation, MCQ part 2.