Determining the degree of randomness in multiple-choice question distractors

Abstract

Multiple-choice questions are a staple of educational assessment due to their efficiency, but their diagnostic ability is inherently limited. Traditional psychometric techniques infer misconceptions by fitting models to predetermined data patterns, making them ineffective for analysing small, novel, or non-standardised samples where such prior trends are absent. To address this, we introduce a new distractor analysis method based on Information Theory, which treats the randomness in incorrect answers not as noise but as a measurable signal in student answer patterns. By examining the entropy of distractor choices alongside overall accuracy, our approach offers a sample-sized-independent framework for classifying performance. Cross-referencing low-scoring item-level response data against independently documented misconceptions on a standard mechanics inventory shows that, for items identified as most concentrated (lowest degree of randomness), a majority of incorrect responses select the exact response coded to that item's named, interview-validated misconception. Furthermore, items independently flagged elsewhere as diagnostically unreliable are exactly the items the measure identifies as having the highest degree of randomness. The pedagogically actionable cases are those with low degree of randomness in a class's incorrect answers, signalling a shared, re-teachable misconception, whereas a high degree of randomness chiefly serves as the null case against which that concentration is judged. Our analysis is intended as a fast, single-administration triage step: a practical, theory-driven complement to established distractor-analysis methods that helps instructors flag which items warrant closer, more resource-intensive investigation.