Reliability: on the reproducibility of assessment data
Article first published online: 24 AUG 2004
Volume 38, Issue 9, pages 1006–1012, September 2004
How to Cite
Downing, S. M. (2004), Reliability: on the reproducibility of assessment data. Medical Education, 38: 1006–1012. doi: 10.1111/j.1365-2929.2004.01932.x
- Issue published online: 24 AUG 2004
- Article first published online: 24 AUG 2004
- Received 8 January 2004; editorial comments to author 5 February 2004; accepted for publication 8 March 2004
- educational measurement/*standards;
- reproducibility of results
Context All assessment data, like other scientific experimental data, must be reproducible in order to be meaningfully interpreted.
Purpose The purpose of this paper is to discuss applications of reliability to the most common assessment methods in medical education. Typical methods of estimating reliability are discussed intuitively and non-mathematically.
Summary Reliability refers to the consistency of assessment outcomes. The exact type of consistency of greatest interest depends on the type of assessment, its purpose and the consequential use of the data. Written tests of cognitive achievement look to internal test consistency, using estimation methods derived from the test-retest design. Rater-based assessment data, such as ratings of clinical performance on the wards, require interrater consistency or agreement. Objective structured clinical examinations, simulated patient examinations and other performance-type assessments generally require generalisability theory analysis to account for various sources of measurement error in complex designs and to estimate the consistency of the generalisations to a universe or domain of skills.
Conclusions Reliability is a major source of validity evidence for assessments. Low reliability indicates that large variations in scores can be expected upon retesting. Inconsistent assessment scores are difficult or impossible to interpret meaningfully and thus reduce validity evidence. Reliability coefficients allow the quantification and estimation of the random errors of measurement in assessments, such that overall assessment can be improved.