Validity threats: overcoming interference with proposed interpretations of assessment data
Article first published online: 30 JAN 2004
Volume 38, Issue 3, pages 327–333, March 2004
How to Cite
Downing, S. M. and Haladyna, T. M. (2004), Validity threats: overcoming interference with proposed interpretations of assessment data. Medical Education, 38: 327–333. doi: 10.1046/j.1365-2923.2004.01777.x
- Issue published online: 30 JAN 2004
- Article first published online: 30 JAN 2004
- Received 6 June 2003; editorial comments to authors 21 July 2003; accepted for publication 20 August 2003
- education, medical, undergraduate/*standards;
- educational measurement/*standards;
- clinical competence/standards;
- reproducibility of results
Context Factors that interfere with the ability to interpret assessment scores or ratings in the proposed manner threaten validity. To be interpreted in a meaningful manner, all assessments in medical education require sound, scientific evidence of validity.
Purpose The purpose of this essay is to discuss 2 major threats to validity: construct under-representation (CU) and construct-irrelevant variance (CIV). Examples of each type of threat for written, performance and clinical performance examinations are provided.
Discussion The CU threat to validity refers to undersampling the content domain. Using too few items, cases or clinical performance observations to adequately generalise to the domain represents CU. Variables that systematically (rather than randomly) interfere with the ability to meaningfully interpret scores or ratings represent CIV. Issues such as flawed test items written at inappropriate reading levels or statistically biased questions represent CIV in written tests. For performance examinations, such as standardised patient examinations, flawed cases or cases that are too difficult for student ability contribute CIV to the assessment. For clinical performance data, systematic rater error, such as halo or central tendency error, represents CIV. The term face validity is rejected as representative of any type of legitimate validity evidence, although the fact that the appearance of the assessment may be an important characteristic other than validity is acknowledged.
Conclusions There are multiple threats to validity in all types of assessment in medical education. Methods to eliminate or control validity threats are suggested.