Script concordance testing: a review of published validity evidence

Authors

  • Stuart Lubarsky,

    1. Department of Neurology and Neurosurgery, McGill University, Montreal, Quebec, Canada
    2. Centre for Medical Education, McGill University, Montreal, Quebec, Canada
    Search for more papers by this author
  • Bernard Charlin,

    1. Centre for Applied Teaching in Health Sciences (Centre de Pédagogie Appliquée aux Sciences de la Santé [CPASS]), University of Montreal, Montreal, Quebec, Canada
    Search for more papers by this author
  • David A Cook,

    1. Department of Internal Medicine, Mayo Clinic College of Medicine, Rochester, Minnesota, USA
    Search for more papers by this author
  • Colin Chalk,

    1. Department of Neurology and Neurosurgery, McGill University, Montreal, Quebec, Canada
    2. Centre for Medical Education, McGill University, Montreal, Quebec, Canada
    Search for more papers by this author
  • Cees P M van der Vleuten

    1. Department of Educational Research and Development, Maastricht University, Maastricht, the Netherlands
    Search for more papers by this author

Stuart Lubarsky, Department of Neurology and Neurosurgery and Centre for Medical Education, McGill University, Montreal, Quebec H3G 1A4, Canada. Tel: 00 1 514 934 8060; Fax: 00 1 514 934 8265; E-mail: stuart.lubarsky@mcgill.ca

Abstract

Medical Education 2011: 45: 329–338

Context  Script concordance test (SCT) scores are intended to reflect respondents’ competence in interpreting clinical data under conditions of uncertainty. The validity of inferences based on SCT scores has not been rigorously established.

Objectives  This study was conducted in order to develop a structured validity argument for the interpretation of test scores derived through use of the script concordance method.

Methods  We searched the PubMed, EMBASE and PsycINFO databases for articles pertaining to script concordance testing. We then reviewed these articles to evaluate the construct validity of the script concordance method, following an established approach for analysing validity data from five categories: content; response process; internal structure; relations to other variables, and consequences.

Results  Content evidence derives from clear guidelines for the creation of authentic, ill-defined scenarios. High internal consistency reliability supports the internal structure of SCT scores. As might be expected, SCT scores correlate poorly with assessments of pure factual knowledge, in which correlations for more advanced learners are lower. The validity of SCT scores is weakly supported by evidence pertaining to examinee response processes and educational consequences.

Conclusions  Published research generally supports the use of SCT to assess the interpretation of clinical data under conditions of uncertainty, although specifics of the validity argument vary and require verification in different contexts and for particular SCTs. Our review identifies potential areas of further validity inquiry in all five categories of evidence. In particular, future SCT research might explore the impact of the script concordance method on teaching and learning, and examine how SCTs integrate with other assessment methods within comprehensive assessment programmes.

Ancillary