Assessment of clinical reasoning in the context of uncertainty: the effect of variability within the reference panel
Article first published online: 22 AUG 2006
Volume 40, Issue 9, pages 848–854, September 2006
How to Cite
Charlin, B., Gagnon, R., Pelletier, J., Coletti, M., Abi-Rizk, G., Nasr, C., Sauvé, É. and Van Der Vleuten, C. (2006), Assessment of clinical reasoning in the context of uncertainty: the effect of variability within the reference panel. Medical Education, 40: 848–854. doi: 10.1111/j.1365-2929.2006.02541.x
- Issue published online: 22 AUG 2006
- Article first published online: 22 AUG 2006
- Received 14 April 2005; editorial comments to authors 23 August 2005; accepted for publication 1 November 2005
- family practice/*education;
- clinical competence/*standards;
- reference values;
- *decision making;
- psychological tests/*standards;
- analysis of variance
Context The Script Concordance Test (SCT) assesses reasoning in the context of uncertainty. Because there is no single correct answer, scoring is based on the comparison of answers provided by examinees with those provided by members of a reference panel made up of experienced practitioners. The study aimed to assess the discriminatory power of the SCT based on the variability of the reference panel's answers.
Method Items from a bank covering different family medicine domains were classified into 3 groups according to the degree of variability of answers provided by a pool of experienced doctors. A variability index (mean squared error) was used to select items in the low, moderate and high variability categories. A 102-item test (Cronbach's α 0.70), made up of 3 subtests of each category, was administered to 3 contrasting groups in family medicine: 157 clerkship students, 30 residents and 30 practising doctors. anova and effect size (ES) were used to quantify and test the discrimination power of the 3 subtests.
Results The high variability subtest showed high effect size for discrimination between extreme groups (ES = 1.5; F = 16.3, P < 0.001), whereas the moderate variability subtest showed less effect size (ES = 0.56; F = 57, P = 0.041). The low variability subtest did not discriminate significantly (ES = 0.31; F = 2.9, P = 0.06).
Conclusions Variability of answers within the reference panel is a key component of the discriminatory power of the SCT. In accordance with theory, the presence of variability ensures discrimination between levels of clinical experience. These results imply important considerations for the construction of efficient SCTs.