Objective structured assessment of technical skills and checklist scales reliability compared for high stakes assessments

Authors

  • Anthony G. Gallagher,

    Corresponding author
    1. School of Medicine, University College Cork, Cork, Ireland
    2. National Surgical Training Centre, Royal College of Surgeons in Ireland, Dublin, Ireland
    • Correspondence

      Professor Anthony G. Gallagher, School of Medicine, Brookfield Health Sciences Complex, College Road, University College Cork, Cork IE, Ireland. Email: anthonyg.ggallagher@btinternet.com

    Search for more papers by this author
  • Gerald C. O'Sullivan,

    1. Cork Cancer Research Centre, Biosciences Institute, University College Cork, Cork, Ireland
    2. National Surgical Training Centre, Royal College of Surgeons in Ireland, Dublin, Ireland
    Search for more papers by this author
  • Gerald Leonard,

    1. Otolaryngology – Head and Neck Surgery, University of Connecticut Health Center, Farmington, Connecticut, USA
    Search for more papers by this author
  • Brendan P. Bunting,

    1. School of Psychology, University of Ulster, Derry, Northern Ireland, UK
    Search for more papers by this author
  • Kieran J. McGlade

    1. School of Medicine, Dentistry and Biomedical Sciences, Queen's University, Belfast, Northern Ireland, UK
    Search for more papers by this author

  • A. G. Gallagher DSc; G. C. O'Sullivan MCh; G. Leonard MD; B. P. Bunting PhD; K. J. McGlade MD.

Abstract

Background

The establishment of assessment reliability at the level of the individual trainee is an important attribute of assessment methodologies, particularly for doctors who have been failed. This issue is of particular importance for the process of competence assessment in the USA, UK, Australia and New Zealand.

Methods

We use data from 19 applicants for higher surgical training in 2008 at the Royal College of Surgeons in Ireland to compare: (i) the objective structured assessment of technical skills (OSATS) method; and (ii) a procedure-specific checklist to assess surgical technical skills in the excision of a sebaceous cyst task by two experienced senior surgeons.

Results

The overall interrater reliability (IRR) of the OSATS assessment as determined by a correlation coefficient was 0.507 (P < 0.03) and 0.67 with coefficient alpha, considerably below the accepted 0.8 level of IRR. The checklist's overall IRR was 0.89. Individually, only five (26%) of the OSATS assessments reached the 0.8 level of IRR in contrast to 18 (95%) of the checklist assessments.

Discussion

We propose binary procedure-based assessment checklists as more reliable assessment instruments with more robust reproducibility.

Ancillary