Development and pilot testing of an operative competency assessment tool for pediatric direct laryngoscopy and rigid bronchoscopy

Authors

  • Stacey L. Ishman MD, MPH,

    Corresponding author
    1. Department of Otolaryngology–Head and Neck Surgery, Johns Hopkins School of Medicine, Baltimore, Maryland, U.S.A.
    Current affiliation:
    1. Triological Society Southern Section, Orlando FL, February 2010
    • Department of Otolaryngology–Head and Neck Surgery, Johns Hopkins School of Medicine, 601 N. Caroline Street, Room 6231, Baltimore, MD 21287
    Search for more papers by this author
  • David J. Brown MD,

    1. Department of Otolaryngology–Head and Neck Surgery, Johns Hopkins School of Medicine, Baltimore, Maryland, U.S.A.
    Search for more papers by this author
  • Emily F. Boss MD,

    1. Department of Otolaryngology–Head and Neck Surgery, Johns Hopkins School of Medicine, Baltimore, Maryland, U.S.A.
    Search for more papers by this author
  • Margaret L. Skinner MD,

    1. Department of Otolaryngology–Head and Neck Surgery, Johns Hopkins School of Medicine, Baltimore, Maryland, U.S.A.
    Search for more papers by this author
  • David E. Tunkel MD,

    1. Department of Otolaryngology–Head and Neck Surgery, Johns Hopkins School of Medicine, Baltimore, Maryland, U.S.A.
    Search for more papers by this author
  • Rose Stavinoha MD,

    1. Department of Otolaryngology–Head and Neck Surgery, Johns Hopkins School of Medicine, Baltimore, Maryland, U.S.A.
    Search for more papers by this author
  • Sandra Y. Lin MD

    1. Department of Otolaryngology–Head and Neck Surgery, Johns Hopkins School of Medicine, Baltimore, Maryland, U.S.A.
    Search for more papers by this author

  • The authors declare no conflicts of interests and have no financial disclosures.

Abstract

Objectives:

To develop and pilot test an objective assessment of technical skills instrument for evaluation of residents' surgical performance of pediatric direct laryngoscopy and rigid bronchoscopy, with emphasis on feasibility, validity, and interrater agreement.

Study Design:

Prospective, unblinded educational quality-improvement project combining paired and unpaired observations.

Methods:

Objective Structured Assessment of Technical Skills (OSATS) evaluation instruments were developed from detailed faculty input. Two instruments were created: a surgical checklist and a global assessment of surgical performance. The instruments were tested in a simulation bronchoscopy course and in the operative suite. Paired observations were used to determine interrater agreement, whereas multiple evaluations were used to calculate construct validity and internal consistency.

Results:

Forty-four assessments were completed for 19 residents. Seven faculty members evaluated residents as they performed laryngoscopy and bronchoscopy in an animal simulation laboratory and in the operating room. The evaluation tool was found to be feasible. The interrater agreement for both instruments was significant at 80.4% to 84.6% (P ≤.0001). Construct validity was confirmed with increasing mean global and task specific scores by postgraduate year (P <.0001). Internal consistency, measured with Cronbach alpha, was high at 0.968.

Conclusions:

This pilot study suggests that a reliable and valid instrument for objective evaluation of surgical competency can be developed for pediatric direct laryngoscopy and rigid bronchoscopy. This instrument can be used for formative and summative feedback of operative performance. In addition, it was easy to use and valid in a limited evaluation; however, larger studies are required to validate its utility. Laryngoscope, 2010

Ancillary