• 1
    Hubbard J, Levit E. The National Board of Medical Examiners: The First Seventy Years. Philadelphia, PA: National Board of Medical Examiners, 1985.
  • 2
    Hubbard J, Levit E, Schumacher CF, Schnabel TG Jr. An objective evaluation of clinical competence. New techniques used by the National Board of Medical Examiners. N Engl J Med. 1965; 272:13218.
  • 3
    Epstein RM. Assessment in medical education. N Engl J Med. 2007; 356:38796.
  • 4
    Grenvik A, Schaefer JJ III, DeVita MA, Rogers P. New aspects on critical care medicine training. Curr Opin Crit Care. 2004; 10:2337.
  • 5
    Berkenstadt H, Erez D, Munz Y, Simon D, Ziv A. Training and assessment of trauma management: the role of simulation-based medical education. Anesthesiol Clin. 2007; 25:6574.
  • 6
    Bradley P. The history of simulation in medical education and possible future directions. Med Educ. 2006; 40:25462.
  • 7
    Srinivasan M, Hwang JC, West D, Yellowlees PM. Assessment of clinical skills using simulator technologies. Acad Psychiatry. 2006; 30:50515.
  • 8
    Cooper JB, Taqueti VR. A brief history of the development of mannequin simulators for clinical education and training. Qual Saf Health Care. 2004; 13(Suppl 1):i118.
  • 9
    McFetrich J. A structured literature review on the use of high fidelity patient simulators for teaching in emergency medicine. Emerg Med J. 2006; 23:50911.
  • 10
    Maran NJ, Glavin RJ. Low- to high-fidelity simulation--a continuum of medical education? Med Educ. 2003; 37(Suppl 1):228.
  • 11
    Boulet JR, Ben David MF, Ziv A, et al. Using standardized patients to assess the interpersonal skills of physicians. Acad Med. 1998; 73(10 Suppl):S946.
  • 12
    Cantrell MJ, Deloney LA. Integration of standardized patients into simulation. Anesthesiol Clin. 2007; 25:37783.
  • 13
    Nestel D, Kneebone R, Black S. Simulated patients and the development of procedural and operative skills. Med Teach. 2006; 28:3901.
  • 14
    Berkenstadt H, Ziv A, Gafni N, Sidi A. Incorporating simulation-based objective structured clinical examination into the Israeli National Board Examination in Anesthesiology. Anesth Analg. 2006; 102:8538.
  • 15
    Ziv A, Rubin O, Sidi A, Berkenstadt H. Credentialing and certifying with simulation. Anesthesiol Clin. 2007; 25:2619.
  • 16
    Medical News Today. Medical Simulation Corporation and the American Board of Internal Medicine Introduce New Simulation for Physician Assessment and Evaluation. Med News Today, 2-11-2008. Available at: Accessed Jul 16, 2008.
  • 17
    Dillon GF, Boulet JR, Hawkins RE, Swanson DB. Simulations in the United States Medical Licensing Examination (USMLE). Qual Saf Health Care. 2004; 13(Suppl 1):i415.
  • 18
    Melnick DE, Dillon GF, Swanson DB. Medical licensing examinations in the United States. J Dent Educ. 2002; 66:5959.
  • 19
    Dillon GF, Clyman SG, Clauser BE, Margolis MJ. The introduction of computer-based case simulations into the United States medical licensing examination. Acad Med. 2002; 77(10 Suppl):S946.
  • 20
    Scoles PV, Hawkins RE, LaDuca A. Assessment of clinical skills in medical practice. J Contin Educ Health Prof. 2003; 23:18290.
  • 21
    Whelan G. High-stakes medical performance testing: the Clinical Skills Assessment program. JAMA. 2000; 283:1748.
  • 22
    Gimpel JR, Boulet JR, Errichetti AM. Evaluating the clinical skills of osteopathic medical students. J Am Osteopath Assoc. 2003; 103:26779.
  • 23
    Harik P, Clauser BE, Grabovsky I, Margolis MJ, Dillon GF, Boulet JR. Relationships among subcomponents of the USMLE Step 2 Clinical Skills Examination, the Step 1, and the Step 2 Clinical Knowledge Examinations. Acad Med. 2006; 81(10 Suppl):S214.
  • 24
    Hauer KE, Hodgson CS, Kerr KM, Teherani A, Irby DM. A national study of medical student clinical skills assessment. Acad Med. 2005; 80(10 Suppl):S259.
  • 25
    Issenberg SB, McGaghie WC, Petrusa ER, Lee GD, Scalese RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach. 2005; 27:1028.
  • 26
    Marshall RL, Smith JS, Gorman PJ, Krummel TM, Haluck RS, Cooney RN. Use of a human patient simulator in the development of resident trauma management skills. J Trauma. 2001; 51:1721.
  • 27
    Hawkins RE, Boulet JR. Direct observation: standardized patients. In: HolmboeES, HawkinsRE (eds). Practical Guide to the Evaluation of Clinical Competence. 1st ed. Philadelphia, PA: Elsevier, 2008, pp 10218.
  • 28
    Boulet JR, Swanson DB. Psychometric challenges of using simulations for high-stakes assessment. In: DunnW (ed). Simulations in Critical Care Education and Beyond. Des Plains, IL: Society of Critical Care Medicine, 2004, pp 11930.
  • 29
    Boulet JR, Gimpel JR, Errichetti AM, Meoli FG. Using National Medical Care Survey data to validate examination content on a performance-based clinical skills assessment for osteopathic physicians. J Am Osteopath Assoc. 2003; 103:22531.
  • 30
    Boulet JR, McKinley DW, Whelan GP, Hambleton RK. Quality assurance methods for performance-based assessments. Adv Health Sci Educ Theory Pract. 2003; 8:2747.
  • 31
    Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006; 119:166.
  • 32
    Downing SM. Reliability: on the reproducibility of assessment data. Med Educ. 2004; 38:100612.
  • 33
    Seropian MA. General concepts in full scale simulation: getting started. Anesth Analg. 2003; 97:1695705.
  • 34
    Dawson S. Procedural simulation: a primer. Radiology. 2006; 241:1725.
  • 35
    Issenberg SB, McGaghie WC, Hart IR, et al. Simulation technology for health care professional skills training and assessment. JAMA. 1999; 282:8616.
  • 36
    Kneebone RL, Nestel D, Vincent C, Darzi A. Complexity, risk and simulation in learning procedural skills. Med Educ. 2007; 41:80814.
  • 37
    McLaughlin SA, Doezema D, Sklar DP. Human simulation in emergency medicine training: a model curriculum. Acad Emerg Med. 2002; 9:13108.
  • 38
    Accreditation Council for Graduate Medical Education. ACGME Outcome Project. 2004. Available at: Accessed Jul 16, 2008.
  • 39
    Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA. 2002; 287:22635.
  • 40
    Farrell SE. Evaluation of student performance: clinical and professional performance. Acad Emerg Med. 2005; 12:302e610.
  • 41
    Schuwirth LW, Van Der Vleuten CP. The use of clinical simulations in assessment. Med Educ. 2003; 37(Suppl 1):6571.
  • 42
    Ziv A, Boulet JR, Burdick WP, Friedman Ben-David M, Gary NE. The use of national medical care surveys to develop and validate test content for standardized patient examinations. In: MelnickD (ed). Proceedings of the Eighth Ottawa Conference on Medical Education and Assessment. Philadelphia, PA: National Board of Medical Examiners, 2000, pp 99105.
  • 43
    National Center for Health Statistics. National Hospital Ambulatory Medical Care Survey (NHAMCS). Hyattsville, MD. Available at: Accessed Jul 16, 2008.
  • 44
    Cumin D, Merry AF. Simulators for use in anaesthesia. Anaesthesia. 2007; 62:15162.
  • 45
    Bond WF, Lammers RL, Spillane LL, et al. The use of simulation in emergency medicine: a research agenda. Acad Emerg Med. 2007; 14:35363.
  • 46
    Bond WF, Spillane L. The use of simulation for emergency medicine resident assessment. Acad Emerg Med. 2002; 9:12959.
  • 47
    Rosen M, Salas E, Wilson K, et al. Measuring team performance in simulation-based training: adopting best practices for healthcare. Sim Healthcare. 2008; 3:3341.
  • 48
    Issenberg SB, Scalese RJ. Simulation in health care education. Perspect Biol Med. 2008; 51:3146.
  • 49
    Ogden PE, Cobbs LS, Howell MR, Sibbitt SJ, DiPette DJ. Clinical simulation: importance to the internal medicine educational mission. Am J Med. 2007; 120:8204.
  • 50
    Carroll JD, Messenger JC. Medical simulation: the new tool for training and skill assessment. Perspect Biol Med. 2008; 51:4760.
  • 51
    Whelan GP, Boulet JR, McKinley DW, et al. Scoring standardized patient examinations: lessons learned from the development and administration of the ECFMG Clinical Skills Assessment (CSA). Med Teach. 2005; 27:2006.
  • 52
    Adler MD, Trainor JL, Siddall VJ, McGaghie WC. Development and evaluation of high-fidelity simulation case scenarios for pediatric resident education. Ambul Pediatr. 2007; 7:1826.
  • 53
    Murray DJ, Boulet JR, Avidan M, et al. Performance of residents and anesthesiologists in a simulation-based skill assessment. Anesthesiology. 2007; 107:70513.
  • 54
    Scavone BM, Sproviero MT, McCarthy RJ, et al. Development of an objective scoring system for measurement of resident performance on the human patient simulator. Anesthesiology. 2006; 105:2606.
  • 55
    Morgan PJ, Cleave-Hogg D, Guest CB. A comparison of global ratings and checklist scores from an undergraduate assessment using ananesthesia simulator. Acad Med. 2001; 76:10535.
  • 56
    Morgan PJ, Pittini R, Regehr G, Marrs C, Haley MF. Evaluating teamwork in a simulated obstetric environment. Anesthesiology. 2007; 106:90715.
  • 57
    Gordon JA, Tancredi DN, Binder WD, Wilkerson WM, Shaffer DW. Assessment of a clinical performance evaluation tool for use in a simulator-based testing environment: a pilot study. Acad Med. 2003; 78(10 Suppl 1):S457.
  • 58
    Murray DJ, Boulet JR, Kras JF, Woodhouse JA, Cox T, McAllister JD. Acute care skills in anesthesia practice: a simulation-based resident performance assessment. Anesthesiology. 2004; 101:108495.
  • 59
    Morgan PJ, Lam-McCulloch J, Herold-McIlroy J, Tarshis J. Simulation performance checklist generation using the Delphi technique. Can J Anaesth. 2007; 54:9927.
  • 60
    Morgan PJ, Cleave-Hogg D, DeSousa S, Tarshis J. High-fidelity patient simulation: validation of performance checklists. Br J Anaesth. 2004; 92:38892.
  • 61
    Boulet JR, Van Zanten M, De Champlain A, Hawkins RE, Peitzman SJ. Checklist content on a standardized patient assessment: an ex post facto review. Adv Health Sci Educ. 2008; 13:5969.
  • 62
    Hunt EA, Walker AR, Shaffner DH, Miller MR, Pronovost PJ. Simulation of in-hospital pediatric medical emergencies and cardiopulmonary arrests: highlighting the importance of the first 5 minutes. Pediatrics. 2008; 121:e3443.
  • 63
    Girzadas DV Jr, Clay L, Caris J, Rzechula K, Harwood R. High fidelity simulation can discriminate between novice and experienced residents when assessing competency in patient care. Med Teach. 2007; 29:4526.
  • 64
    Baker DP, Salas E, King H, Battles J, Barach P. The role of teamwork in the professional education of physicians: current status and assessment recommendations. Jt Comm J Qual Patient Saf. 2005; 31:185202.
  • 65
    Van Zanten M, Boulet JR, McKinley DW, De Champlain A, Jobe AC. Assessing the communication and interpersonal skills of graduates of international medical schools as part of the United States Medical Licensing Exam (USMLE) Step 2 Clinical Skills (CS) Exam. Acad Med. 2007; 82(10 Suppl l):S658.
  • 66
    Weller JM, Bloch M, Young S, et al. Evaluation of high fidelity patient simulator in assessment of performance of anaesthetists. Br J Anaesth. 2003; 90:437.
  • 67
    Boulet JR, Rebbecchi TA, Denton EC, McKinley DW, Whelan GP. Assessing the written communication skills of medical school graduates. Adv Health Sci Educ Theory Pract. 2004; 9:4760.
  • 68
    Swanson DB, Clauser BE, Case SM. Clinical skills assessment with standardized patients in high-stakes tests: a framework for thinking about score precision, equating, and security. Adv Health Sci Educ Theory Pract. 1999; 4:67106.
  • 69
    Holmboe ES, Cassel CK. The role of physicians and certification boards to improve quality. Am J Med Qual. 2007; 22:1825.
  • 70
    Klass D. Assessing doctors at work--progress and challenges. N Engl J Med. 2007; 356:4145.
  • 71
    Van Der Vleuten CP, Norman GR, De Graaff E. Pitfalls in the pursuit of objectivity: issues of reliability. Med Educ. 1991; 25:1108.
  • 72
    Shayne P, Gallahue F, Rinnert S, Anderson CL, Hern G, Katz E. Reliability of a core competency checklist assessment in the emergency department: the Standardized Direct Observation Assessment Tool. Acad Emerg Med. 2006; 13:72732.
  • 73
    Brett-Fleegler MB, Vinci RJ, Weiner DL, Harris SK, Shih MC, Kleinman ME. A simulator-based tool that assesses pediatric resident resuscitation competency. Pediatrics. 2008; 121:e597e603.
  • 74
    Boulet JR. Generalizability theory: Basics. In: EverittBS, HowellDC (eds). Encyclopedia of Statistics in Behavioral Science. Chichester, UK: John Wiley & Sons, Ltd, 2005, pp 70411.
  • 75
    Thompson B. A brief introduction to generalizability theory. In: ThompsonB (ed). Score Reliability: Contemporary Thinking on Reliability Issues. Thousand Oaks, CA: Sage Publications, Inc., 2003, pp 4358.
  • 76
    Vu NV, Barrows HS. Use of standardized patients in clinical assessments: Recent developments and measurement findings. Educ Res. 1994; 23:2330.
  • 77
    Downing SM. Validity: on the meaningful interpretation of assessment data. Med Educ. 2003; 37:8307.
  • 78
    Kane MT. Current concerns in validity theory. J Educ Measur. 2001; 38:31942.
  • 79
    Clauser BE, Margolis MJ, Swanson DB. Issues of validity and reliability for assessments in medical education. In: HolmboeES, HawkinsRE (eds). Practical Guide to the Evaluation of Clinical Competence. 1st ed. Philadelphia, PA: Mosby/Elsevier, 2008, pp 1023.
  • 80
    Newble D. Techniques for measuring clinical competence: objective structured clinical examinations. Med Educ. 2004; 38:199203.
  • 81
    Grand’Maison P, Brailovsky CA, Lescop J. Content validity of the Quebec licensing examination (OSCE). Assessed by practising physicians. Can Fam Phys. 1996; 42:2549.
  • 82
    Berkenstadt H, Ziv A, Gafni N, Sidi A. The validation process of incorporating simulation-based accreditation into the anesthesiology Israeli national board exams. Isr Med Assoc J. 2006; 8:72833.
  • 83
    Gordon JA, Wilkerson WM, Shaffer DW, Armstrong EG. “Practicing” medicine without risk: students’ and educators’ responses to high-fidelity patient simulation. Acad Med. 2001; 76:46972.
  • 84
    Rosenthal R, Gantert WA, Hamel C, et al. Assessment of construct validity of a virtual reality laparoscopy simulator. J Laparoendosc Adv Surg Tech A. 2007; 17:40713.
  • 85
    Schwid HA, Rooke GA, Carline J, et al. Evaluation of anesthesia residents using mannequin-based simulation: a multiinstitutional study. Anesthesiology. 2002; 97:143444.
  • 86
    Young JS, Stokes JB, Denlinger CE, Dubose JE. Proactive versus reactive: the effect of experience on performance in a critical care simulator. Am J Surg. 2007; 193:1004.
  • 87
    Boulet JR, Murray D, Kras J, Woodhouse J. Setting performance standards for mannequin-based acute-care scenarios. Sim Healthcare. 2008; 3:7281.
  • 88
    Reznick RK, MacRae H. Teaching surgical skills--changes in the wind. N Engl J Med. 2006; 355:26649.
  • 89
    Tamblyn R, Abrahamowicz M, Dauphinee D, et al. Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA. 2007; 298:9931001.
  • 90
    Hatala R, Issenberg SB, Kassen B, Cole G, Bacchus CM, Scalese RJ. Assessing cardiac physical examination skills using simulation technology and real patients: a comparison study. Med Educ. 2008; 42:62836.
  • 91
    Leape LL, Fromson JA. Problem doctors: is there a system-level solution? Ann Intern Med. 2006; 144:10715.
  • 92
    Rosenblatt MA, Abrams KJ. The use of a human patient simulator in the evaluation of and development of a remedial prescription for an anesthesiologist with lapsed medical skills. Anesth Analg. 2002; 94:14953.
  • 93
    Scalese RJ, Obeso VT, Issenberg SB. Simulation technology for skills training and competency assessment in medical education. J Gen Intern Med. 2008; 23(Suppl 1):469.
  • 94
    Fox-Robichaud AE, Nimmo GR. Education and simulation techniques for improving reliability of care. Curr Opin Crit Care. 2007; 13:73741.
  • 95
    Knudson MM, Khaw L, Bullard MK, et al. Trauma training in simulation: translating skills from SIM time to real time. J Trauma. 2008; 64:25563.
  • 96
    Ferris TG, Vogeli C, Marder J, Sennett CS, Campbell EG. Physician specialty societies and the development of physician performance measures. Health Aff (Millwood). 2007; 26:17129.