ACADEMIC EMERGENCY MEDICINE 2010; 17:1–3 © 2010 by the Society for Academic Emergency Medicine
Objectives: Osteopathic medical students must take the Comprehensive Osteopathic Medical Licensing Examination (COMLEX)–USA series of examinations, but not the United States Medical Licensing Examination (USMLE-1) series. Few data are available describing the comparability of the two tests. This study sought to determine if COMLEX-1 scores could predict USMLE-1 scores among osteopathic medical students applying to an emergency medicine (EM) residency and to determine if the scores are interchangeable.
Methods: This was a retrospective analysis of osteopathic medical students applying to an EM residency program in the 2005–2006 and 2006–2007 application seasons. Students were included if they took both the COMLEX-1 and the USMLE-1 examinations. Linear regression was performed and a Bland-Altman plot of the standardized mean scores of each test was created.
Results: Ninety students were included. The mean (± standard deviation [SD]) COMLEX-1 score was 559.5 (±68.6), and the mean (±SD) USMLE-1 score was 207.6 (±15.5). The correlation was 0.79, with an R2 of 62.3%. The Bland-Altman plot showed a mean difference between the standardized scores of 0, with 95% confidence intervals (CIs) of −1.28 to +1.28 standard normal units. Limitations include that this was a single center study, and only students who took both tests could be studied.
Conclusions: COMLEX-1 scores predict only 62.3% of the variance in USMLE-1 scores, and the scores are not interchangeable.
Osteopathic medical students take the Comprehensive Osteopathic Medical Licensing Examination–USA (COMLEX-USA) as part of their licensing procedure. Although osteopathic students desiring certification by the National Board of Osteopathic Medical Examiners are required to take and pass the COMLEX-1 and COMLEX-2 examinations before beginning a residency training program, they are not required to take the United States Medical Licensing Examination (USMLE). Many allopathic emergency medicine (EM) residencies require that students take either COMLEX-1 or USMLE-1, and some require USMLE-1 only. Whether the two tests correlate with each other well enough for COMLEX-1 to be considered equivalent to USMLE-1 is unknown.
Studying the two tests at one osteopathic medical school, Slocum and Louder1 found a correlation of 0.83 among 155 students taking both tests. In a larger study of 588 applicants to 23 internal medicine programs, Chick et al.2 found a correlation of 0.85. We sought to determine if COMLEX-1 scores could predict USMLE-1 scores among osteopathic medical students applying to an EM residency program and if the scores are interchangeable.
This was a retrospective review of COMLEX-1 and USMLE-1 scores to assess their interchangeability. No identifying data were abstracted from the application, and therefore the institutional review board classified the study as exempt from review.
Study Setting and Population
Maricopa Medical Center has a 3-year, PGY-1–3 EM residency program that accepts applications through the Electronic Residency Application Service. All applications from osteopathic medical students applying to our program during the 2005–2006 and 2006–2007 application seasons who took both tests were eligible for inclusion. If an applicant applied in both seasons, only the application for the 2005–2006 season was included. If a COMLEX-1 or USMLE-1 test was taken more than once, only the highest score was used. Students were not required to take USMLE-1 to be considered for the program.
The three-digit scores from both the USMLE-1 and the COMLEX-1 were used. USMLE-1 scores range from 0 to 300, and COMLEX-1 scores range from 200 to 800. For first-time test takers, the mean USMLE-1 score varies from 200 to 220, with a standard deviation (SD) of approximately ±20.3 For the COMLEX examination, the mean score is 500, with an SD that varies with each testing period.4 For both tests, the three-digit scores are calculated to ensure that scores from different years are on a common scale and have the same meaning.
Linear regression was used to predict the USMLE-1 score, with COMLEX-1 score as the independent variable. A Bland-Altman plot was created to assess interchangeability of scores. Because the tests are scored on different scales, they are not directly comparable. For instance, a score of 300 on the USMLE-1 does not have the same meaning as a score of 300 on the COMLEX-1, and a change of 10 points in the USMLE-1 score does not indicate the same change in COMLEX-1. Therefore, each score was transformed into a standardized score, by subtracting the mean score and dividing by the standard deviation of the sample. The difference between the standardized scores of COMLEX-1 and USMLE-1 was plotted against the average of the standardized COMLEX-1 and USMLE-1 scores. A range of USMLE-1 scores of ±10 for a given COMLEX-1 score was considered important. For instance, if a given COMLEX-1 score was associated with a USMLE-1 score of 200 ± 10, that would be considered comparable. However, an associated USMLE-1 score of 200 ± 20 would be considered to represent noncomparable tests.
Means (±SD) are presented, with 95% confidence intervals (95% CIs) calculated, and a p < 0.05 was considered statistically significant. SPSS 10.1 (SPSS Inc., Chicago, IL) was used to perform most analyses. MedCalc Software version 10.3 (Medcalc, Mariakerke, Belgium) was used to create the Bland-Altman plot.
Ninety applicants over the two seasons were included in the study. The mean (±SD) COMLEX-1 score was 559.5 (±68.6), and the mean (±SD) USMLE-1 score was 207.6 (±15.5). The correlation between USMLE-1 and COMLEX-1 was 0.79 (95% CI = 0.69 to 0.85; R2 = 62.3%). The beta-coefficient was 0.18 (95% CI = 0.15 to 0.21), and the constant was 105.8 (95% CI = 91.4 to 124.8). The resulting equation is: USMLE-1 = 105.8 + 0.18 × COMLEX-1. There was no evidence for heteroscedasticity. A Bland-Altman plot is presented (Figure 1). The mean (±SD) difference is 0 (±0.65). The upper and lower limits of the 95% CI are 1.28 and −1.28, respectively.
Osteopathic medical students have the option of taking the USMLE series of licensing examinations in addition to the COMLEX series, but most do not. Little research exists on the comparability of the two series. This study suggests that the USMLE-1 and COMLEX-1 scores are correlated, but that COMLEX-1 explains only 62% of the variance associated with the two scores. Additionally, the Bland-Altman plot suggests that the two scores are not interchangeable. Given that the 95% CI of the difference between the two standardized scores is ±1.28, a wide range of possible USMLE-1 scores exists for any given COMLEX-1 score. For instance, although a score of 560 (the mean COMLEX-1 score in this study) would be associated with a score of 208 (the mean USMLE-1 score in this study) if the difference between the standardized scores was 0, the true USMLE-1 score could range from 188 to 227 (mean USMLE-1 score ± 1.28 SD). This range is greater than the range we specified as indicating interchangeability of the two tests. This is a value judgment, and each residency program would need to decide if this range of possible values is narrow enough for them to consider the tests interchangeable or generally comparable for their purposes. If a program uses COMLEX-1 or USMLE-1 as rigid tools for stratifying applicants, our data would not support accepting one of the tests in place of the other.
The mean scores and correlations are similar to those found by Slocum and Louder1 and Chick et al.,2 studying different populations of osteopathic medical students. The regression equation derived by Slocum and Louder1 also explained a similar amount of the variance (R2 = 0.68). This implies that the findings in these studies may be robust and that the two tests measure different areas of knowledge.
Because the two tests may measure different components of medical student education, residency programs may need to decide which test they consider most important for the type of program they have and for the types of students they will recruit. Green et al.5 found that USMLE Step 1 scores ranked second in importance in selection criteria for residencies in many specialties, but only fourth for EM. It would be helpful to know what differences exist between each test, besides the questions specifically relating to osteopathic principles. It is also possible that differences between the tests do not translate into differences in performance during the residency years and beyond. The correlation between internal medicine interns’ USMLE-1 scores and their scores on a standardized patient encounter was only 0.2 in one study.6 On the other hand, a study by Cavalieri et al.7 found a correlation of 0.72 between scores on Part 1 of the former COMLEX examination and scores on the American Osteopathic Board of Internal Medicine certification examination. Future studies will help determine the true relationships between the examinations.
Only one residency program was studied. Applicants to other EM residencies may differ from those who applied to ours, limiting generalizability to other programs. Students were not required to take USMLE-1 to apply to our program, nor were they required to supply USMLE-1 scores if taken. Those who scored highly on one and poorly on another might not have submitted both scores. Only students who reported taking both tests were studied. There may be differences between students who take both tests and those who take only the COMLEX series. This is an area for further study.
Although a correlation exists between USMLE-1 and COMLEX-1 scores for osteopathic students applying to an EM residency, only slightly more than half of the variance in the USMLE-1 score is accounted for by the COMLEX-1 score, and the two tests scores are not interchangeable. Larger studies should be performed to confirm this.