- Top of page
Medical Education 2012:46: 163–171
Context Medical schools continue to seek robust ways to select students with the greatest aptitude for medical education, training and practice. Tests of general cognition are used in combination with markers of prior academic achievement and other tools, although their predictive validity is unknown. This study compared the predictive validity of the Undergraduate Medicine and Health Sciences Admission Test (UMAT), the admission grade point average (GPA), and a combination of both, on outcomes in all years of two medical programmes.
Methods Subjects were students (n = 1346) selected since 2003 using UMAT scores and attending either of New Zealand’s two medical schools. Regression models incorporated demographic data, UMAT scores, admission GPA and performance on routine assessments.
Results Despite the different weightings of UMAT used in selection at the two institutions and minor variations in student demographics and programmes, results across institutions were similar. The net predictive power of admission GPA was highest for outcomes in Years 2 and 5 of the 6-year programme, accounting for 17–35% of the variance; UMAT score accounted for < 10%. The highest predictive power of the UMAT score was 9.9% for a Year 5 written examination. Combining UMAT score with admission GPA improved predictive power slightly across all outcomes. Neither UMAT score nor admission GPA predicted outcomes in the final trainee intern year well, although grading bands for this year were broad and numbers smaller.
Conclusions The ability of the general cognitive test UMAT to predict outcomes in major assessments within medical programmes is relatively minor in comparison with that of the admission GPA, but the UMAT score adds a small amount of predictive power when it is used in combination with the GPA. However, UMAT scores may predict outcomes not studied here, which underscores the need for further validation studies in a range of settings.
- Top of page
The selection of medical students is controversial and the choice of tools with which to select candidates for admission from the much larger applicant pool represents a major challenge.1 Selection processes are required to determine those with the aptitude to complete the programme and go on to become the best doctors, but must also rank students for entry to the limited places available. Most selection tools have face validity; however, determining the extent to which a tool predicts an outcome measure, or its predictive validity, is more difficult. One issue concerns deciding which outcome to validate the tool against; another is that usually only candidates with higher scores gain entry and thus it is uncertain how those with lower scores would have fared.2 A final issue refers to how tools might be combined to enhance their predictive validity.3,4
Prior academic achievement predicts most strongly who will remain in medical school, performance during medical school (especially early on), junior doctor performance and time taken to become a specialist.3–5 Prior academic achievement is measured by results on standardised tests within the school system or a grade point average (GPA) calculated from university papers. Professional attributes may be assessed using personal statements, testimonials, personality and emotional intelligence tests,5,6 or interviews, including multiple mini-interviews.7,8 Although some of these methods show promise, none has proved a better predictor of subsequent performance than prior academic achievement.3,4
Tools have been developed that combine assessments of basic medical science knowledge with measures of more general cognitive skills, such as problem-solving, reasoning and writing skills. Examples include the Medical College Admission Test (MCAT; Association of American Medical Colleges, Washington, DC, USA), used since 1928,9 and the Graduate Australian Medical School Admissions Test (GAMSAT; Australian Council for Educational Research [ACER], Melbourne, Vic, Australia); both are used to select for graduate medical programmes. Test scores on the MCAT are most predictive of performance early in a medical programme,9–11 whereas GAMSAT scores correlate weakly with routine in-course assessments12 and clinical reasoning abilities.13
Tests of general cognition are used increasingly in selection into undergraduate medical programmes.7 By contrast with tools used for graduate-entry programmes, these do not test candidates’ knowledge of basic medical sciences. The UK Clinical Aptitude Test (UKCAT; UKCAT Consortium) scores student ability in four distinct domains: quantitative reasoning; verbal reasoning; abstract reasoning, and decision analysis.14 In Australia and New Zealand, the Undergraduate Medicine and Health Sciences Admission Test (UMAT; ACER) is used. This multiple-choice test consists of three sections which focus, respectively, on: logical reasoning and problem solving; understanding people, and non-verbal reasoning.15
Calls have been made to better establish the predictive validity of the UKCAT and the UMAT across a range of schools and curricula.6,16–19 Furthermore, although these tests are relatively convenient for medical schools,1 applicants are required to travel to specified locations on specified days prior to application to medical school. Such tests may prove a barrier to application for students already disadvantaged socio-economically, educationally or by distance.2 Ironically, these may be the very students schools wish to recruit to meet their social mission.20,21 As these tests do not pre-suppose a curriculum, a counter-argument proposed by test administrators in their support is that they minimise prior educational disadvantage.14
We undertook a cross-institution study with the aim of comparing the predictive validity of UMAT scores, the admission GPA, and combined GPA and UMAT scores for student performance on all routine assessments in two undergraduate medical programmes.
The study setting offers three advantages. For the majority of students, the institutions do not use minimum thresholds in UMAT scores, which allows for a wider range of scores on which to determine predictive validity. Secondly, prior academic achievement is determined on a common set of university courses. Finally, this was a national study.
- Top of page
This study’s main finding is that when the score on a general cognitive test, the UMAT, is combined with admission GPA for selection purposes, the ability to predict outcomes on all major summative assessments in an undergraduate medical programme is enhanced, but not by much. On outcomes in which general cognitive skills might be expected to come into their own, such as the Year 4 communication skills test or Year 5 clinical examination, the UMAT score does not show a predictive advantage over GPA. As the predictive power of the UMAT score is so low, we cannot draw firm conclusions about the relative predictive power of individual UMAT sections.
The final year (Year 6) is the most closely related to junior doctor practice.23 Neither selection tool predicts performance in this year well, although the UMAT was at least as predictive as admission GPA in both programmes. One explanation for the poor predictive ability of selection tools is that performance in Year 6 is measured using relatively blunt instruments, such as supervisor reports, by contrast with Year 5 assessments. Another is that performance at the end of the programme is likely to reflect the numerous confounding factors that influence students’ abilities as they progress, particularly the curriculum itself.
Another finding is that student background has a predictive association with outcomes of a magnitude approximate to that of the UMAT score. Although schools are unlikely to use components of student background, such as age, gender or ethnicity, as selection criteria, this finding underscores the need to consider background factors when evaluating the predictive validity of selection tools.
Like others, we found that prior academic achievement is a moderate predictor of outcomes, but this predictive power drops throughout the programme.3,5,9,27,28 Furthermore, a recent study found UMAT scores to correlate poorly with GPAs over 4 years of university study (2 years in a prior degree and 2 years in medicine),19 but did not evaluate UMAT scores against performance in the final 2 years of the programme, nor in individual assessments. Our data suggest the UMAT has some predictive power for academic outcomes later in a medical programme and that this is of an order of magnitude similar to that of the GPA.
A strength of this study is that it was conducted nationally across two distinct medical programmes. There was consistency in the findings, despite minor differences in students, selection and curricula. Furthermore, the mean UMAT scores (and spread) were similar in the two programmes. This lends support to the claim that these are robust and generalisable findings.
In Australia, most medical schools set a minimum threshold for UMAT scores. With the exception of graduate entrants at Otago, who must score above the 25th centile, no UMAT threshold is imposed in NZ, which yields a potentially wider range of scores upon which to test its predictive ability. It could be argued that as all successful applicants had high GPAs, they would have higher UMAT scores. This is not the case, as the minimum average UMAT score was 31. A future area of study will concern the performance of students with lower UMAT scores and high admission GPAs. Because the GPA contributes the majority of weight to ranking decisions for admission, testing the converse is not possible in this population.
Weaknesses include the fact that outcomes for fewer subjects were evaluable in later years of the programmes, raising the possibility of a type II error. The UMAT may be more predictive at different weightings or when used as a threshold score. Given the low predictive power seen in this study, and the similarity of findings across both universities, we believe it is unlikely that any change in weighting would improve the UMAT’s predictive power significantly for the outcomes we have reported.
Although both medical schools invest considerably in making student assessments authentic and ensuring they are aligned with the domains of medical practice,23,29–31 it is possible that the UMAT predicts outcomes that were not studied. Already UMAT scores have been found not to correlate with emotional intelligence in final-year students.6 If UMAT scores were to predict attrition from a medical programme, the negative impacts on stakeholders of admitting a student who is unlikely to complete medical studies could be reduced. Additionally, the UMAT may be more useful in certain student groups; if so, it may represent a helpful refinement to selection policies. If selection tools are to help predict which candidates will become effective doctors, longer-term follow-up is required, including on eventual specialty and location of practice. As the UMAT has been used in NZ only since 2003, it is not yet possible to measure how it predicts workforce outcomes.
Our study supports the inclusion of a measure of academic achievement in medical student selection.32 As used in the NZ setting, the UMAT score has less predictive ability than the admission GPA, but it does add to the GPA’s predictive power by a small amount, especially later in a medical programme. Given the shortage of reliable tools with which to select medical students, we would favour a strategy in which the validation of the UMAT is continued in various settings and the outcomes on which predictive ability is tested are broadened.
Contributors: PP contributed to the study concept and design, application for ethical approval, and the acquisition, collation and interpretation of data, and led the first and all subsequent drafts and revisions of the manuscript. BS contributed to the analysis and interpretation of data, and the drafting and revision of the manuscript. JR contributed to the study concept and design, the collation and interpretation of data, and the drafting and revision of the manuscript. TW contributed to the study concept and design, application for ethical approval, the acquisition, collation and interpretation of data, and the first and subsequent drafts and revisions of the manuscript. All authors approved the final manuscript for publication.
Acknowledgements: the authors acknowledge the support of Dr John Monigatti, Director of Medical Admissions, Michelle Chung, Student Services Centre, and Ian Wood and Belinda May of the Medical Programme Directorate, University of Auckland, and Melany Rohan, Manager of Health Sciences Admissions, University of Otago.