SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References

Medical Education 2012:46: 163–171

Context  Medical schools continue to seek robust ways to select students with the greatest aptitude for medical education, training and practice. Tests of general cognition are used in combination with markers of prior academic achievement and other tools, although their predictive validity is unknown. This study compared the predictive validity of the Undergraduate Medicine and Health Sciences Admission Test (UMAT), the admission grade point average (GPA), and a combination of both, on outcomes in all years of two medical programmes.

Methods  Subjects were students (= 1346) selected since 2003 using UMAT scores and attending either of New Zealand’s two medical schools. Regression models incorporated demographic data, UMAT scores, admission GPA and performance on routine assessments.

Results  Despite the different weightings of UMAT used in selection at the two institutions and minor variations in student demographics and programmes, results across institutions were similar. The net predictive power of admission GPA was highest for outcomes in Years 2 and 5 of the 6-year programme, accounting for 17–35% of the variance; UMAT score accounted for < 10%. The highest predictive power of the UMAT score was 9.9% for a Year 5 written examination. Combining UMAT score with admission GPA improved predictive power slightly across all outcomes. Neither UMAT score nor admission GPA predicted outcomes in the final trainee intern year well, although grading bands for this year were broad and numbers smaller.

Conclusions  The ability of the general cognitive test UMAT to predict outcomes in major assessments within medical programmes is relatively minor in comparison with that of the admission GPA, but the UMAT score adds a small amount of predictive power when it is used in combination with the GPA. However, UMAT scores may predict outcomes not studied here, which underscores the need for further validation studies in a range of settings.


Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References

The selection of medical students is controversial and the choice of tools with which to select candidates for admission from the much larger applicant pool represents a major challenge.1 Selection processes are required to determine those with the aptitude to complete the programme and go on to become the best doctors, but must also rank students for entry to the limited places available. Most selection tools have face validity; however, determining the extent to which a tool predicts an outcome measure, or its predictive validity, is more difficult. One issue concerns deciding which outcome to validate the tool against; another is that usually only candidates with higher scores gain entry and thus it is uncertain how those with lower scores would have fared.2 A final issue refers to how tools might be combined to enhance their predictive validity.3,4

Prior academic achievement predicts most strongly who will remain in medical school, performance during medical school (especially early on), junior doctor performance and time taken to become a specialist.3–5 Prior academic achievement is measured by results on standardised tests within the school system or a grade point average (GPA) calculated from university papers. Professional attributes may be assessed using personal statements, testimonials, personality and emotional intelligence tests,5,6 or interviews, including multiple mini-interviews.7,8 Although some of these methods show promise, none has proved a better predictor of subsequent performance than prior academic achievement.3,4

Tools have been developed that combine assessments of basic medical science knowledge with measures of more general cognitive skills, such as problem-solving, reasoning and writing skills. Examples include the Medical College Admission Test (MCAT; Association of American Medical Colleges, Washington, DC, USA), used since 1928,9 and the Graduate Australian Medical School Admissions Test (GAMSAT; Australian Council for Educational Research [ACER], Melbourne, Vic, Australia); both are used to select for graduate medical programmes. Test scores on the MCAT are most predictive of performance early in a medical programme,9–11 whereas GAMSAT scores correlate weakly with routine in-course assessments12 and clinical reasoning abilities.13

Tests of general cognition are used increasingly in selection into undergraduate medical programmes.7 By contrast with tools used for graduate-entry programmes, these do not test candidates’ knowledge of basic medical sciences. The UK Clinical Aptitude Test (UKCAT; UKCAT Consortium) scores student ability in four distinct domains: quantitative reasoning; verbal reasoning; abstract reasoning, and decision analysis.14 In Australia and New Zealand, the Undergraduate Medicine and Health Sciences Admission Test (UMAT; ACER) is used. This multiple-choice test consists of three sections which focus, respectively, on: logical reasoning and problem solving; understanding people, and non-verbal reasoning.15

Calls have been made to better establish the predictive validity of the UKCAT and the UMAT across a range of schools and curricula.6,16–19 Furthermore, although these tests are relatively convenient for medical schools,1 applicants are required to travel to specified locations on specified days prior to application to medical school. Such tests may prove a barrier to application for students already disadvantaged socio-economically, educationally or by distance.2 Ironically, these may be the very students schools wish to recruit to meet their social mission.20,21 As these tests do not pre-suppose a curriculum, a counter-argument proposed by test administrators in their support is that they minimise prior educational disadvantage.14

We undertook a cross-institution study with the aim of comparing the predictive validity of UMAT scores, the admission GPA, and combined GPA and UMAT scores for student performance on all routine assessments in two undergraduate medical programmes.

The study setting offers three advantages. For the majority of students, the institutions do not use minimum thresholds in UMAT scores, which allows for a wider range of scores on which to determine predictive validity. Secondly, prior academic achievement is determined on a common set of university courses. Finally, this was a national study.

Methods

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References

Setting

The study was carried out in New Zealand (NZ), in which two medical programmes serve a total population of 4.4 million people. Each is a 6-year hybrid programme in which students are selected for entry into the second year (Year 2) of medicine after completing 1 year at university or a prior degree.22 Major summative written and clinical assessments are administered at the end of Year 5, prior to the student undertaking a final apprentice-type year as a trainee intern.23 Students must pass each year before they can progress to the next and performance is assessed against predefined standards.

To be eligible for admission, students must meet predefined academic thresholds during their first-year university courses or prior degrees. Those meeting the threshold are then ranked on their achievements. The University of Auckland calculates the mean of course grades, whereas the University of Otago uses the mean of percentage scores. For simplicity, these are both referred to as the ‘admission GPA’.

Otago introduced the UMAT as a selection tool in 2003, and Auckland in 2004; their first cohorts completed their medical degrees in November 2008 and 2009, respectively. There are minor differences in selection policies between the two programmes. For example, Auckland uses an interview as a third selection tool,24,25 whereas Otago does not. Furthermore, the weightings given to UMAT scores vary between the two programmes: at Otago the UMAT score has a weighting of 34% and admission GPA provides the other 66%; at Auckland, the UMAT score is weighted at 15%, the structured interview score at 25% and admission GPA at 60%.26

In addition to the standard entry pathway, each programme has two pathways of affirmative entry, of which one applies to rural students and the other applies to students of Māori (indigenous) and Pacific backgrounds.26

A bi-university research group was established in 2006 to study the predictive validity of selection tools in the New Zealand setting. Ethics approvals were granted in 2009 by the University of Auckland Human Participants Ethics Committee and the University of Otago Ethics Committee for Study on Human Subjects.

Subjects and outcomes

Data were obtained for all NZ medical students for whom UMAT formed part of their selection process, up to and including the 2009 entry cohort. Those admitted through the Māori and Pacific affirmative pathway were not included in this study because the selection tools are used differently in this group. We recorded the following for each study subject: demographic variables of gender, age and ethnicity; whether the student had been admitted as a graduate or through the rural pathway; admission GPA; scores on the three UMAT sections, and average UMAT score.

Outcomes were results on routine summative in-programme assessments, such as the global year grade. Table 1 shows a description of the components of each outcome and the measurement scales used. Outcomes that are similar between the two programmes appear in the same row. The University of Otago changed some of its assessments during the study period, such as the Year 2 short-answer question (SAQ) examination. As a result, there was some variation in numbers of subjects in each analysis. Where students repeated a year, only the initial grade for that year was included in order to ensure predictive validity was established among students undergoing the same assessments for the first time.

Table 1.   Outcomes in the two programmes in which the predictive validity of the admission grade point average (GPA) and Undergraduate Medicine and Health Sciences Admission Test (UMAT) score was determined
AucklandOtago
OutcomeComponentsScaleOutcomeComponentsScale
  1. GPA = grade point average; MCQ = multiple-choice question; SAQ = short-answer question; OSCE = objective structured clinical examination; SP = standardised patient

Year 2  GPAAverage of grades across all Year 2 coursesContinuous scale: 1–9Year 2 GPAAverage of marks from MCQ and SAQ tests In 2008–2009, it included an OSCEContinuous scale: 1–100
   Year 2 OutcomesAggregate performance3-point scale: distinction, pass, fail
   Year 2 SAQ testMark from end-of-year SAQ testContinuous scale: 1–5
Year 3 GPAAverage of grades across all Year 3 coursesContinuous scale: 1–9Year 3 GPAAverage of grades on MCQ and SAQ tests In 2009, it included an OSCEContinuous scale: 1–5
   Year 3 OutcomesAggregate performance3-point scale: distinction, pass, fail
Year 4 Overall gradeRules-based grade derived from seven clinical attachments, projects, written examinations and communication skills3-point scale: distinction, pass, failYear 4 OutcomesRules-based determination derived from clinical attachments and projects3-point scale: distinction, pass, fail
Year 4 Communication skillsPerformance on history taking and communication task with SP3-point scale: distinction, pass, fail   
Year 5 Overall gradeRules-based grade derived from written examination and clinical assessments3-point scale: distinction, pass, failYear 5 OutcomesRules-based determination derived from clinical attachments, projects and end-of-year examinations3-point scale: distinction, pass, fail
Year 5 Written examinationCombined MCQ and SAQ testContinuous scale: 1–100Year 5 Written examinationCombined MCQ and SAQ testContinuous scale: 1–100
Year 5 Clinical gradeRules-based grade derived from clinical short cases examination, seven clinical attachments, and projects3-point scale: distinction, pass, failYear 5 OSCE7–12-station OSCEContinuous scale: 1–100
Year 6 Overall gradeRules-based grade derived from nine clinical attachments3-point scale: distinction, pass, failYear 6 OutcomesRules-based grade derived from six clinical attachments3-point scale: distinction, pass, fail

Analyses

Separate analyses were conducted for each programme. Each analysis used regression models to measure the predictive association between UMAT score, admission GPA or combined GPA and UMAT score, and the outcomes listed in Table 1 when background factors were controlled for (age, ethnicity, graduate, rural pathway). For students at the University of Auckland, the interview score was included among the background factors.

An R2 multiple linear regression model was used when the dependent variable consisted of continuous scores; a Nagelkerke pseudo R2 ordinal regression model was used when the dependent variable was categorical (Distinction, Pass, Fail). Seven regression models were established for each outcome in each programme; an example is given in Table 2. One of these models included the background variables alone. The remaining models included background variables with admission GPA, UMAT score, admission GPA and UMAT score combined, or one of the UMAT sections. Thus, for each outcome in each year and university, it was possible to quantify the net predictive effect of the UMAT score (overall score or any section score), admission GPA, or both, on outcomes over and above other information available at selection. This was calculated by extracting the percentage of variance explained by the background factors from the total variance explained by the model.

Table 2.   Regression model examining the predictive power of admission grade point average (GPA) and Undergraduate Medicine and Health Sciences Admission Test (UMAT) score on Year 2 GPA; ‘Ref’ represents the reference group
Variable typeDescriptionExample
  1. * People of white ancestry

Dependent variableOutcome measureYear 2 GPA
Independent variablesDemographic variableGender (Ref = female) Ethnicity (Ref = European*) Age at Year 2 Entrance path (Ref = general admission path) Previous degree (Ref = none)
Admission score variablesAdmission GPA UMAT Section 1 score UMAT Section 2 score UMAT Section 3 score

Interactions among the independent variables were not measured because the outcome of interest was the predictive power of these variables. Multi-collinearity among the admission GPA and scores on the UMAT sections was measured by the variance inflation factor (VIF). This was < 1.4 for each regression, well below the unacceptable level of 10. There are no published data on the reliability of the UMAT. Although reliability data are calculated for many of the assessment outcomes that contribute to an overall year result, the reliability of the overall result cannot be determined.

Results

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References

Subject characteristics

Data were available for 1346 students. At least one set of final-year outcome data were available for each programme, along with data for between two and five cohorts for the other outcomes. Demographic data, admission GPA and UMAT scores are shown in Table 3. Cohorts did not vary significantly over time (data not shown).

Table 3.   Subject characteristics, by programme
 Auckland (n = 563)Otago (n = 783)
  1. * Students may report more than one ethnicity; numbers do not include students admitted on the indigenous student pathway

  2. People of white ancestry

  3. SD = standard deviation; GPA = grade point average; UMAT = Undergraduate Medicine and Health Sciences Admission Test

Age at start of Year 2, mean (SD)20.5 years (2.8)19.6 years (1.3)
GenderFemale 52.7%Female 54.7%
Ethnicity, n (%)*European, 286 (51.0%) Māori, 7 (1.2%) Pacific, 8 (1.4%) Asian, 214 (38.0%) (incl. Indian) Other, 43 (7.6%)European, 503 (64.2%) Māori, 14 (1.8%) Pacific 7 (0.9%) Asian, 223 (28.5%) Indian, 40 (5.1%) Other, 39 (5.0%)
Graduate entrants, n (%)93 (16.5%)50 (6.4%)
Rural pathway students, n (%)96 (17.1%)135 (17.2%)
Admission GPA, mean (SD)8.2 (0.76) calculated from grades, maximum 987.2 (4.6) calculated from percentages, maximum 100
UMAT score, mean (SD)57.8 (6.4)58.0 (5.9)
 UMAT Section 159.1 (9.6)59.7 (8.7)
 UMAT Section 256.2 (8.3)56.8 (9.9)
 UMAT Section 358.1 (9.7)57.4 (8.2)

Predictive validity of UMAT score and GPA

The results of the regression model analyses for each programme are shown in Table 4. The net predictive power, or percentage of variance in each of the outcomes explained by the tool(s) above background variance, appears in the last three columns of the table.

Table 4.   Predictive ability in the regression models by dependent variable and models for the University of Auckland and University of Otago medical programmes
 R 2/Nagelkerke pseudo R 2Net predictive power (variance explained in %)
Outcome predictedTypenBackgroundGPAUMATGPA and UMATNet GPA and UMATNet GPANet UMAT
  1. R2 is the ratio of the sum of squares explained by a regression model and the total sum of squares around the mean. The variables in ‘Background’ are gender, ethnicity, age, entrance path, previous degree and interview score (for University of Auckland only). All R2-values are significant at p < 0.05, except those in italics

  2. GPA = grade point average; UMAT = Undergraduate Medicine and Health Sciences Admission Test; L = linear regression; O = ordinal regression; SAQ = short-answer question; OSCE = objective structured clinical examination

University of Auckland medical programme
 Year 2 GPAL5630.0490.3870.0970.39734.833.84.8
 Year 3 GPAL4380.0600.3270.0810.33327.326.72.1
 Year 4 Overall gradeO3150.0710.1610.0810.1649.39.01.0
 Year 4 Communication skillsO3150.0960.1060.1090.1151.91.01.3
 Year 5 Overall gradeO1690.1770.2560.2590.32214.57.98.2
 Year 5 Clinical gradeO1650.1990.2610.2310.2818.26.23.2
 Year 5 Written examinationL1900.1090.1650.2080.24413.55.69.9
 Year 6 Overall gradeO 870.1290.1290.1890.1906.10.06.0
University of Otago medical programme
 Year 2 GPAL5560.0240.2580.0280.26323.923.40.4
 Year 2 OutcomesO7830.0350.2330.0440.24020.519.80.9
 Year 2 SAQL2250.0700.2610.1260.32125.119.15.6
 Year 3 OutcomesO6720.0540.1740.0570.17812.412.00.3
 Year 3 GPAL5470.0580.2910.0660.30324.523.30.8
 Year 4 OutcomesO5290.1130.2000.1210.2089.58.70.8
 Year 5 OutcomesO3630.1250.2980.1780.35422.917.35.3
 Year 5 Written examinationL3610.0730.2720.1010.28921.619.92.8
 Year 5 OSCEL3620.2350.3500.2470.36913.411.51.2
 Year 6 OutcomesO2200.0810.1140.1230.1648.33.34.2

Combining the admission GPA and UMAT scores improved the predictive power of the regression models across all outcome measurements in both programmes. In general, GPA models were more predictive than UMAT score models for all outcome measures, except the Year 6 overall grade in both programmes and some Year 5 outcomes at Auckland. The combined models were at most only 1.5 times more predictive than the GPA models, but up to four times more predictive than the UMAT score models. In both programmes, the combined model was best at predicting outcomes in Years 2 and 5.

The maximum variance explained by any regression model was 35% for the Year 2 GPA at Auckland, by the admission GPA and UMAT score combined; the least variance explained was 0%, for Year 6 outcomes at Auckland, by the GPA alone. At best, the total UMAT score explained 10% of variance in the Year 5 written examination at Auckland. Although the UMAT score had greater predictive power than the GPA for Year 6 outcomes, it explained only 6% of variance at Auckland and 4% at Otago (R2 not statistically significant). When results were analysed by scores on the three individual UMAT sections, there was no substantive change to the findings (data not shown). The highest predictive power of an individual UMAT section was seen for Section 1 in Year 5 at Auckland, where it explained 8% of variance in the written examination mark and 7% in the Year 5 overall grade. For all other outcomes, the net predictive power of any UMAT section was ≤ 5%.

Discussion

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References

This study’s main finding is that when the score on a general cognitive test, the UMAT, is combined with admission GPA for selection purposes, the ability to predict outcomes on all major summative assessments in an undergraduate medical programme is enhanced, but not by much. On outcomes in which general cognitive skills might be expected to come into their own, such as the Year 4 communication skills test or Year 5 clinical examination, the UMAT score does not show a predictive advantage over GPA. As the predictive power of the UMAT score is so low, we cannot draw firm conclusions about the relative predictive power of individual UMAT sections.

The final year (Year 6) is the most closely related to junior doctor practice.23 Neither selection tool predicts performance in this year well, although the UMAT was at least as predictive as admission GPA in both programmes. One explanation for the poor predictive ability of selection tools is that performance in Year 6 is measured using relatively blunt instruments, such as supervisor reports, by contrast with Year 5 assessments. Another is that performance at the end of the programme is likely to reflect the numerous confounding factors that influence students’ abilities as they progress, particularly the curriculum itself.

Another finding is that student background has a predictive association with outcomes of a magnitude approximate to that of the UMAT score. Although schools are unlikely to use components of student background, such as age, gender or ethnicity, as selection criteria, this finding underscores the need to consider background factors when evaluating the predictive validity of selection tools.

Like others, we found that prior academic achievement is a moderate predictor of outcomes, but this predictive power drops throughout the programme.3,5,9,27,28 Furthermore, a recent study found UMAT scores to correlate poorly with GPAs over 4 years of university study (2 years in a prior degree and 2 years in medicine),19 but did not evaluate UMAT scores against performance in the final 2 years of the programme, nor in individual assessments. Our data suggest the UMAT has some predictive power for academic outcomes later in a medical programme and that this is of an order of magnitude similar to that of the GPA.

A strength of this study is that it was conducted nationally across two distinct medical programmes. There was consistency in the findings, despite minor differences in students, selection and curricula. Furthermore, the mean UMAT scores (and spread) were similar in the two programmes. This lends support to the claim that these are robust and generalisable findings.

In Australia, most medical schools set a minimum threshold for UMAT scores. With the exception of graduate entrants at Otago, who must score above the 25th centile, no UMAT threshold is imposed in NZ, which yields a potentially wider range of scores upon which to test its predictive ability. It could be argued that as all successful applicants had high GPAs, they would have higher UMAT scores. This is not the case, as the minimum average UMAT score was 31. A future area of study will concern the performance of students with lower UMAT scores and high admission GPAs. Because the GPA contributes the majority of weight to ranking decisions for admission, testing the converse is not possible in this population.

Weaknesses include the fact that outcomes for fewer subjects were evaluable in later years of the programmes, raising the possibility of a type II error. The UMAT may be more predictive at different weightings or when used as a threshold score. Given the low predictive power seen in this study, and the similarity of findings across both universities, we believe it is unlikely that any change in weighting would improve the UMAT’s predictive power significantly for the outcomes we have reported.

Although both medical schools invest considerably in making student assessments authentic and ensuring they are aligned with the domains of medical practice,23,29–31 it is possible that the UMAT predicts outcomes that were not studied. Already UMAT scores have been found not to correlate with emotional intelligence in final-year students.6 If UMAT scores were to predict attrition from a medical programme, the negative impacts on stakeholders of admitting a student who is unlikely to complete medical studies could be reduced. Additionally, the UMAT may be more useful in certain student groups; if so, it may represent a helpful refinement to selection policies. If selection tools are to help predict which candidates will become effective doctors, longer-term follow-up is required, including on eventual specialty and location of practice. As the UMAT has been used in NZ only since 2003, it is not yet possible to measure how it predicts workforce outcomes.

Our study supports the inclusion of a measure of academic achievement in medical student selection.32 As used in the NZ setting, the UMAT score has less predictive ability than the admission GPA, but it does add to the GPA’s predictive power by a small amount, especially later in a medical programme. Given the shortage of reliable tools with which to select medical students, we would favour a strategy in which the validation of the UMAT is continued in various settings and the outcomes on which predictive ability is tested are broadened.

Contributors:  PP contributed to the study concept and design, application for ethical approval, and the acquisition, collation and interpretation of data, and led the first and all subsequent drafts and revisions of the manuscript. BS contributed to the analysis and interpretation of data, and the drafting and revision of the manuscript. JR contributed to the study concept and design, the collation and interpretation of data, and the drafting and revision of the manuscript. TW contributed to the study concept and design, application for ethical approval, the acquisition, collation and interpretation of data, and the first and subsequent drafts and revisions of the manuscript. All authors approved the final manuscript for publication.

Acknowledgements:  the authors acknowledge the support of Dr John Monigatti, Director of Medical Admissions, Michelle Chung, Student Services Centre, and Ian Wood and Belinda May of the Medical Programme Directorate, University of Auckland, and Melany Rohan, Manager of Health Sciences Admissions, University of Otago.

Funding:  none.

Conflicts of interest:  none.

Ethical approval:  this study was approved in 2009 by the University of Auckland Human Participants Ethics Committee and the University of Otago Ethics Committee for Study on Human Subjects.

References

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References