- Top of page
Objectives The reliability of individual non-cognitive admission criteria in medical education is controversial. Nonetheless, non-cognitive admission criteria appear to be widely used in selection to medicine to supplement the grades of qualifying examinations. However, very few studies have examined the overall test generalisability of composites of non-cognitive admission variables in medical education. We examined the generalisability of a composite process for selection to medicine, consisting of four variables: qualifications (application form information); written motivation (in essay format); general knowledge (multiple-choice test), and a semi-structured admission interview. The aim of this study was to estimate the generalisability of a composite selection.
Methods Data from 307 applicants who participated in the admission to medicine in 2007 were available for analysis. Each admission parameter was double-scored using two random, blinded and independent raters. Variance components for applicant, rater and residual effects were estimated for a mixed model with the restricted maximum likelihood (REML) method. The reliability of obtained applicant ranks (G coefficients) was calculated for individual admission criteria and for composite admission procedures.
Results A pre-selection procedure combining qualification and motivation scores showed insufficient generalisability (G = 0.45). The written motivation in particular, displayed low generalisability (G = 0.10). Good generalisability was found for the admission interview (G = 0.86), and for the final composite selection procedure (G = 0.82).
Conclusions This study revealed good generalisability of a composite selection, but indicated that the application, composition and weighting of individual admission variables should not be random. Knowledge of variance components and generalisability of individual admission variables permits evidence-based decisions on optimal selection strategies.
- Top of page
Few test situations during medical school are more deserving of the ‘high-stakes’ designation than the admission test. Therefore, reliability and validity, as well as acceptability and feasibility, must be cornerstones of student selection.1
In our context, reliability of selection is concerned with the level of accuracy with which applicants can be rank-ordered. Unreliable selection is unfair, unethical, a waste of resources and potentially counterproductive to the intended purpose. Furthermore, lack of reliability will have a negative impact on subsequent validity studies that entail looking at relationships between variables.2
There is a preponderance of evidence in support of using previous academic achievements (A-levels, MCAT [Medical College Admission Test] scores etc.) as selection criteria in medical education.3 However, although a previous grade profile is the best independent predictor of pregraduate success known, it is only of moderate strength.3–6 Unfortunately, the overall evidence on most other predictors (often termed ‘non-cognitive’ predictors) is comparatively scarce3 and even less convincing.1,5,7–9 The admission interview is widely used and it is probably the best examined of the non-cognitive selection tools.8 However, although the admission interview appears to have high face validity, its reliability is controversial in health education.5,10,11 Even less evidence exists on the reliability and validity of submitted written statements as selection tools.5,7,12
Only a handful of studies in medical education have examined the reliability of admission criteria using generalisability theory and most of these revolve around the admission interview.10,11,13–15 Very little has been published on the generalisability of composite admission procedures,13 which is most critical to all stakeholders. Generalisability theory is an extension of classical test theory, which incorporates the disentanglement and estimation of multiple sources of error variance within the same study.16 The results may reveal important sources of error variance, which can be targeted to improve reliability. Subsequent mathematical modelling allows for reliability estimates of test conditions and of alternative test strategies. This makes generalisability theory more flexible than classical test theory.16–20
Since 2002, the Faculty of Health Sciences at the University of Southern Denmark has used an admission programme to supplement the traditional admission parameter (previous grades). One incentive for this has been the relatively high attrition rates in Denmark.21,22 Attrition appeared to stem, at least in part, from applicants being insufficiently reflective and informed regarding choice of programme and future career.23,24 The supplementary battery of admission criteria subsequently developed consisted of measures of level of qualification, motivation, general knowledge and interview performance.
The aim of this study was to estimate the generalisability of the admission to medicine process during spring 2007. The objectives were: to estimate the contributions to variance in scores; to estimate the generalisability of individual admission parameters, and to estimate the composite generalisability of the overall admission process.
- Top of page
The number of medical applicants participating in the pre-selection process was 1404. Of these, the top 307 applicants participated in the admission test and yielded data for analysis. Of the 307 participants, 203 (66.1%) were women and 104 (33.9%) were men. The mean applicant age was 22.2 years (standard deviation [SD] = 2.9). The participating applicant nationalities were: Danish (224/307, 73.0%); Swedish (53/307, 17.3%); Norwegian (17/307, 5.5%), and other (13/307, 4.2%).
The disattenuated correlation between admission variables was generally quite low, indicating good discriminant validity for the four variables (Table 1).
Table 1. Disattenuated correlation matrix for admission parameters
|Qualification||1.0|| || || |
|Motivation||− 0.12*||1.0|| || |
|Knowledge||0.00||− 0.11||1.0|| |
The variance component analysis revealed that the written motivation generally contained the largest proportion of undifferentiated error compared with the other admission elements (Table 2). It had a very low applicant effect, a considerable rater effect and a large residual (undifferentiated error). By contrast, the admission interview had relatively high applicant effects and only small rater effects and modest residuals (Table 2). Consequently, the G coefficients were poor for the written motivation and good for the admission interview (Table 3).
Table 2. Variance components for admission to medicine 2007
|Effect||Qualification||Written motivation||General knowledge||Admission interview|
Table 3. Generalisability of individual admission variables
|Admission variable||G coefficient (95% confidence interval)|
|nr = 1||nr = 2||nr = 3|
|Qualification||0.65 (0.59–0.72)||0.79 (0.74–0.84)||0.85 (0.81–0.89)|
|Motivation||0.10 (− 0.18 to 0.21)||0.18 (− 0.01 to 0.37)||0.25 (0.00–0.49)|
|General knowledge||1.00 (1.00–1.00)||1.00 (1.00–1.00)||1.00 (1.00–1.00)|
|Interview||0.75 (0.70–0.80)||0.86 (0.83–0.89)||0.90 (0.88–0.93)|
The composite pre-selection process was found to have poor generalisability, whereas the final round of the composite selection indicated good generalisability (Table 4). One example of an alternative test strategy to improve the overall composite test generalisability is presented in Table 4 and many other solutions are, of course, possible.
Table 4. Generalisability of composite selection processes for medicine 2007 and possible alternative test strategies
|Admission parameter/process||Qualification||Motivation||General knowledge||Interview||Composite G|
|Pre-selection, medicine 2007||1||0.60||1||0.40||–||–||–||–||0.45|
|Final selection, medicine 2007||1||0.35||–||–||1||0.20||2||0.45||0.82|
|Alternative pre-selection test strategy||2||1.0||–||–||–||–||–||–||0.79|
|Alternative final selection test strategy||2||0.15||–||–||1||0.25||2||0.60||0.87|