SEARCH

SEARCH BY CITATION

Keywords:

  • *school admission criteria;
  • *schools, medical;
  • students, medical/*psychology;
  • *cognition;
  • educational measurement;
  • Denmark;
  • education, medical, undergraduate;
  • communication;
  • motivation

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References

Objectives  The reliability of individual non-cognitive admission criteria in medical education is controversial. Nonetheless, non-cognitive admission criteria appear to be widely used in selection to medicine to supplement the grades of qualifying examinations. However, very few studies have examined the overall test generalisability of composites of non-cognitive admission variables in medical education. We examined the generalisability of a composite process for selection to medicine, consisting of four variables: qualifications (application form information); written motivation (in essay format); general knowledge (multiple-choice test), and a semi-structured admission interview. The aim of this study was to estimate the generalisability of a composite selection.

Methods  Data from 307 applicants who participated in the admission to medicine in 2007 were available for analysis. Each admission parameter was double-scored using two random, blinded and independent raters. Variance components for applicant, rater and residual effects were estimated for a mixed model with the restricted maximum likelihood (REML) method. The reliability of obtained applicant ranks (G coefficients) was calculated for individual admission criteria and for composite admission procedures.

Results  A pre-selection procedure combining qualification and motivation scores showed insufficient generalisability (G = 0.45). The written motivation in particular, displayed low generalisability (G = 0.10). Good generalisability was found for the admission interview (G = 0.86), and for the final composite selection procedure (G = 0.82).

Conclusions  This study revealed good generalisability of a composite selection, but indicated that the application, composition and weighting of individual admission variables should not be random. Knowledge of variance components and generalisability of individual admission variables permits evidence-based decisions on optimal selection strategies.


Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References

Few test situations during medical school are more deserving of the ‘high-stakes’ designation than the admission test. Therefore, reliability and validity, as well as acceptability and feasibility, must be cornerstones of student selection.1

In our context, reliability of selection is concerned with the level of accuracy with which applicants can be rank-ordered. Unreliable selection is unfair, unethical, a waste of resources and potentially counterproductive to the intended purpose. Furthermore, lack of reliability will have a negative impact on subsequent validity studies that entail looking at relationships between variables.2

There is a preponderance of evidence in support of using previous academic achievements (A-levels, MCAT [Medical College Admission Test] scores etc.) as selection criteria in medical education.3 However, although a previous grade profile is the best independent predictor of pregraduate success known, it is only of moderate strength.3–6 Unfortunately, the overall evidence on most other predictors (often termed ‘non-cognitive’ predictors) is comparatively scarce3 and even less convincing.1,5,7–9 The admission interview is widely used and it is probably the best examined of the non-cognitive selection tools.8 However, although the admission interview appears to have high face validity, its reliability is controversial in health education.5,10,11 Even less evidence exists on the reliability and validity of submitted written statements as selection tools.5,7,12

Only a handful of studies in medical education have examined the reliability of admission criteria using generalisability theory and most of these revolve around the admission interview.10,11,13–15 Very little has been published on the generalisability of composite admission procedures,13 which is most critical to all stakeholders. Generalisability theory is an extension of classical test theory, which incorporates the disentanglement and estimation of multiple sources of error variance within the same study.16 The results may reveal important sources of error variance, which can be targeted to improve reliability. Subsequent mathematical modelling allows for reliability estimates of test conditions and of alternative test strategies. This makes generalisability theory more flexible than classical test theory.16–20

Since 2002, the Faculty of Health Sciences at the University of Southern Denmark has used an admission programme to supplement the traditional admission parameter (previous grades). One incentive for this has been the relatively high attrition rates in Denmark.21,22 Attrition appeared to stem, at least in part, from applicants being insufficiently reflective and informed regarding choice of programme and future career.23,24 The supplementary battery of admission criteria subsequently developed consisted of measures of level of qualification, motivation, general knowledge and interview performance.

The aim of this study was to estimate the generalisability of the admission to medicine process during spring 2007. The objectives were: to estimate the contributions to variance in scores; to estimate the generalisability of individual admission parameters, and to estimate the composite generalisability of the overall admission process.

Methods

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References

Participants

The study sample consisted of applicants to the medical programme who participated in the admission test at the Faculty of Health Sciences, University of Southern Denmark in May 2007. The 307 highest ranking applicants of the 1404 participants in a pre-selection process were invited to participate on the admission test day. Pre-selection for medicine was based on a combination of qualification and motivation scores. Approximately 150 available spaces were to be allocated to the highest ranking applicants in the final composite admission process.

Admission variables

The qualification score was derived from the application form. Applicants submitted a standard national application form which contained specific questions to be used in admission to higher education in Denmark. Scores were assigned for: relevance and quantity of previous work experience (0–35 points); past educational qualifications (0–35 points); foreign exchange experience (0–10 points), and organisational or voluntary work experience (0–20 points). Highest scores by category were assigned to jobs involving care, qualifications in health sciences, exchange experiences involving care or voluntary work, and organisational or voluntary work with leadership experience, respectively. An overall qualification score of 0–100 points was assigned to each applicant by summing the scores in each category.

The motivation measure was based on a statement written in essay format to assess: written communication skills; knowledge of the chosen programme and profession; reflections on past experiences; reflections on choice of programme, and future employment plans. The written motivation was prepared off-site and submitted with the application form. Each of the five sub-domains was tentatively assigned a plus or a minus which was intended to guide one overall score of 0–100 on a global rating scale.

General knowledge was assessed with a 60-question, 15-minute multiple-choice test, covering a wide variety of content sub-domains, such as biology, physics, arts, current events, music, health, politics etc. The format was ‘one best answer’. The number of correct answers in the general knowledge test was converted to a 0–100 percentage score. The test was developed by a chief psychologist from the Section for Selection at the Danish Army’s Institute for Military Psychology.

The admission interview was a 25-minute, semi-structured interview which aimed to assess: subject interest; expectations; maturity for age; social skills; stress tolerance; empathy, and general interview behaviour. Each of these domains was given a tentative score of 1–5 which was used to guide the interviewer towards one overall score of 0–100 on a global rating scale. A list of 68 appropriate key questions was available, but interviewers were free to supplement these with their own questions where necessary to uncover relevant information on the domains to be assessed only. The general knowledge test and admission interview both took place on the admission test day.

The 0–100 global rating scales used for the written motivation and the interview were subdivided into 11 numbered points (i.e. 0, 10, 20…100), mainly to ensure a self-explanatory midpoint. No testing of the scales has currently been undertaken. Each admission variable was, for the purpose of this study, scored by two random, independent raters on separate marking sheets. The general knowledge questionnaires were scanned and scored electronically on two separate occasions, thus eliciting two independent ratings of the same performance. The raters of qualification and motivation were relatively inexperienced hired student raters. Interview panels consisted of one staff and one student interviewer, and each panel interviewed around six or seven applicants.

The raters of qualification and motivation were blinded to scores other than their own. Interview raters were given thorough instructions on good habits for independent scoring on at least two occasions, both in writing and verbally. To prevent bias, they were asked to avoid discussing scores before scoring, to maintain neutral body language until scoring was complete, and not to change the first score given. Afternoon training sessions were held for all raters.

Analysis

For quality control purposes, paper data was converted to electronic format twice by two different operators. The two sets of entries were compared against one another once and with the original scores on the marking sheets once.

The disattenuated correlation matrix presented in the results was estimated with stata 9.2 using the best linear unbiased predictor (BLUP) method.

Variance components for a mixed model were estimated with the restricted maximum likelihood (REML) method. This method was used because the design was incomplete and unbalanced. The data can be described as naturalistic and as such contained both crossed and nested situations between applicants and raters.25 However, it was possible to estimate three variance components using the REML method:

  • 1
    inline image, the applicant effect, or the variance in scores which can be attributed to applicant differences (i.e. poor versus good performers);
  • 2
    inline image, the rater effect, or the variance in scores resulting from difference in rater stringency, and
  • 3
    inline image, the residual, or the variance attributable to the applicant and rater interaction plus random error.

Analysis was performed using stata 9.2, which supplied the standard error (SE) and confidence interval (CI) presented as well as the covariances and correlations used to calculate generalisability (G) coefficients.

The G coefficient, a measure of the reliability of the obtained rank order of an applicant, was calculated from the formula:

  • image

for each admission variable. Generalisations were made to a universe of a random rater and a fixed admission variable on this occasion. The decision studies for composite G coefficients for the pre-selection and final selection processes were calculated with mgenova (Robert L Brennan, Iowa Testing Programs, University of Iowa, Iowa City, IA, USA) by direct input of the correlations, variances and covariances for each admission variable estimated with stata 9.2. In mgenova, composite p and pr,e effects were found by summing the weighted elements in the respective variance–covariance matrices and from this a composite G was derived. The formula for estimating the composite G coefficients with mgenova was:

  • image

where Σ denotes that variance–covariance matrices were used. The weights assigned to each variable in the composite G for the 2007 admission were intuitive on the part of the administrative stakeholders.

Alternative test strategies were found by performing alternative multivariate D studies based on the estimated variance–covariance matrices. The examples of alternative test strategies presented were found by weighting those admission parameters with the highest G more and assigning more raters to the admission parameters with the lowest G.

Results

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References

The number of medical applicants participating in the pre-selection process was 1404. Of these, the top 307 applicants participated in the admission test and yielded data for analysis. Of the 307 participants, 203 (66.1%) were women and 104 (33.9%) were men. The mean applicant age was 22.2 years (standard deviation [SD] = 2.9). The participating applicant nationalities were: Danish (224/307, 73.0%); Swedish (53/307, 17.3%); Norwegian (17/307, 5.5%), and other (13/307, 4.2%).

The disattenuated correlation between admission variables was generally quite low, indicating good discriminant validity for the four variables (Table 1).

Table 1.   Disattenuated correlation matrix for admission parameters
Admission variableQualificationMotivationKnowledgeInterview
  1. * P ≤ 0.05; †  0.01

  2. n = 307 medical applicants participating in the 2007 admission process

Qualification1.0   
Motivation− 0.12*1.0  
Knowledge0.00− 0.111.0 
Interview0.050.18− 0.011.0

The variance component analysis revealed that the written motivation generally contained the largest proportion of undifferentiated error compared with the other admission elements (Table 2). It had a very low applicant effect, a considerable rater effect and a large residual (undifferentiated error). By contrast, the admission interview had relatively high applicant effects and only small rater effects and modest residuals (Table 2). Consequently, the G coefficients were poor for the written motivation and good for the admission interview (Table 3).

Table 2.   Variance components for admission to medicine 2007
EffectQualificationWritten motivationGeneral knowledgeAdmission interview
VCSE%d.f.VCSE%d.f.VCSE%d.f.VCSE%d.f.
  1. * The p effect is the variance attributable to differences in applicants

  2. † The r effect is the variance attributable to differences in rater stringency

  3. ‡ The pr,e effect (residual) is the variance attributable to applicant and rater interaction plus random error

  4. VC = variance component value; SE = standard error; % = the relative size of the variance component (i.e. the variance component value divided by the total variance for the admission element); d.f. = degrees of freedom; N/A = not applicable

p*146.0315.5059.5030627.2616.677.61306137.1011.0899.99306345.0332.9674.90306
r21.389.158.712079.9930.4722.3219N/AN/AN/A12.615.420.57120
pr,e78.036.4831.79287251.0720.9370.072880.010.000.01306113.0310.2224.54187
Table 3.   Generalisability of individual admission variables
Admission variableG coefficient (95% confidence interval)
nr = 1nr = 2nr = 3
  1. The G coefficient is the reliability of the obtained rank order of an applicant. The G coefficient was estimated from the formula: inline image, where nr is the number of raters, using the estimated variance components for inline image andinline image reported in Table 2

Qualification0.65 (0.59–0.72)0.79 (0.74–0.84)0.85 (0.81–0.89)
Motivation0.10 (− 0.18 to 0.21)0.18 (− 0.01 to 0.37)0.25 (0.00–0.49)
General knowledge1.00 (1.00–1.00)1.00 (1.00–1.00)1.00 (1.00–1.00)
Interview0.75 (0.70–0.80)0.86 (0.83–0.89)0.90 (0.88–0.93)

The composite pre-selection process was found to have poor generalisability, whereas the final round of the composite selection indicated good generalisability (Table 4). One example of an alternative test strategy to improve the overall composite test generalisability is presented in Table 4 and many other solutions are, of course, possible.

Table 4.   Generalisability of composite selection processes for medicine 2007 and possible alternative test strategies
Admission parameter/processQualificationMotivationGeneral knowledgeInterviewComposite G
nrWtnrWtnrWtnrWt
  1. Nr = number of raters; wt = weights

  2. The composite G coefficients was found by performing D studies with mgenova using direct input of the variance components reported in Table 1. Alternative pre-selection and final selection test strategies are illustrative examples of how reliability could be improved without using extra raters

Pre-selection, medicine 200710.6010.400.45
Final selection, medicine 200710.3510.2020.450.82
Alternative pre-selection test strategy21.00.79
Alternative final selection test strategy20.1510.2520.600.87

Discussion

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References

The final composite admission showed good generalisability. However, random application, composition and weighting of individual admission variables cannot be assumed to be adequate. Knowledge of variance components and generalisability of individual admission variables permits improvements of the generalisability of composite selection procedures.

The written motivation

This written motivation had a very low applicant effect, a considerable rater effect and a large residual (Table 2). It must be remembered that the 307 medical applicants sampled here represented a pre-selected sample of the highest ranked eligible applicants (n = 1404), resulting in a smaller applicant effect. The off-site completion of the written motivation may also have facilitated the result by introducing ghost editing, thereby endangering the uniqueness of the product and resulting in a lower applicant effect.

Two factors in particular may have encouraged ghost editing activities. Firstly, competition for places was fierce. Secondly, the written motivation was part of the first filter – the pre-selection process; hence the stakes involved with it were quite high. Private courses that specialise in coaching applicants to medical school represent an example of organised ghost editing. A post hoc count of this dataset revealed that around a sixth of the invited applicants had attended one particular such course, and that their motivation accounts appeared very similar in composition. Likewise, examples of high-scoring written motivations have been found posted on the Internet. Recent research found that applicants’ performance on an autobiographical submission under normal uncontrolled conditions was unrelated to their performance in on-site, controlled conditions.26 It appears likely that controlled on-site production of written motivation submissions might improve both their reliability and validity.

The rater effect (difference in rater stringency) for the written motivation was also quite large compared with that for qualifications (Table 2). It appears that the task of reading, interpreting and scoring the written motivation essay was much more vulnerable to the effects of rater heterogeneity and rater inexperience than that of simply scoring application form information.

The resulting G coefficient for the written motivation (G = 0.10; Table 3) was in the low range compared with those reported in the existing literature on autobiographical submissions.5,7,12,26 A thorough literature review reported inter-rater reliability coefficients for written submissions in admissions to health education programmes to be 0.15–0.59.5 One study found high overall test generalisability (G = 0.76–0.78) for an off-site written personal statement used in admission to physiotherapy training.14 Sufficient sampling of content (17-item questionnaire) and raters (nr = 3), as well as horizontal scoring, is likely to have contributed positively to this result. Another study estimated the generalisability of an application form as a selection tool to medical school. This application form had a structured, short-answer type format composed of four questions and displayed poor generalisability (G = 0.28–0.37, nr = 1).13

In summary: restriction of range, ghost editing, insufficient sampling of content, too few ratings per applicant and inexperienced raters may have affected the generalisability of the written motivation negatively.

The admission interview

The admission interview is probably the most widely used non-cognitive admission parameter in medical education.10,11,27 Only a handful of studies have examined the generalisability of the admission interview.10,11,13,15,28,29 We found good generalisability for the admission interview (G = 0.86, nr = 2; Table 3). Two studies found similar coefficients (G = 0.70–0.80, nr = 2).15,29 Both these studies generalised to one test form and to a single occasion, as in our case. Most of the other studies have found lower G coefficients (G = 0.27–0.65) for the admission interview,10,11,15,28 probably because a limited number of items were used10,11,28 or because an element of test-retest was involved.10,11,15 In fact, most studies indicate that the student–item interaction (content-specific student performance) is an important source of variance in the admission interview10,11,28–30 as in all assessments.31 Interestingly, one of these studies found that whether the same or different questions were asked of each applicant in an admission interview had very little effect on generalisability (G = 0.57 versus G = 0.55, respectively).28

The rater effect of the admission interview was found to be quite small in our study. One other study found a similar result.29 However, most studies have found different interactions of raters contributing with considerable variance,10,11,15,28,29 so rater is an important variable to include in the design of generalisability studies.

The applicant–occasion interaction is probably also an important source of error variance in scores.11 Usually measurements are taken on one occasion; hence occasion may act as a hidden facet. Consequently, generalisations cannot be made across occasions. The variance associated with a hidden fixed facet will be confounded with the variance components estimated, and result in deflated error variance and inflated applicant effect and generalisability.16 In summary, the generalisability of the admission interview estimated in this study was high compared with previous studies, but the possibility of some confounding from hidden factors (item, occasion) is likely.

Pre-selection process

The pre-selection process used a combination of qualification and motivation scores. The pre-selection process was estimated to have an unacceptably low G coefficient of only 0.45 (Table 4); in other words, the obtained rank order of an applicant in the pre-selection process could not be trusted. It appears that a pre-selection based purely on qualification scored by two raters would have resulted in considerable improvement of generalisability to G = 0.79 (Table 4). However, simply discarding the written motivation could potentially lower test validity and the acceptability of the selection process amongst stakeholders. Furthermore, if qualification scores alone represented the first filter, extra administrative resources might be required to verify the submitted documentation of qualifications instead.

Composite final selection

Not many other studies have reported the overall generalisability of composite admission procedures for comparison. We found good generalisability (G = 0.82) for the final round of a composite selection procedure (Table 4). In other words, we can be relatively confident about the final rank order of participating applicants as long as standardised scores are used. Of course, good reliability means little in the absence of validity, so validity studies of the admission parameters in relation to important outcomes (e.g. attrition, completion time, grades etc.) are paramount. One other study examined a combined admission procedure to medical school and found moderate–good generalisability (G = 0.72–0.74).13 However, most educational measurement experts suggest reliability coefficients of at least 0.90 for very high-stakes assessments.18 Examining alternative final selection strategies showed that using two qualification raters and weighting general knowledge and interview results more heavily would have increased the final composite G from 0.82 to 0.87 (Table 4). The extra rater for qualification could be sponsored by discarding the written motivation (alternative pre-selection; Table 4) with the potentially negative consequences mentioned.

The main limitation of this and similar studies typically involve a failure to disentangle important effects (e.g. items and occasions in this case), which contributes to error and leads to confounding of results. Another (common) limitation is the degree of confinement of the universe of generalisation; in this case it was confined to a fixed universe of admission criteria employed on one fixed occasion. Unfortunately, true replications of authentic admission tests are rarely feasible and items are not always readily disentangled from all the admission parameters used, as was the case for our written motivation and admission interview.

It is recommended that admission boards are made aware of the importance of supporting the generalisability of all steps in the selection process and of composite admission processes because it is overall test generalisability in a specific setting which is most important to all stakeholders. Future research should try to simultaneously disentangle as many important effects (item, rater, occasion etc.) as possible.

Contributors:  LDO’N carried out the data collection and is responsible for the integrity of the data. LDO’N and LK had full access to data and are responsible for the accuracy of the statistical analysis. All authors made substantial contributions to the study conception and design, data acquisition, analysis and interpretation, and the drafting and revision of the paper. All authors approved the final manuscript for publication.

Acknowledgements:  the authors thank all their colleagues at the Admissions Office at the University of Southern Denmark, and particularly Annette Schmidt and Jacob Jensen, for their help and positive attitudes. Special thanks are conveyed to Lis Dyhrberg for her help during data collection.

Funding:  this study was funded by the Institute of Sports Science and Clinical Biomechanics and the Education Development Unit, both of the University of Southern Denmark, Odense, Denmark.

Conflicts of interest:  none.

Ethical approval:  this study was registered with, and fulfilled the requirements of, the Regional Ethics Committee and the Danish Data Protection Agency.

References

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. References
  • 1
    McManus IC. Student selection. In: DentJA, HardenRM, eds. A Practical Guide for Medical Teachers, 2nd edn. Edinburgh: Elsevier 2005;36473.
  • 2
    Kirkwood BR, Sterne AC, eds. Measurement error: assessment and implications. Medical Statistics, 2nd edn. Malden, MA: Blackwell Science 2003;42946.
  • 3
    Ferguson E, James D, Madeley L. Factors associated with success in medical school: systematic review of the literature. BMJ 2002;324 (7343):9527.
  • 4
    McManus IC, Smithers E, Partridge P, Keeling A, Fleming PR. A-levels and intelligence as predictors of medical careers in UK doctors: 20-year prospective study. BMJ 2003;327 (7407):13942.
  • 5
    Salvatori P. Reliability and validity of admissions tools used to select students for the health professions. Adv Health Sci Educ Theory Pract 2001;6 (2):15975.
  • 6
    Christensen J, Juul S. Effects of admission qualifications on medical education. Ugeskr Laeger 1999;161 (3):25762.
  • 7
    Kulatunga-Moruzi C, Norman GR. Validity of admissions measures in predicting performance outcomes: the contribution of cognitive and non-cognitive dimensions. Teach Learn Med 2002;14 (1):3442.
  • 8
    Goho J, Blackman A. The effectiveness of academic admission interviews: an exploratory meta-analysis. Med Teach 2006;28 (4):33540.
  • 9
    McManus IC, Iqbal S, Chandrarajan A, Ferguson E, Leaviss J. Unhappiness and dissatisfaction in doctors cannot be predicted by selectors from medical school application forms: a prospective, longitudinal study. BMC Med Educ 2005;5:38.
  • 10
    Eva KW, Rosenfeld J, Reiter HI, Norman GR. An admissions OSCE: the multiple mini-interview. Med Educ 2004;38 (3):31426.
  • 11
    Kreiter CD, Yin P, Solow C, Brennan RL. Investigating the reliability of the medical school admissions interview. Adv Health Sci Educ Theory Pract 2004;9 (2):14759.
  • 12
    Dore KL, Hanson M, Reiter HI, Blanchard M, Deeth K, Eva KW. Medical school admissions: enhancing the reliability and validity of an autobiographical screening tool. Acad Med 2006;81 (Suppl 10):703.
  • 13
    Oosterveld P, Ten Cate O. Generalisability of a study sample assessment procedure for entrance selection for medical school. Med Teach 2004;26 (7):6359.
  • 14
    Nayer MHJ. The reliability of a personal statement as a selection tool in the admission process. In: ScherpbierAJJA, Van Der VleutenCPM, RethansJJ, Van Der SteegAFW, eds. Advances in Medical Education. Dordrecht: Kluwer Academic Publishers 1997;46578.
  • 15
    Harasym PH, Woloschuk W, Mandin H, Brundin-Mather R. Reliability and validity of interviewers’ judgements of medical school candidates. Acad Med 1996;71 (Suppl 1):402.
  • 16
    Brennan RL. Generalizability Theory. New York, NY: Springer Verlag 2001;14.
  • 17
    Streiner DL, Norman GR. Health Measurement Scales: a Practical Guide to their Development and Use, 3rd edn. Oxford: Oxford University Press 2003;15371.
  • 18
    Downing SM. Reliability: on the reproducibility of assessment data. Med Educ 2004;38 (9):100612.
  • 19
    Shavelson RJ, Webb NM. Generalizability Theory. A Primer. Newbury Park, CA: Sage Publications 1991;1279.
  • 20
    Crossley J, Davies H, Humphris G, Jolly B. Generalisability: a key to unlock professional assessment. Med Educ 2002;36 (10):9728.
  • 21
    Wallstedt B. Admission of applicants to medical studies. Ugeskr Laeger 2004;166 (21):19803.
  • 22
    Undervisningsministeriet. Tal der Taler 2007. Uddannelsesnøgletal 2007. Copenhagen: Undervisningsministeriet 2008. http://www.pub.uvm.dk/2008/taldertaler. [Accessed 22 November 2008.]
  • 23
    Studenterrådet ved Aarhus Universitet. Frafald og Studiemiljø. Århus: Studenterrådet ved Aarhus Universitet 2000. http://www.sr.au.dk/images/stories/dokumenter/frafald%20og%20studiemiljoe.pdf. [Accessed 9 December 2008.]
  • 24
    Undervisningsministeriet. Gennemførelse, Studieskift og Frafald - fra Ungdomsuddannelse til Phd. Copenhagen: Undervisningsministeriet, Institutionsstyrelsen, Statisitk- og Informationskontoret 2000.
  • 25
    Crossley J, Russell J, Jolly B, Ricketts C, Roberts C, Schuwirth L, Norcini J. ‘I’m pickin’ up good regressions’: the governance of generalisability analyses. Med Educ 2007;41 (10):92634.
  • 26
    Hanson MD, Dore KL, Reiter HI, Eva KW. Medical school admissions: revisiting the veracity and independence of completion of an autobiographical screening tool. Acad Med 2007;82 (Suppl 10):811.
  • 27
    McManus IC, Richards P, Winder BC. Do UK medical school applicants prefer interviewing to non-interviewing schools? Adv Health Sci Educ Theory Pract 1999;4 (2):15565.
  • 28
    Kreiter CD, Solow C, Brennan RL, Yin P, Ferguson K, Huebner K. Examining the influence of using same versus different questions on the reliability of the medical school preadmission interview. Teach Learn Med 2006;18 (1):48.
  • 29
    Arce-Ferrer AJCI. Investigating postgraduate college admission interviews: generalisability theory reliability and incremental predictive validity. J Hisp High Educ 2007;6:11834.
  • 30
    Eva KW, Reiter HI, Rosenfeld J, Norman GR. The relationship between interviewers’ characteristics and ratings assigned during a multiple mini-interview. Acad Med 2004;79 (6):6029.
  • 31
    Wass V, Van Der Vleuten C, Shatzer J, Jones R. Assessment of clinical competence. Lancet 2001;357 (9260):9459.