Candidate evaluation of national recruitment in oral surgery Candidate evaluation of national recruitment in oral surgery

Aim: To ascertain the perceptions and experiences of candidates of national recruitment and multi-station interviews for the selection of oral surgery specialty trainees to UK during the first 3 years of being introduced. Material and methods: A pre-piloted questionnaire was administered in paper format to applicants immediately following completion of the final recruitment day. Applicants were asked six questions regarding appropriateness of each of the stations using a Likert scales (1 strongly disagree to 5 strongly agree). Results: During a 3-year period, feedback was favourable for those stations most likely to be associated with a career in oral surgery, which was in stark contrast to the critical appraisal station which consistently received feedback regarding the time allocated to it. Survey feedback each year was presented to the working group and helped to develop and inform future stations. Conclusions: Candidates considered national recruitment to be both a fair and transparent process, which the majority supporting the move away from traditional CV-focused interviews.


I N TRODUC TION
National recruitment for dental specialty training was piloted for the recruitment of orthodontic trainees in 2012.This followed on from the success of this process in the recruitment of Dental Foundation Training (DFT) posts, formerly known as Vocational Training, in 2011.The selection process for orthodontic specialty trainees was deemed to be robust and fair by both interviewers and interviewees. 1,2This led to the Medical and Dental Recruitment Service, with support from the Committee of Postgraduate Dental Deans (COPDEND), introducing national recruitment across five further dental specialties, including oral surgery.
Organisation of national recruitment in oral surgery, was developed and led by a working group established with key stakeholders from the Speciality Advisory Committee (SAC), British Association of Oral Surgeons (BAOS), Association of British Academic and Oral and Maxillofacial Surgeons (ABAOMS), and Lead Deanery for Oral Surgery (Yorkshire and Humber Deanery).The working group developed a personal specification, self-assessment questionnaire, and designed a multi-station interview process for recruitment for entry into oral surgery specialty training.This was then implemented in 2017.
Candidates applied through the well-established Oriel website. 3Oriel is a UK-wide portal for recruitment to postgraduate medical, dental, public health, healthcare science and pharmacy training programmes.The portal was named Oriel to represent the window of opportunity.The first step in the process was that applicants were required on Oriel to complete an application and the self-assessment questionnaire regarding their ability to meet the person specification for the training programme, as determined by their previous experience and achievements as detailed in their portfolio.A multi-station interview process was then chosen as it has a well-established evidence-base throughout dental and medical, undergraduate and postgraduate recruitment processes. 4,5][8] Furthermore, CV-based interviews have been demonstrated to poorly correlate with successful applicants in work performance. 9Multi-station interviews were designed to incorporate those skills which the working group deemed most important in applicants to oral surgery specialty training.Factors considered included the competencies deemed historically appropriate as per the training curriculum, review of traditional criteria for recruitment, and the experiences of orthodontic national recruitment.Stations had a predefined question or scenario, and mark scheme which was not available to candidates.
The first round of national recruitment took place in 2017.The number of posts increased from 2017 to 2018.Independently, the number of applicants also increased from 2017 to 2018 and was static thereafter (Table 1).Although consistent in subsequent years, in each year, a small number of applicants declined to attend for interview at short notice, for example three applicants in 2017.
In response to interviewer feedback, candidate performance and feedback, and review of the evidence-base for validity and reliability of the assessment tools, stations were reviewed annually for inclusion or reformatting.For example, review of portfolios at the time of interview in 2017 revealed a small number of discrepancies between selfassessment scores and subsequent scoring of portfolios by interviewers.This feedback was instrumental in introducing a validation process from 2018 onwards.This involved all short-listed applicants submitting their evidence in advance of the recruitment days.The working group was then tasked with validating all candidates submitted evidence and any discrepancies in scores were discussed at the working group meeting prior to the national recruitment process.In 2019, 59 candidates' evidence was validated, 31 self-assessment scores remained the same, four candidates scores increased, and 23 candidate's scores decreased.Additionally, the inclusion of a suturing station in 2017 was thereafter omitted as it was not considered suitably discriminatory between candidates.In its place, an alternative clinical skills station with a manual dexterity assessment were introduced.The manual dexterity station consisted of a Purdue Peg Board, which is a valid and reliable method of measuring the quality and speed of performance of the hand as the person accomplishes a task. 10nclusion of additional stations, most notably the inclusion of a clinical stills station, necessitated the extension of the national recruitment process from 1 day to two consecutive days.

A I M
A 3-year study was designed to ascertain the perceptions and experiences of candidates of national recruitment and multistation interviews for the selection of oral surgery specialty trainees to UK training programmes from 2017 to 2019.

M ATER I A L S A N D M ETHODS
A 48 item questionnaire designed and pre-piloted by Health Education England, was administered in paper format to applicants immediately following completion of the final recruitment day.Survey completion was optionally anonymous.Candidates were reassured of the anonymous nature of the survey and that following data extraction, paper copies of the questionnaire would be destroyed.
The questionnaire captured applicants' agreement in relation to the appropriateness of interview stations via Likert scales (1 strongly disagree to 5 strongly agree).Applicants were asked six questions regarding the appropriateness of each of the stations:

Scientific rationale for study
As described in the NHS Long Term Plan, the NHS's greatest strength is its people, and through better recruitment it is ensured that there are enough people working to support patients and to deliver the best possible care.

Principal findings
The findings of this study inform the profession, and the future workforce, of the experiences of candidates applying to enter clinical training in oral surgery.

Practical implications
National recruitment for oral surgery continues to evolve as a result of this study.
Each station, therefore, had a minimum score of and a maximum score of 30.Free text comments were encouraged.
Quantitative data were entered in Excel to produce cumulative scores for each of the stations.Qualitative data were copied verbatim into NVivo for A) coding as positive, negative or neutral B) thematic analysis.

2017
100% (n = 102) of applicants completed the survey.The applicant feedback was favourable for those stations which assessed skills that were likely to be completed on a routine basis by those interested in a career in oral surgery (Figure 1) with suturing receiving the highest score for relevance to oral surgery of any of the individual stations at 96% (n = 98).This was in stark contrast to critical appraisal where 53% (n = 54) applicants reported the least relevance to oral surgery training.The majority of negative written comments 33% (n = 22) were in relation to critical appraisal, with candidates stating insufficient time was allocated to this station.This station was a short article, with candidates asked a series of questions relating to study methodology, but also critical appraisal in general.

Free-text comments
'Critical appraisal is good but needed more time'.'Insufficient time for this station'.Also drawing negative feedback in this year was the medical emergency station which applicants deemed to be a memory test rather than application of knowledge.
'If I were to set up a practice I would investigate this information rather than do it from memory'.
A word frequency cloud is a visual representation of word frequency derived from written text.The more often the word appears within the free texts being analysed, the larger it appears in the image generated.The NVivo analysis produced a word cloud frequency for 2017 which drew out the main theme of time (Figure 2).Candidates also supported the move towards a national recruitment process, deeming it fair, relevant and the recruitment team friendly.
The feedback was presented and discussed by the oral surgery working group to in order to inform development of the process.It was felt as critical appraisal had a cumulative score of 50% (15.3) and the qualitative comments reported a perceived lack of time to complete this station, that going forwards the allocated time should be increased.Despite the poor feedback from candidates, the working group felt that an ability to critically appraise evidence was an essential skill for all clinicians whether academic or not.All themed comments were discussed and incorporated into planning for the following year.

2018
In 2018, 97% of candidates completed the optional questionnaire seeking their views on the national recruitment process.The portfolio station was altered in response to best practice guidance published by Dental MDRS regarding time allocation.This station had a greater time allocation and feedback from candidates improved from 2017 (25.9) to 2018 (27.2).Negative comments provided in 2018 in relation to portfolio were suggestive of candidates wishing for a more formal CV-driven interview: 'No opportunity to demonstrate your CV to date'.
Once again critical appraisal received the lowest cumulative score of any station with only 60% (18.1).Applicant feedback again was centred around time allocation for this station.
'Not enough time to read AND answer the questions'.'This is appropriate for OS training, but I was not given enough time so could not complete'.
Both new stations; clinical skills station and Purdue Peg board, drew both positive and negative feedback from candidates.During post-interview analysis, both stations were found to positively correlate with the highest scoring candidates in the process.The candidates free-text comments for the Purdue Peg Board varied between enjoyable and explaining the room temperature effected their performance: 'It was very cold so I was shivering!' 'Very good method to test dexterity.Acted as a good break from remaining assessment.Good as not something you could revise for, but tested your ability and skills'.
The management scenario drew negative feedback for the use of English-based guidance which candidates felt disadvantaged those who had only worked in Scotland, Wales or Northern Ireland.Overall, the comments received were positive and in support of this station remaining as part of the recruitment process.
'Different complaint processes, structure of government between Scotland (where I train) and England!' 'Good station -keep'.The NVivo analysis produced a word cloud frequency for 2018 (Figure 3) which drew out the largest theme of time, this was once again in relation to critical appraisal.The introduction of a new environment, namely the introduction of a clinical skills station in a phantom head suite, created the next biggest themes.

2019
In 2019, the medical emergency station was withdrawn and replaced with two new stations prescription 70% (21) and radiographic report 78% (23.5) which received mainly positive comments from candidates.Negative-free text comments centred around the curriculum of oral surgery and whether candidates felt these stations accurately reflected that.The NVivo analysis produced a word frequency cloud for 2019 (Figure 4) which drew out once again time as the biggest theme.Also strongly featured in 2019 were fair and relevant, once again demonstrating candidates support for National Recruitment.
'Appropriate for oral surgery trainee given maxfax experience'.
'If I were going to prescribe something for a condition, I would ask oral medicine, not guess'.
'Radiographic reports in my hospital are written by radiology consultants, I don't see the relevance to oral surgery'.
The critical appraisal station score improved on the previous 2 years for the first time not receiving the lowest score of any station.
The four consistent stations throughout all 3 years (consent, portfolio, management and communication) received slightly lower scores compared to the previous 2 years (Figure 1).

DISCUS SION
One hundred percent of applicants completed the survey in 2017, but this declined in both 2018 and 2019.It was thought this may be due to applicants re-attending, therefore in 2019, an additional question was asked seeking to find out if applicants had attended national recruitment previously.Of the 94% (n = 34) of applicants who completed the survey in 2019, 33% (n = 12) had attended  Data analysis has provided evidence to support candidate experience in a novel recruitment process, has informed future delivery, and furthermore has supported the consistency of the short-listing process, as a result of data presented by repeat applicants.In line with similar studies, applicants found national recruitment to be a fair process. 2 There are, however, limitations to the study design which must be acknowledged, notably the collection of data following the completion of day two.There is potential for candidates to provide positive feedback at this stage, not knowing whether they are successful or not.Of course, the counter could be argued.Although candidates were reassured that the data were anonymous, there was potential for candidates to believe their handwriting could be used to identify them from candidate answer sheets, therefore, providing more favourable written comments.
Candidate feedback has been crucial to developing and improving the recruitment process.For example critical appraisal in 2017, candidates reported was too challenging in the time provided.Despite low satisfaction with this station, it was retained as the working group felt that critical appraisal ability is a core requisite of a future specialist.For future recruitment, however, the time was extended and the feedback improved consistently with this.
Suturing station, which statistically showed no discrimination between candidates was removed despite positive candidate feedback as it was not deemed to be adding to the recruitment process.
Validation of self-assessment scores was introduced in the second year of National Recruitment (2018) following concerns raised by interviewers in 2017 during the portfolio station.This process has consistently resulted in self-assessment scores being reduced each year, albeit the numbers are gradually decreasing.The process of validating self-assessment scores has now been adopted by other dental specialities undertaking National Recruitment.
In line with previous evidence, candidates supported the transition to National Recruitment, with candidates supporting a more open playing field.

CONCLUSION
The findings from this research have informed further development of the process.Candidates have consistently considered the process to be fair and well organised.In 2020, national recruitment across speciality training was disrupted by COVID-19.No formal evaluation of the process was planned for 2020.National Recruitment for 2021 followed the same structure as set out by COPDEND for all dental speciality recruitment.

F I G U R E 1
Mean applicant feedback for each station from 2017 to 2019.
1752248x, 2023, 1, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/ors.12754by Test, Wiley Online Library on [16/10/2023].See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions)on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License previously.This demonstrates competition for oral surgery training posts, but also the popularity of the speciality.This is ref lected in the year-on-year increase in applications for training posts, with initially 110 in 2017 and most recently 150 in 2021.However, the number of training posts is rising too, with 11 available in 2021 and 13 in 2022.Oral surgery consistently remains among the top three specialities in terms of available training post numbers each year.