SEARCH

SEARCH BY CITATION

Editor – We enjoyed the review by Patterson et al.,1 who hold that it is feasible to construct situational judgement tests (SJTs) that are not susceptible to coaching. With the introduction of SJTs into UK Foundation Programme (UKFP) selection, candidates and commercial organisations are likely to seek ways to improve scores. This issue of coaching is likely to be controversial in this high-stakes selection process.

Questions in SJTs relate to professional behaviour that is underpinned by values encapsulated in guidelines such as the General Medical Council’s ‘Good Medical Practice’.2 One could argue that formal teaching of these is beneficial, especially for students who fail to passively acquire such skills through clinical work or lack an inherent sense of professionalism. Indeed, medical schools deliberately provide instruction in these values in the belief that they can be taught.

The assumption that a suitably designed SJT is not amenable to coaching benefits also gives rise to uncertainty surrounding which of the attributes of a good doctor are actually being tested. If these attributes cannot be learned, then they presumably reflect ‘personality’ traits. Such assessments may still be valid; certain personalities may lead to ‘better’ doctors. However, candidates with ‘less suitable’ personalities may still have sufficiently good insight into how they ‘should’ behave to enable them to answer correctly, even though their answers may deviate from what they would actually do in reality. Notably, the SJT for the UKFP specifically uses the word ‘should’ in its questions.

There are limited data on the effectiveness of formal coaching for SJTs. Cullen et al.3 demonstrated that one SJT was susceptible to the effects of training, whereas another SJT, which employed more difficult strategies, was not sensitive to coaching. By contrast, a very recent study from Belgium demonstrated a 0.5 standard deviation improvement in SJT scores in a high-stakes setting.4 This latter study might be deemed more relevant as the cohort studied were applying for medical school: although the sample was not randomised, applicants who were coached showed a clear improvement in their scores.

A test design which is not amenable to formal coaching has the appeal of ensuring ‘fairness’ between candidates, especially as the cost of training schemes may distort access to such schemes and medical schools are likely to differ in the level of support they provide. Despite this, the existing evidence suggests that SJTs may well be amenable to coaching.

Situational judgement tests may be preferable to the previous ‘white space’ questions used in UKFP selection, but coaching is a confounding variable that requires further consideration and research.

References

  1. Top of page
  2. References