Judging competence, providing feedback and changing organisations

In their article "Beyond summative decision-making: illuminating the broader roles of Competence Committees", Pack et al.1 report on a grounded theory study that highlights the range of roles that Canadian Competence Committees (CCs) undertake, through an analysis of data collected at seven postgraduate programmes within one institution. They show how the in-depth data review, synthesis and analysis undertaken to make summative assessment judgments, provided CC members with unique insights and opportunities to identify problems linked to individual learners or programmes and develop practical solutions. Such activities went beyond the committee name or terms of reference but afforded real potential to ensure trainees received the best possible education.

In their article 'Beyond Summative Decision Making: Illuminating the Broader Roles of Competence Committees', Pack et al 1 report on a grounded theory study that highlights the range of roles undertaken by Canadian competence committees (CCs) through an analysis of data collected from seven postgraduate programmes within one institution. They show how the in-depth data review, synthesis and analysis undertaken to make summative assessment judgements provides CC members with unique insights and opportunities to identify problems linked to individual learners or programmes, and to develop practical solutions. Such activities extend beyond the committee's designation or terms of reference but afford real potential to ensure trainees receive the best possible education. In this commentary, we offer a perspective from the UK equivalent of the CC, argue for a greater role for assessment for learning in postgraduate medical education, and explore opportunities to engage more holistically with training organisations, which is ultimately cost-effective and better for everyone.
The annual review of competence progression (ARCP) panel is the UK equivalent of the CC. Such panels review all doctors in training each year to ensure they are providing safe, high-quality patient care and to assess their progression against standards set out in the training programme curriculum. Like CCs, ARCP panels have a summative function in reviewing and validating the doctor's progress. The key difference seems to lie in how the ARCP panel draws upon, and is embedded within, the ongoing supervision process. Each trainee in the UK has an education supervisor with whom they meet regularly to review progress and whose recommendation carries real weight for the ARCP panel judgement. In addition, whereas the CC may hold a greater degree of information on any one individual, the ARCP panel seems to have a more direct and interactive relationship with the trainee. For example, if the ARCP panel issues an unsatisfactory progress recommendation, it meets directly with the trainee to discuss the underpinning reasons and provide feedback to inform subsequent activities.

Feedback is known to be a powerful way to increase individual learning and performance
We believe postgraduate medical education would benefit from greater emphasis on assessment for learning, rather than assessment of learning. 2 Feedback is known to be a powerful way to increase individual learning and performance 3,4 and is typically more effective when baseline performance is low, 4  early-career professionals such as postgraduate medical trainees. 5 Based on feedback intervention theory, 6 the Feedback Framework highlights the points that trainees may be more easily disheartened than senior colleagues by feedback without appropriate support, may not have established support networks and therefore, may need additional help to maximise the benefit to be derived from feedback, may find feedback hard to interpret, and may have dual goals of training and practice, which may be in tension. The Feedback Framework divides the process into three stages: (a) audit (or data collection); (b) feedback, and (c) goal setting. Both CCs and ARCP panels currently place most emphasis on the audit stage, but a greater focus on feedback and goal setting could be beneficial. One innovative approach to feedback and goal setting is the education prescription, 7 which can help to ensure clarity about the goals the trainee needs to achieve. Soon after an ARCP panel, an education prescription may be shared with the education supervisor to allow the supervisor to co-create a remediation plan with the trainee. Early remediation allows the ARCP panel to signal a doctor in difficulty and initiate a more directive programme that can lead to competency progression without a training extension.

Both CCs and ARCP panels currently place most emphasis on the audit stage, but a greater focus on feedback and goal setting could be beneficial
We know that postgraduate medical trainees learn within complex health care organisations. We now need a better understanding of how educational processes might leverage improvements beyond education in this wider health care system. Routinely collected data relating to training experiences and outcomes give an important perspective on the organisational culture of the host institution. The information synthesised by CCs and ARCP panels provides unique insights into the workplace environments in which training and service provision occur, and these operate at multiple levels (eg, individual, team, department, organisation, curriculum, health system).
Such insights could help enable health care leaders to improve clinical services, as well as helping curriculum leaders to improve education programmes. This does not happen systematically at the moment as it is not the stated purpose of CCs or ACRP panels, and therefore the terms of reference, membership and reporting lines are not designed with this in mind. An expanded remit for CCs and ARCP panels might require a broader membership to include health care executives, human resources professionals and education leaders, as well as lay members. In Pack et al's study, 1 the CC members were proactively analysing the rich information collated for their competence judgements in additional ways because they could see how much more they could offer the trainee, department, hospital and curriculum. However, they feared that their ability to keep doing this could be eroded by workload pressures. Current concerns about the recruitment, retention and psychological well-being of trainees in some areas 8 suggest that such multi-level insights are urgently needed to inform change. Archives of the data collected for CCs and ARCP panels might also provide a rich historical resource for medical education researchers, building on successes such as the UK Medical Education Database. 9 One innovative approach to feedback and goal setting is the education prescription … Understanding the cost benefits of CCs or ARCP panels with extended remits will be important to justify their ongoing existence in resource-constrained environments. Such panels typically involve several senior professionals and are therefore expensive to convene.
The data collection process may also be time-consuming and expensive. However, being explicit about the area in which the trainee has not reached the required standard is the first action in planning the steps required to remediate the situation and early intervention can ensure that the trainee is able to continue. Failure to progress can result from exposure to clinical material that is insufficient to gain the required competencies, or poor supervision and guidance, and both CCs and ARCP panels are well placed to see patterns of failure that may reflect the environment rather than the individual trainee.
Timely intervention can save money in costly training programmes and should be better for the trainee. Extensions to training to achieve competencies can impact on a trainee's mental health and potentially displace other trainees awaiting that rotation, and occur in a context in which salary costs must still be met. Justifying the costs of CCs or ARCP panels with extended remits by quantifying their positive impact in financial terms on clinical and education service provision will therefore be important, although methodologically challenging.
Timely intervention can save money in costly training programmes and should be better for the trainee embedded within the wider organisational context. Both CCs and ARCP panels have important roles to play in overseeing and approving the progression of competent trainees, and removing those who are not competent, in order to protect patients. However, assessment for learning needs greater weighting because feedback based on high-quality data can provide significant benefits. Committee titles, membership, terms of reference and reporting lines will probably need to be rethought to reflect this extended role. Further, to justify and maximise the potential of such committees, we will need to grapple with some of the most important and challenging topics for medical education scholarship: evaluating educational impact at a systems level, and education economics.