- Top of page
- Supporting Information
Objectives In addition to the extrinsic effects of assessment and examinations on students’ study habits, testing can have an intrinsic effect on the memory of studied material. Whether this testing effect also applies to skills learning is not known. However, this is especially interesting in view of the need to maximise learning outcomes from costly simulation-based courses. This study was conducted to determine whether testing as the final activity in a skills course increases learning outcome compared with an equal amount of time spent practising the skill.
Methods We carried out a prospective, controlled, randomised, single-blind, post-test-only intervention study, preceded by a similar pre- and post-test pilot study in order to make a power calculation. A total of 140 medical students participating in a mandatory 4-hour in-hospital resuscitation course in the seventh semester were randomised to either the intervention or control group and were invited to participate in an assessment of learning outcome. The intervention course included 3.5 hours of instruction and training followed by 30 minutes of testing. The control course included 4 hours of instruction and training. Participant learning outcomes were assessed 2 weeks after the course in a simulated scenario using a checklist. Total assessment scores were compared between the two groups.
Results Overall, 81 of the 140 students volunteered to participate. Learning outcomes were significantly higher in the intervention group (n = 41; mean score 82.8%, 95% confidence interval [CI] 79.4–86.2) compared with the control group (n = 40; mean score 73.3%, 95% CI 70.5–76.1) (P < 0.001). Effect size was 0.93.
Conclusions Testing as a final activity in a resuscitation skills course for medical students increases learning outcome compared with spending an equal amount of time practising the skills.
- Top of page
- Supporting Information
It is a general assumption that ‘assessment drives learning’ through its format, content and programming.1,2 This is generally believed to pertain to its influence on students’ learning strategies.3 In addition, assessments that include feedback may induce learning from, for example, an objective structured clinical examination (OSCE).4 However, as well as these extrinsic effects of assessment and examinations on students’ learning, it has been demonstrated that testing can have an intrinsic effect on the memory of studied materials.5
In a recent review of studies on the intrinsic effect of testing, Roediger and Karpicke5 provided evidence that testing students on studied material results in improved retention of that material compared with spending an equivalent amount of time restudying the material. This so-called testing effect has, in both laboratory and classroom studies, been demonstrated to be a robust and independent phenomenon that applies to a variety of test formats and levels of knowledge learning.5 Wheeler et al. described two strengths involved in the act of remembering: storage strength and retrieval strength. Storage is the process induced by study sessions and retrieval is induced by testing.6 Studies converge on naming repeated retrieval as the key to the testing effect.5,7,8 The more elaborative the retrieval process is, the better the effect of testing.9
Consequently, including testing as part of a course might be an efficient strategy to improve learning outcome. This is especially relevant for courses on topics that cannot be left to individual study, as is the case for a variety of procedural skills. We searched the extensive literature on skills learning and assessment and found that whether the intrinsic testing effect applies to skills learning had never been thoroughly investigated.
Simulation-based skills courses are used to train people in procedures that are difficult to learn in real-life settings.10 The problem of poor retention of learning outcome following simulation-based courses has been demonstrated repeatedly.11,12 The risk of decay of learning outcome is especially augmented when opportunities to practise the learned skills after the course are limited, as is the case with in-hospital resuscitation and safe defibrillation. Because of the poor retention of in-hospital resuscitation skills and the unremitting need for health professionals to be able to act appropriately in emergencies, it is recommended that in-hospital resuscitation courses are repeated frequently.13,14
In general, simulation-based skills courses are fairly expensive15 and are confined to a limited amount of time. Thus, it is important to determine whether learning outcomes are better if some of this time is spent on testing rather than on more training. Although it has been speculated that testing might have only limited effect on skills learning,16 we found no systematic studies enquiring into whether the impact of the intrinsic testing effect on knowledge retention also pertains to skills learning. A major review by Rosenbaum et al. found that ‘intellectual and perceptual motor skills are acquired in fundamentally similar ways’ in multiple areas like feedback, massed or blocked learning and, most interestingly in view of the area explored in this paper, transfer and retention.17 A study by Shin and Rosenbaum suggested that motor skills and intellectual skills are co-ordinated in the same manner.18
Seen in the light of Roediger and Karpicke’s claim – that the testing effect is applicable on all grades and in all settings5– these studies lead us to the suggestion that simulation-based teaching might benefit from the integration of testing as a final activity.
The aim of this study was to examine the testing effect related to skills learning in the context of a simulation-based, in-hospital resuscitation skills course for medical students. The research question was: does testing as a final activity, in an in-hospital resuscitation skills course, increase learning outcome compared with an equal amount of time spent practising?
- Top of page
- Supporting Information
This study suggests that testing as a final activity in a resuscitation skills course increases learning outcome compared with an equal amount of time spent in practice. The mean score of the intervention group was superior to that of the control group, at 82.8% compared with 73.3%, yielding an ES of 0.93.
Bangert-Drowns and Kulik’s 1991 review reports that ESs in testing effect studies are generally moderate to high,22 and Roediger and Karpicke found an ES of 0.82 in knowledge learning in a recent study.23 Accordingly, our study on skills learning demonstrating an ES of 0.93 corresponds well with prior studies on the testing effect and indicates that the testing effect can be reproduced in skills learning.
Isolating the intrinsic testing effect on skills learning in a classroom setting represents some challenges. In the context of the in-hospital resuscitation course, several extrinsic effects of testing might confound the intrinsic testing effect, including: better preparation by students; greater familiarity with the test scenarios; observational learning, and intention to learn. We made great effort to eliminate these extrinsic effects of testing at the point of study design.
Preparation by the participants
As the participants in the intervention group did not know they were to be tested before the start of the course, it is improbable that there was a difference between the groups with regard to the amount of preparation carried out for the course. Moreover, both groups had equal opportunity to prepare for the final assessment 2 weeks later. However, this opportunity was, in reality, minimal. None of them were offered any opportunity to practise on our simulators.
To avoid bias caused by a large assessment time-span, we scheduled only 2 days for the final outcome assessment. On each of these days we assessed 40 participants, from 0800 h to 2000 h. If participants contacted us to cancel, we offered them the opportunity to be assessed on the day before that scheduled. Thus seven of the 81 participants were assessed 1 day before the others. The primary reason for not participating in the assessment was that the students had other appointments (e.g. classes, work etc.) on assessment days. This explains the 40% drop-out rate. We are aware that using volunteer samples results in a positive selection bias (i.e. volunteers often outperform others).24 However, the two groups in our study were comparable regarding volunteer bias. Furthermore, the ES in our study is of the same magnitude as those of Roediger and Karpicke in studies that did not use volunteer samples.23 Thus, we find it unlikely that volunteer bias threatens our results.
The assessments were completed during 1 day for each of the two rotations in order to limit the extent to which participants conferred with one another. Furthermore, in order to limit the bias arising from conferral among participants, all participants were told about the assessment set-up and content at the time of signing the informed consent. Thus all participants knew that they were to be assessed in a simple cardiac arrest scenario which would include a demonstration of their resuscitation skills and their ability to adhere to the algorithm.
The same six basic scenarios were used repeatedly in both courses, ensuring that both groups were thoroughly familiarised with all scenarios. Furthermore, the scenario used for the assessment 2 weeks after the course had been used by both the intervention and control groups as part of the course. Thus we consider both groups to have been equally familiar with the test scenarios.
The final half-hour of the courses differed with regard to the relative weighting of observational learning and practical involvement. Studies have shown that watching the performance of procedures can have a reinforcing effect on practical training.25,26 However, the final half-hour included observational learning and practical involvement in both groups and thus we find that if observational learning has influenced our results, it has been to a limited extent.
Intention to learn
It has been hypothesised that the expectation of later testing stimulates intention to learn, leading to higher learning outcomes.27 However, a 1984 review by Crocker and Dickinson28 claims that intentional learning is only beneficial to learning outcome if it prompts rehearsal in the subjects. As our course was very intensive, none of the participants had any opportunity for additional rehearsal and therefore we consider intention to learn to be of limited importance to our results.
Furthermore, participants were excluded if they had prepared for the assessment in the time between the course and the assessment of learning outcome. We consider this policy to have minimised the effect of intention to learn.
Ideally, educational interventional research includes a pre-test. However, in this study examining the effect of testing, pre-testing would have interfered with the intervention and possibly induced an effect on the control group. In our pilot study we used a pre- and post-test design and demonstrated rather low and homogeneous pre-test levels (mean score 19.7%, 95% CI 15.4–24.0). We therefore felt confident in choosing a post-test-only design for the main study in order to prevent bias induced by pre-testing.
All assessments in this study were performed by one assessor using only one test scenario on the assumption that using one assessor only would result in more consistent assessment practice compared with using different assessors. In a recent study, we found high inter-rater and inter-case reliability using checklists to assess resuscitation performance.29 These high reliabilities probably reflect the narrow and homogeneous content area involved in assessing resuscitation skills and the highly standardised performance indicators in the checklists. The checklists and scenarios for the current study were similar to those used in the prior study29 and hence we consider that reliability of the assessments did not threaten the results.
Although results from this study indicate a testing effect applied to skills learning, additional studies regarding the generalisability of the results are needed. This includes reproducing the testing effect on a wider range and complexity of skills, as well as conducting similar studies in different settings and on a wider range of study populations.30 Moreover, there is a need for future studies with longer follow-up periods to estimate the effect on long-term retention of learning outcome.
This study demonstrates that it is quite feasible to implement testing as a final activity in small-group, simulation-based training, even in a rather short programme (i.e. 4 hours). Any provider of medical education seeks to establish the most efficient use of its simulation facilities in order to get the most out of its costly investment. In this context the enhanced learning outcome of testing could prove valuable for all skills laboratories. The testing effect seems to be especially relevant to in-hospital resuscitation training because retention of rarely used skills is known to be poor.31
Acknowledgements: we wish to thank Deborah Davis for editorial assistance and the following skilled instructors at the Centre for Clinical Education for willingly implementing our study in their daily work: A Kyhnel, A Gustafsson, A B Hasselager, C Bohnstedt, E L Bessmann, H Spielberg, J Sommer, J W Egholm, L R Wahlstén, L I Hennings, M Engdahl, M G Tolsgaard, M Koefoed, M J V Henriksen, M J Andersen, P P Höiby, S V Bjørck, M B Rasmussen, S S Mogensen, I Bostadløkken and M H Larsen. None of these individuals were compensated for their role in the study. We also acknowledge the Centre for Clinical Education, Rigshospitalet, Copenhagen University Hospital, Copenhagen for administrative, technical and material support.