ACADEMIC EMERGENCY MEDICINE 2010; 17:859–864 © 2010 by the Society for Academic Emergency Medicine
Objectives: Studies have suggested that poor knowledge of the Ottawa Ankle Rules (OAR) limits its clinical impact. This study evaluated the ability of a mnemonic to improve knowledge of the OAR.
Methods: This was a single-blind randomized controlled trial performed among residents and medical students doing a pediatric emergency medicine rotation. At baseline, all participants were tested for their baseline knowledge of the OAR. The intervention was a standardized information sheet providing a mnemonic of the OAR (44-55-66-PM), while control subjects received its classic description. Block randomization (medical student vs. type of resident) was used. Each participant answered the same questionnaire at the end of rotation (3 weeks later) and via a Web-based survey 5 to 9 months postrandomization. Main outcome measures were knowledge of the components of the ankle rule based on a 13-item criterion grid and the foot rule based on a 10-item criterion grid. All questionnaires were marked at the end of the study by two reviewers blinded to the randomization. Discrepancies in final scores were resolved by consensus. Student’s t-test was performed to compare mean scores on the evaluation between groups using an intention-to-treat approach.
Results: Among the 206 eligible participants, 96 medical students and 94 residents were recruited and agreed to participate. Primary outcomes were measured in 95% of the participants at 3 weeks postrandomization and in 72% on the long-term follow-up. Participants in both groups were similar with regard to baseline characteristics and prior knowledge of the OAR. Both groups showed improvement in their knowledge of the rule during the study period. At mid-term, knowledge of the OAR was similar for the ankle components (score for mnemonic 10.9; control 10.2; 95% confidence interval [CI] for difference = −0.3 to 1.7) and for the foot (mnemonic 7.6 vs. control 7.5; 95% CI for difference = −0.7 to 0.9). On the long term, randomization to the mnemonic was associated with a better knowledge of the OAR as demonstrated by a higher score for the ankle component (mnemonic 10.1 vs. control 8.9; 95% CI for difference = 0.6 to 1.8) and for the foot (mnemonic 7.8 vs. control 6.5; 95% CI for difference = 0.8 to 1.9).
Conclusions: Mid-term knowledge of the OAR drastically improved for all participants of the study. The use of the mnemonic 44-55-66-PM was associated with a better long-term knowledge of the OAR among medical students and residents. The improvement in knowledge of the OAR among the control group highlights the importance of using controlled trials for studies evaluating knowledge transfer.
The Ottawa Ankle Rules (OAR) have been thoroughly evaluated in the adult and pediatric population.1–5 A systematic review published in 2003 by Bachmann et al.6 that included a total of 27 studies (total number of patients exceeding 15,000) found that the OAR was 100% sensitive in predicting which patients require ankle radiography. Studies have shown that use of clinical decision rules could decrease the number of ankle x-rays by 30% to 40% in clinical studies.6
While many studies reported an important reduction in the rate of radiographic studies performed with the use of the OAR,7 there are discrepancies between the results expected from clinical studies and the impact reported in clinical settings. In 1999, Cameron and Naylor8 reported no effect of an educational intervention to promote the use of the OAR in 63 hospitals across Ontario, Canada. In 2004, Holroyd et al.9 reported that multiple dissemination strategies for the OAR failed to reduce extremity radiography uses.
The reason for disagreement between clinical studies and real-life situations may be related to the fact that physicians do not adequately remember the OAR. Recently, Brehaut et al.10 reported a survey performed among Canadian emergency physicians to evaluate barriers to the implementation of clinical decision rules. While 99.2% of the respondents reported to be familiar with the OAR, only 30.9% were able to correctly remember all its components. Their main conclusion was that, “future work should study how different memory aid strategies might improve the accuracy of rule application and reduce the use of non-predictive cues.”10 The primary objective of this study was to evaluate the ability of a mnemonic to improve the retention of the OAR among residents and medical students. The study was funded by Sainte-Justine Hospital Research Institute. The funding source had no involvement in the study design, conduct, or decision regarding publication, and researchers were completely independent from the funding source.
This was a single-blind randomized controlled trial to evaluate whether the use of a mnemonic would improve mid-term and long-term knowledge of the OAR. It was registered with the National Institutes of Health (ClinicalTrials.gov identifier NCT00652353) and was approved by the institutional review board of our institution. Participants provided informed consent before enrolment. At enrolment, they were informed that they would be evaluated at recruitment and again in the following weeks.
Study Setting and Population
The study was performed in a pediatric emergency department (ED) located in an urban, tertiary care, Level 1 trauma center. The ED is staffed with full-time accredited pediatric emergency physicians, general pediatricians, and general practitioners. Study recruitment occurred during 8 months in 2008, and final evaluation was completed in May 2009.
Eligible participants were residents doing a rotation in pediatric emergency medicine, and all medical students doing their rotation in general pediatrics during the study period. They were part of the pediatrics, family medicine, or emergency medicine training programs.
Participants were recruited on their first week of rotation. The study was done in four steps:
- 1Participants filled out a questionnaire measuring baseline characteristics and preintervention knowledge regarding the OAR.
- 2After completion of the baseline evaluation, participants received an opaque envelope containing either a mnemonic to help remember the components of the OAR or a simple description of it.
- 3Evaluation of the mid-term knowledge of the OAR was performed 3 weeks after intervention. To do so, participants filled out another paper questionnaire asking them to name all components of the OAR.
- 4Long-term evaluation was performed 5 to 9 months following randomization. To do so, participants filled out a short electronic survey asking them to describe all components of the OAR. A maximum of three electronic reminders were sent to each participant.
The intervention was a standardized information sheet providing both a mnemonic (developed by a member of the research team and shown in Figure 1) and the standard description of the OAR itself. The control group received only the standard description of the OAR. In both groups, no other formal training regarding the OAR was performed by the study members. With the exception of the three investigators, none of the ED staff members knew the mnemonic. All physicians were instructed not to modify their regular bedside teaching regarding ankle rules.
Block randomization using a random-digit table was performed by someone not otherwise involved in the study. Randomization was stratified according to the type and training level (pediatrics, family medicine, emergency medicine, medical student). Opaque envelopes were provided to ensure blinding of the investigators and evaluators.
All participants were asked not to discuss the project with each other until the end of the study. Each questionnaire was corrected by two reviewers blinded to the randomization. Finally, to evaluate potential contamination, all participants were asked in Phase III of the study (during the short-term evaluation performed 3 weeks postrandomization) to indicate on a separate document if they knew a mnemonic to remember the OAR.
The primary outcome measures were the knowledge of the Ottawa Ankle and Foot Rules at 3 weeks postrandomization and at long-term follow-up (5 to 9 months postrandomization). Knowledge of the ankle components of the OAR was based on a 13-item criterion grid, while the foot components were based on a 10-item criterion grid.
A secondary outcome was the proportion of participants who knew all the components of the rule (perfect score). Secondary analyses of the data for the medical students only, and for the residents, were performed. Finally, efficacy of the mnemonic was measured for each type of resident.
Baseline characteristics and outcomes were evaluated through written questionnaires performed before intervention and 3 weeks and 5 to 9 months after intervention. Before intervention, all participants were asked to report their previous knowledge of the OAR on a visual analog scale going from 0 mm (I never heard about the OAR) to 100 mm (I could apply it easily without help). All members of the research team participated in the questionnaires’ marking at the end of study recruitment. Each questionnaire was marked separately by two reviewers blinded to the randomization arm and blinded to each other. Discrepancies in scores were resolved by consensus. Knowledge of the OAR was evaluated by asking participants to name all components of the OAR for 1) an ankle trauma and 2) a foot trauma. The criterion grid for marking was developed from a consensus agreement by the three investigators. It was a simple summation of all the components of the OAR. It was, however, never validated. The criterion grid for marking was the following:
Ankle Components of the OAR: To be called perfect, 13 items had to be present and no incorrect items (0.5 point was subtracted for each incorrect supplementary item):
- • tenderness in 6 cm posterior edge of lateral malleolus (4 points)
- • tenderness in 6 cm posterior edge of medial malleolus (4 points)
- • inability to bear weight (1 point)
- • immediately (1 point)
- • 4 steps in the ED (2 points)
- • and (1 point).
Foot Components of the OAR: To be called perfect, 10 items had to be present and no other incorrect items (0.5 point was subtracted for each incorrect supplementary item):
- • tenderness at the base of the fifth metatarsal (3 points)
- • tenderness at the navicular/scaphoid bone (2 points)
- • inability to bear weight (1 point)
- • immediately (1 point)
- • 4 steps in the ED (2 points)
- • and (1 point).
All data were entered in an Excel database (Microsoft Corp., Richmond, WA) and analyzed with SPSS v.16 (SPSS Inc, Chicago, IL). The 95% confidence interval (CI) was measured for every result.
A pilot study performed in our ED in November 2007 showed that none of the 10 residents evaluated knew the OAR perfectly. We expected to see an improvement in the perfect retention of the OAR in both groups, just from participating in a study. We thus aimed to demonstrate a greater improvement of this proportion for the intervention group. A sample size of 50 participants in each arm of treatment provided a power of 80% to demonstrate a difference of 20% in the proportion of participants who knew the rule perfectly with an α error of 0.05.
The scores for the ankle component (max 13/13) and for the foot component (max 10/10) were calculated for each participant on pre- and postintervention at mid-term and long-term follow-up. A Student’s t-test was performed to compare mean scores for the ankle components between intervention and control. A Student’s t-test was also performed to compare mean scores for the foot components between intervention and control. The difference in the proportion of perfect answers was compared using chi-square. An intention-to-treat approach was used with exclusion of those subjects lost to follow-up. The interrater agreement between markers was evaluated through intraclass correlation using a two-way random model with measures of absolute agreement.
Contamination between participants was evaluated by asking them to specify whether they had a specific mnemonic tool to help them remember the rule. Answers to this question were evaluated by a reviewer blinded to the randomization and to the outcomes.
Another secondary analysis evaluated potential selection bias on the long-term evaluation. To do so, we compared long-term participants versus nonparticipants on their scores at 3 weeks postrandomization.
During the study period, 106 residents performed a rotation in the pediatric ED. Among them, 95 were approached and 94 agreed to participate. One-hundred medical students were eligible for the study during the same period and 96 agreed to participate. Figure 2 describes the flow of participants and nonparticipants. The primary outcome was measured in 181 (95%) participants at 3 weeks postrandomization (mid-term knowledge). The only reason for missing data at that phase was because the participants were absent during evaluation either for vacation or because of sickness. Long-term knowledge of the OAR was evaluated among 138 participants (response proportion, 72.6%; Figure 2). The mean (± standard deviation [SD]) number of days between randomization and final evaluation was of 231 (±42) days for the intervention versus 241 (±52) days for the control group. The main reasons for failure to complete the survey were because we did not have the proper e-mail address or there was no answer after three reminders.
The baseline characteristics of study participants are described in Table 1. Participants in both groups were similar with regard to baseline characteristics and prior knowledge of the OAR.
|Characteristics||Mnemonic, % (n = 95)||Control, % (n = 95)|
|Sex (male)||23 (0.24)||21 (0.22)|
|Number of medical students||47 (0.49)||49 (0.51)|
|Resident specialty (total)||48 (0.51)||46 (0.48)|
|Previous knowledge of the OAR (ankle aspects) on the VAS (mm), mean (95% CI)||40.0 (23.4–46.6)||42.9 (35.5–50.3)|
|Previous knowledge of the OAR (foot aspects) on the VAS (mm), mean (95% CI)||26.8 (21.2–32.4)||31.3 (26.4–38.1)|
|Score for the ankle component before intervention (of 13), mean (95% CI)||3.5 (2.8–4.2)||3.8 (3.0–4.6)|
|Score for the foot component before intervention (of 10), mean (95% CI)||2.3 (1.7–2.9)||2.4 (1.8–3.0)|
|Number of perfect answers for the ankle component before intervention||0 (0.01)||1 (0.01)|
|Number of perfect answer for the foot component before intervention||0||2 (0.02)|
Agreement between raters was excellent, as demonstrated by intraclass correlation coefficients varying from 0.93 to 0.95 (Table 2). Both groups showed an improvement in their knowledge of the rule during the study period. This is demonstrated by the improvement in the total scores and for the proportion of perfect knowledge of the rule for both groups (Tables 1 and 3). However, after 3 weeks, knowledge of the OAR was similar among the two groups for the ankle components (score for mnemonic 10.9; control 10.2; 95% CI for difference = −0.3 to 1.7) and for the foot components (mnemonic 7.6 vs. control 7.5; 95% CI for difference = −0.7 to 0.9). The proportion of participants with perfect knowledge of the rule was similar among those randomized to the mnemonic and the control group (0.19 vs. 0.17; 95% CI for the difference 0.09 in favor of the control to 0.13 in favor of the mnemonic). Secondary analysis performed on the subgroups of medical students or residents failed to disclose a significant difference between the mnemonic and the control group.
|Ankle components preintervention||0.979||0.972–0.984|
|Foot components preintervention||0.931||0.908–0.948|
|Ankle components postintervention||0.947||0.928–0.961|
|Foot components postintervention||0.946||0.927–0.960|
|Mnemonic (n = 89),|
Mean (95% CI)
|Control (n = 92),|
Mean (95% CI)
|Knowledge of the OAR on the VAS ankle component (mm)||77.9 (73.3 to 82.6)||72.6 (68.0 to 77.2)||5.3 (–1.3 to 11.9)||0.115|
|Knowledge of the OAR on the VAS foot component (mm)||75.9 (70.9 to 80.9)||70.6 (66.1 to 75.2)||5.3 (–1.4 to 12.0)||0.127|
|Score for the ankle component (/13)||10.9 (10.3 to 11.6)||10.2 (9.6 to 10.9)||0.7 (–0.3 to 1.7)||0.16|
|Score for the foot component (/10)||7.6 (7.0 to 8.1)||7.5 (6.9 to 8.0)||0.1 (–0.7 to 0.9)||0.801|
|Proportion of perfect answers for all components, n (%)||18 (0.20) (0.12 to 0.28)||16 (0.17) (0.10 to 0.25)||0.03 (–0.09 to 0.14)||0.507|
|Mention of the 44-55-66-PM mnemonic||0.72 (51/71)||0.01 (1/76)||0.71 (0.60 to 0.82)||<0.0001|
The presence of contamination was evaluated among the 147 participants who responded to the question regarding whether they use a mnemonic to improve their knowledge of the OAR. It revealed that 72% of the participants randomized to the intervention remembered, at least in part, the mnemonic while only one participant randomized to the control group mentioned the 44-55-66-PM mnemonic.
A secondary analysis showed that knowledge of the mnemonic was associated with a better knowledge of the OAR (mean total score 19.3 for those who remember at least one item of the mnemonic vs. 17.2 for those who could not mention the mnemonic; 95% CI for the difference = 0.2 to 4.0)
In the long term, randomization to the mnemonic was associated with better knowledge of the OAR, as demonstrated in Table 4. The mnemonic group had a higher mean score for the ankle components, the foot components, and the total mean score. Also, the proportion of participants who could perfectly describe the OAR was three times higher in the mnemonic group. Subanalysis performed on medical students or residents failed to find an association between intervention and most outcomes.
|Mnemonic (n = 70),|
Mean (95% CI)
|Control (n = 68),|
Mean (95% CI)
|Score for the ankle component (/13)||10.1 (9.5–10.6)||8.9 (8.3–9.5)||1.18 (0.57–1.81)||0.039|
|Score for the foot component (/10)||7.8 (7.2–7.3)||6.5 (5.9–7.1)||1.32 (0.78–1.87)||0.004|
|Proportion of perfect answers for all components (n)||0.20 (0.12–0.28) (14)||0.06 (0–0.08) (4)||0.14 (0.03–0.25)||0.012|
To evaluate for a potential selection bias, a secondary analysis demonstrated similar knowledge of the OAR at 3 weeks postrandomization for those who participated and for those who did not participate (mean score for ankle participants 10.7 vs. non participants 10.2, p = 0.36; and mean score for foot participants 7.5 vs. non participants 7.4, p = 0.74).
This study showed a quantifiable improvement in the knowledge of the OAR among residents and medical students participating in a study evaluating the efficacy of a mnemonic. While both groups (intervention and control) showed a similar improvement in their knowledge at 3 weeks postintervention, those randomized to the mnemonic demonstrated a significantly higher score on an assessment of their knowledge of the OAR on the long-term follow-up.
To our knowledge, this is the first study evaluating the usefulness of a mnemonic for teaching and retention of the OARs. As mentioned, the development of such mnemonics could potentially improve the application of the rule in clinical practice.10 Glasziou and Haynes11 and Gaddis et al.12 described seven steps between new medical knowledge development and its application in a clinical setting. They suggested that information can be lost at any step of the “research-to-practice pipeline.” The 44-55-66-PM mnemonic could seal a leak in the pipeline by improving the ability of the physician to apply the OAR (Step 4 of the model).
The proportion of participants randomized to the mnemonic that, on long-term follow-up, knew all components of the OAR at the time of final evaluation may seem low (20%). However, it is comparable to results of a previous study by Brehaut et al.10 that demonstrated that only 30% of Canadian emergency physicians were able to recall all components of the OAR.
The failure to find a significant impact of the mnemonic at mid-term follow-up in our study was mainly due to the improvement in the knowledge of the OAR that also occurred in the control group. Many possibilities can explain this improvement. First, the fact that the participants were asked the same questions 3 weeks apart increases the risk of test–retest bias. Second, participants randomized to the control group received a standardized description of the OAR. These participants have great academic skills and may have remembered the rule for a few weeks. It is probable that most participants specifically studied the OAR since they knew that they would be questioned again before the end of the rotation. Another explanation may be related to the learning gained through the rotation. While this may be true for the residents rotating in pediatric emergency, this would be unlikely for medical students rotating in general pediatrics. A subgroup analysis of medical students randomized to the control group showed a similar improvement in their knowledge of the OAR (data not shown) compared to residents randomized to the control group. Finally, Table 2 shows that the possibility of contamination is low.
Our results also demonstrate the importance of using a control group for the evaluation of strategies to improve clinical practices. Knowledge transfer studies often use a “before-and-after” design where the primary outcome is measured among participants before the intervention and after the intervention.7,13 Recently, Riesenberg et al.14 reported that among 46 articles describing a “handoff” mnemonic, seven described a research project and only four evaluated the mnemonic. Moreover, none of these studies used a randomized controlled design. The use of a before-and-after design for our study, without a control group, would have led one to inappropriately conclude that the mnemonic improves the mid-term knowledge of the OARs.
The fact that participants were asked the same questions 3 weeks apart may have biased results toward better mid-term knowledge of the rule for every participant. This recall effect is, however, less likely for the long-term evaluation.
Long-term evaluation was performed through a Web-based survey that carries some limitations. First, there is a risk of selection bias. However, the fact that long-term participants and nonparticipants had similar results on the mid-term evaluation limits the risk of selection bias. Also, the proportion of missing data was similar among the two groups. Second, participants could have consulted the Web or their notebooks to answer the survey. This information was, however, available to both intervention and control groups and, if used, would bias the results toward a decreased likelihood of concluding a significant difference in the mean score and toward a decreased likelihood of finding a significant difference for the proportion of learners who obtained a perfect long-term score. Another limitation comes from the fact that we only evaluated theoretical knowledge of the OAR. We have not evaluated if participants can apply the rule to real patients. Also, we have not evaluated the clinical impact of the intervention on clinical outcomes, such as the ordering of radiographs. Finally, we do not know what the clinical impact is of a difference of 1.2 in the mean score, because to use the OAR appropriately, one must know all its components. The difference of 14% in the proportion of perfect knowledge of the OAR may be more clinically relevant.
Mid-term knowledge of the Ottawa Ankle Rules improved for all participants of the study. The use of the mnemonic 44-55-66-PM was associated with a better long-term knowledge of the Ottawa Ankle Rules among medical students and residents. This improvement could potentially lead to a better use of the Ottawa Ankle Rules in the clinical setting. Finally, the improvement in knowledge of the Ottawa Ankle Rules among the control group highlights the importance of using controlled trials for studies evaluating knowledge transfer.