Evaluating decision aids – where next?
Medical Care Research Unit
School of Health and Related Research
University of Sheffield
30 Regent Street
Decision aids have been developed to help patients become involved in decision-making about their individual health care. During the evaluation of a particular decision aid in maternity care – a set of 10 ‘Informed Choice’ leaflets – we considered the lessons learnt for evaluation of decision aids in the future. Decision aids have been tested mainly in explanatory trials and have been found to be effective. We argue that existing decision aids should be subjected to more pragmatic trials to test their effectiveness in the real world. The small amount of evidence on their use in the real world shows that they face challenges, resulting in poor implementation. Therefore, we propose that implementation strategies are developed which take heed of the findings of research on getting evidence into practice, and in particular address structural barriers such as the lack of time available to health professionals. We recommend that these ‘decision aid implementation packages’ are developed in conjunction with both health professionals and patients, and identify and address potential barriers to both the delivery of patient involvement in decision-making, and the use of decision aids, in the real world. These ‘packages’ can then be submitted to pragmatic evaluation.
There is a growing expectation for patients to be involved in decision-making about their individual health care. Decision aids, designed to help people make specific and deliberative choices, by providing information on the options available and outcomes relevant to their health, have been developed to help this process.1 They come in many formats including decision boards, videotapes, computer programs and booklets, and have been shown to improve knowledge and reduce decisional conflict.1–3
Between 1998 and 2000 we undertook an evaluation of a decision aid – a set of 10 Informed Choice leaflets produced by the Midwives’ Resource and Information Service and the NHS Centre for Reviews and Dissemination.4 We evaluated the use of these leaflets in everyday practice in maternity care. Maternity units were randomized to either receive or not receive the leaflets, with managers and health professionals in each maternity unit deciding how best to use the leaflets after an initial training session. The primary outcome measure was the change in the proportion of women indicating that they had exercised informed choice in their maternity care, reported through a postal questionnaire of users of the maternity units before and after the availability of the leaflets. We found that only three-quarters of women reported receiving any of the leaflets and that the leaflets were not effective in promoting informed choice. An accompanying qualitative study identified the prevailing culture of ‘informed compliance’ rather than ‘informed choice’ in maternity services as a barrier to their effectiveness.5 When interpreting the results of our trial, we considered the lessons learnt for evaluation of decision aids in the future.
Undertaking more trials of decision aids in everyday practice
Our trial produced a negative result, with little evidence that these decision aids improved women's knowledge or their perception that they had made informed choices. At first, we felt that we had found the opposite of the current evidence that decision aids can be effective in improving knowledge, reducing decisional conflict and increasing participation in decision-making.1– 3 However, on reflection we realized that we produced evidence where little existed, because we had tested a decision aid in the real world rather than under experimental conditions. That is, our trial had been pragmatic where others had been explanatory.
Explanatory trials show that decision aids can work and pragmatic trials show whether they work in the real world. In explanatory trials, research nurses may deliver the decision aid outside routine care,6, 7 or a small number of committed or enthusiastic health professionals may deliver it within routine care. In pragmatic trials, the implementation of the decision aid is in the hands of a larger number of health professionals, with a range of levels of commitment to the intervention, and it is delivered within routine care. In our trial, hundreds of midwives working in 11 maternity units in Wales delivered the decision aids to thousands of women. We have only identified one other pragmatic study of a decision aid,8 involving 34 health professionals in three hospitals. Both pragmatic studies found problems with the implementation of the decision aids, in particular with health professionals not making use of the decision aids although they reported that they approved of them.
There is a need for both explanatory and pragmatic trials of decision aids, and it would be helpful if researchers were more explicit in the future about the type of trial undertaken to ensure that findings are correctly interpreted. There is a gap in the knowledge about how decision aids fare in the real world and we suggest that they are subjected to more pragmatic evaluation in the future. However, for decision aids to have a chance to succeed in such trials, we argue that there is a need to consider first the barriers and challenges to their implementation in the real world.
Learning from the implementation of evidence
In our trial, although health professionals initially expressed positive views about the Informed Choice leaflets, pragmatic usage resulted in leaflets being withheld from women. The reasons for this included scenarios in which some midwives did not agree with the content of the leaflets, or some choices were not available locally, or where midwives held assumptions about women's inability to participate in decision-making.5 In addition, some technological interventions such as ultrasound scans and foetal heart monitoring were considered to be so much a part of routine care that health professionals did not perceive a choice to be made on these issues.
Decision aids are essentially evidence-based interventions. Evidence-based guidelines have been evaluated in everyday practice, with a pragmatic trial in general practice finding remarkably similar results to our study, that is, low usage of the intervention and no effect.9 It is by no means always the case that pragmatic trials of evidence-based interventions yield negative results, for example, guidelines have brought about modest improvements in referrals for infertility.10 However, concerns about the implementation of evidence in practice have led to a considerable amount of research around barriers to implementation. For example, compliance with clinical guidelines in general practice has been shown to be 61%, and considerably lower for controversial recommendations, vague recommendations and recommendations that demand a change in existing practice routines.11 The lack of implementation of the evidence-based leaflets in our trial seems unsurprising in this context when many maternity units offer ultrasound scans as part of routine practice and would need to change their systems to accommodate the presentation of ultrasound scans as a choice. Considering the findings of research undertaken within the wider context of the implementation of evidence in the health service may help us to develop strategies around the implementation of decision aids in the real world.12, 13
Paying more attention to health professionals’ experiences
In our trial, health professionals were key to the delivery of the decision aids and to informed choice itself.5 The significance of health professionals in delivering patient involvement in decision-making is well recognized.14–17 They take an influential role in clinical decision-making,18 for example, the effect of the midwife was the biggest predictor of uptake of an antenatal HIV test.19 Research on getting evidence into practice has shown that health professionals may need to change long-held patterns of behaviour. Resistance to change can come from a number of sources including a lack of belief in the evidence, a desire to apply individualized rather than standard care to patients, being too stressed to consider change, and persistence to the status quo.12 Concerns about the evidence, and maintenance of the status quo, were issues identified as barriers to the use of the Informed Choice leaflets in our trial, and indeed to informed choice.5 Consideration may therefore need to be given to devising a strategy around the implementation of any decision aid which helps health professionals to address their concerns about the evidence20 and how changes might occur in practice. This strategy might include training to equip health professionals with the competencies required for delivering shared decision-making,15,16 but should not neglect the structural issues facing them in everyday practice.
Paying attention to structural barriers – time and organizational issues
In our trial, we had expected midwives to discuss an issue with a woman with reference to the relevant leaflet, give the leaflet to the woman to take away, and return later to discuss any concerns. In practice, midwives tended to hand out the Informed Choice leaflets in a batch and ask women to take them away and read them.5 Lack of time was cited as a barrier to involving women in decision-making and the delivery of the decision aids, and midwives even viewed the leaflets as a solution to time pressures by using them as a substitute rather than a focus for discussion. The pressure of work, and time as a scarce commodity for health professionals, has been cited as a problem preventing the use of other decision aids8 and the implementation of shared decision-making.14, 16, 21 The accommodation into routine practice of decision aids which may take 20 min to deliver appropriately7 may seem impossible to many health professionals. A strategy around the delivery of a decision aid, and any ensuing decision-making process, will need to address how health professionals might best accommodate these within the time available to them. This may require longer consultation times,14 extra staffing, or the sacrifice of other duties in order to accommodate the new process.
The midwives in our study found a way of delivering the decision aid that did not disrupt the organization of their service by handing them out in batches alongside information materials which tended to render the leaflets invisible.5 In another study, the decision aid was not used in practice because it was offered in a building separate from where patients were treated, and did not fit in with the timing of test results and treatments.8 The use of computerized evidence-based guidelines in general practice has met similar problems, with the system not being used because it failed to fit into the general practice context.22 Therefore it is important to consider structural issues such as the user-friendliness of decision aids for health professionals and health services, and the amount of effort they have to make to use these aids. It may be possible to identify ways for decision aids to fit into the working context of a health service, and even to simplify the tasks that health professionals currently deliver. Alternatively, it may be that services need to be re-engineered to accommodate decision aids.8
Using active and multifaceted interventions
In our trial we used the dissemination strategy of the developers of the Informed Choice leaflets – a training session for health professionals in how to use them and a supply of leaflets for health professionals and women. However, the evidence around changing health professionals’ behaviour suggests that passive approaches are generally ineffective, with active approaches more likely to be successful.13 Indeed, an active intervention of an interview to help women to sort out their preferences for treatment has been found to be better than passive dissemination of a decision aid.7 Evaluations of other interventions delivered in routine practice have met with problems around implementation and effectiveness, resulting in a recommendation that more intensive interventions are needed, possibly delivered by dedicated staff.23 Additionally, multifaceted interventions, based on an assessment of potential barriers to change, have been shown to be more effective than single interventions.13 Therefore it seems that any implementation strategy for decision aids would need to be active and multifaceted, although the latest evidence introduces some uncertainty around this.24
Developing a ‘decision aid implementation package’
In the light of the uncertainty around the most effective implementation strategies, we propose that ‘decision aid implementation packages’ are developed. Such packages might include identifying the barriers and facilitators for offering choice in the context of the decision under consideration, the extent to which health professionals contest the evidence presented within the decision aid, how the aid might be accommodated within routine care, the need for reorganization of services to accommodate the aid and involvement in decision-making, monitoring of patients making decisions which health professionals think are risky, and the provision of training in some clinical techniques. Given the partnership required for decision-making in health care, the development of such packages would need to involve both health professionals and patients.
Concerns have already been expressed about the costs of developing decision aids and the need to look beyond whether decision aids have value to whether that value is important enough to warrant their development at the expense of other endeavours.3 Active dissemination and multifaceted interventions to change health professionals’ behaviour are likely to be costly.13 Packages will have to be developed without knowledge of whether they are more active or multifaceted than necessary. Therefore it will be important to consider the cost of the development of such packages and the necessity of the different components within them. Further, evaluations may need to consider the overall balance of cost and benefit within the health care system of increasing patient involvement in decision-making. The knowledge that decision aids may result in a reduction in demand for a procedure at no risk to health status,7 and thus potentially in reductions in workload or waiting lists, may give health professionals the motivation to use them.
Evaluations of decision aids to date have been invaluable. They have led to better understanding of decision-making processes in health care and of the role of decision aids in affecting those processes. However, to date, decision aids have been tested mainly in explanatory trials. The small amount of evidence available on decision aids in everyday practice shows that they face challenges in the real world.5,8 Evidence-informed patient choice is a complex concept to deliver in public health care systems25 and it seems overambitious to expect what are essentially pieces of paper and videotape to deliver it alone. The key to obtaining evidence-based patient involvement in decision-making in the real world may not lie in decision aids themselves, but in addressing the culture of health care and the incentives to health professionals to move towards patient involvement. A decision aid can act as a focus within a ‘decision aid implementation package’ that helps health professionals to address barriers to implementing patient involvement in decision-making, in particular the structural barrier of availability of time. Such packages may be costly to develop and deliver, and thus economic evaluations will be needed. We recommend that future evaluations of decision aids should focus on developing ‘decision aid implementation packages’ around existing decision aids and submitting these to pragmatic trials.