The ups and downs of EBTs as reflected in Figure 1 should also be interpreted in light of the decades of research on implementation, particularly implementation of tested practices in settings other than those in which they were originally tested (Fixsen et al., 2005). That body of research strongly suggests that any practices - including EBTs - risk a loss of potency and impact when moved into contexts that are new and different from those where they were previously used and tested. Indeed, successful implementation typically requires much more than simply relocating the intervention; instead, multiple steps of intervention adaptation, intervener selection and coaching, and perhaps even recipient and organisational preparation may be required (Fixsen et al., 2005). The modest mean effect size shown in Figure 1 and the fact that a number of EBTs did not outperform usual care might be seen as illustrating the general challenge of implementation. A case can be made that this challenge is particularly serious in cases where the implementation context differs most markedly from the original development and testing context.
This brings us back to a concern noted earlier in this paper, one that our research group has stressed for many years - i.e. that the contexts and conditions under which most EBTs have been developed and tested tend to differ rather markedly from the conditions of everyday clinical care to which treatment developers wish to see their protocols deployed. In other words, the ways most evidence-based psychotherapies for children have been developed and tested do not expose these therapies to the full array of factors present in everyday, real-world clinical care. As a consequence, we have argued, the therapies emerging from such development and testing procedures may not be very robust in the face of real-world treatment conditions, and in fact may not look so strong in relation to the everyday clinical care that takes place under those conditions. This, of course, is quite consistent with the findings of our EBT vs. usual care meta-analysis (Weisz, Jenson-Doss, & Hawley, 2006), as depicted in Figure 1.
Taken together, these concerns and findings raise the question of whether our field needs a new model of treatment development and testing. We certainly believe that empirical testing is critical to the development of beneficial treatments. However, a critical question for the field is what approach to empirical testing will give us the strongest treatments that are most robust in actual clinical practice. Our recent findings lead us to wonder whether the primary model that has guided the development and testing of psychotherapies for decades is well-suited to the task.
The successive efficacy trial model, derived from medical-pharmaceutical research.
Most of the child psychotherapies now identified as evidence-based have been developed and tested via a model quite similar to the one that has guided medical and pharmaceutical research for years. That model - we will call it the ‘successive efficacy trial (SET) model’ - may work reasonably well for biological interventions, but it may not be quite as well-suited to the production of clinic-ready psychotherapies. The SET model involves a stepwise sequence in which experimental treatments and their protocols are first developed in the laboratory, and then tested via an extensive array of ‘efficacy studies’. Efficacy studies use experimental control to test treatment impact under carefully arranged idealised conditions - e.g. with just the right kind of clients selected, often with the most troubling comorbidities excluded, with treatment done by particularly skilled therapists who are selected and paid by the researcher and trained to deliver the target treatment (and that treatment alone) as faithfully as possible, and with arrangements designed to keep therapists functioning at their best and treated clients engaged and attending. Efficacy studies are contrasted with ‘effectiveness studies’, in which intervention effects are assessed under ordinary clinical conditions, with treatment delivered to ‘average’ or representative patients or clients, by ‘average’ or representative practitioners, working under conditions that reflect typical practice realities (e.g. large caseloads, clinic productivity pressures, frequent appointment no-shows). Given the nature of the research conducted to date, we know a good deal about the efficacy of our tested treatments; we know far less about their effectiveness. In the SET model, a sometimes lengthy series of efficacy trials may be conducted, perhaps testing various ways of structuring and delivering an intervention, perhaps dismantling the intervention to test the relative impact of different components, and perhaps testing such variations as post-treatment booster sessions or the inclusion of additional family members. Typically, it is only after a sometimes extended series of such efficacy trials that the intervention is brought into community settings ‘to measure the public health impact’ (Greenwald & Cullen, 1984; National Institutes of Health, 1994).
From an implementation perspective, this SET approach may work reasonably well for interventions that operate directly on the biological system - e.g. psychoactive drugs and medical procedures for cancer treatment - or other treatment targets for which differences between research and clinic conditions may not greatly alter the intervention effect. Under such conditions, placing effectiveness tests at the very end of a series of efficacy trials may be reasonable, because the intervention may require relatively little modification to be rendered effective in real-world clinical care contexts. [Evidence on stimulant treatment of ADHD in community settings does suggest, though, that the ‘bringing to scale’ process can be quite necessary, even if the treatment involved is a medication (MTA Cooperative Group, 1999).]
However, the gap between research and practice is often quite wide in the case of psychotherapies, arguably much wider than with biologically-focused treatments. For psychotherapies the gap includes (1) psychological and social characteristics of the treated individuals (e.g. clinic-referred youth tend to be more severely disturbed, more likely to meet criteria for a diagnosis, more likely to have comorbidities, and more likely to miss appointments or drop out of treatment); (2) characteristics of their families (e.g. more parental psychopathology, family life event stressors, and perhaps even child maltreatment); (3) reasons for seeking treatment (e.g. not recruited through ads or screening, but referred by caregivers because of serious problems or family crisis, perhaps even court-ordered); (4) the settings in which treatment is carried out (e.g. more financial forms to complete, more bureaucracy, and sometimes a less welcoming approach in the clinic); (5) the therapists who provide treatment (e.g. not graduate students or research assistants hired by and loyal to the advisor/employer and committed to her/his treatment program, but rather staff therapists who barely know the treatment developer or the specific treatment, and who may prefer different treatment methods); (6) the incentive system (e.g. not paid by the treatment developer to deliver the treatment with close adherence to the manual, but paid by the clinic to see many cases and with no method prescribed); and (7) the conditions under which therapists deliver the treatment (not graduate students’ flexible time, but strict productivity requirements, paperwork to complete, insurance requirements to satisfy, and little time to learn a manual or adhere closely to it).
Perhaps the numerous differences between psychotherapy in efficacy research and psychotherapy in actual clinical practice are too pronounced to be bridged as simply the final step after a series of efficacy experiments. [And, truth be told, this final step has not actually been taken yet for most protocol-guided EBTs.] Perhaps the number of dimensions along which treatment would need to be adjusted to span the lab-to-clinic gap makes the task of moving from efficacy trials to effectiveness tests so complex that the task needs to be made an integral part of the treatment development process. In fact, the very real-world factors that efficacy trial investigators might view as a nuisance (e.g. child comorbidity, parent pathology, life stresses that produce no-shows and dropouts, therapists with heavy caseloads) and thus attempt to avoid (e.g. by recruiting and screening cases, applying exclusion criteria, hiring their own therapists), may in fact be precisely the kinds of factors that need to be included, understood, and addressed, if psychotherapy treatment protocols are to be created that fit well into everyday clinical practice. Treatments that cannot cope with these real-world factors may not fare so well in practice, no matter how beneficial they are in efficacy trials.
A related point is that implementation of EBTs in real-world practice settings may require certain interventions in the settings themselves, to eliminate obstacles to effective use of the EBTs. As an example, treatments that are built on weekly instalments of skill-building with children or parents may require new family engagement procedures to generate faithful attendance in a setting where appointments are often missed in usual care. As another example, new interventions that do not fit neatly within the organisational structure or standard procedures of a clinic (e.g. because they require approaches to assessment or supervision that differ from the clinic routine) may only be workable and effective if paired with procedures for organisational problem-solving. Development of effective versions of such setting-focused interventions would seem to require situations in which investigators take EBTs into the real-world settings for which those treatments are ultimately intended and identify the setting-focused interventions that are needed to make the target treatments succeed.