IDENTIFYING THE TREATMENT EFFECT SIGNAL FROM THE NOISE 
Version of Record online: 15 JUL 2008
Journal compilation © 2008 Society for the Study of Addiction
Volume 103, Issue 8, pages 1403–1404, August 2008
How to Cite
FERRI, M., AMATO, L. and DAVOLI, M. (2008), IDENTIFYING THE TREATMENT EFFECT SIGNAL FROM THE NOISE . Addiction, 103: 1403–1404. doi: 10.1111/j.1360-0443.2008.02296.x
- Issue online: 15 JUL 2008
- Version of Record online: 15 JUL 2008
Cochrane Reviews are ‘living documents’ which are updated in a timely fashion. The authors of Cochrane Reviews seek comments and criticisms actively from experts in the field to improve the contents of reviews and to eventually modify results as new evidence emerges.
We believe that the definitions of the interventions are explained adequately in both the introduction to the review , where they are described and substantiated, and in the descriptions of enclosed studies where a specific warning is reported about a possible ambiguity of definitions of the interventions. We regret to note that the citation the authors have quoted in inverted commas in the letter commenting on the review published in this issue of Addiction has been created by pulling together extensive text from different parts of the review .
We recognize the incompleteness of the Project MATCH analysis, which was explained in the ‘description of studies’ section. Here we stated that the outcomes were derived from individual articles, specifying that the whole Project had a broader objective than meeting the inclusion criteria decided at protocol stage, and we took into consideration only some of the outcomes. Cochrane Reviews are preceded by the publication of peer-reviewed protocols which define inclusion and exclusion criteria for studies, search strategies and the methods of the review.
Dr Kaskutas’ comments about the description of Project MATCH's outcomes concerning reduction of drinking are similar to those of our review. In fact, we say that 12-Step facilitation (TSF) had more effective results than comparison treatments in terms of percentage days abstinence (PDA) at 1 year follow-up. At 3 years’ follow-up  the Project Match Research Group reported (p. 1307) that they had analysed the treatment differences by way of a regression model in which the treatment was the only independent variable. The remaining variables (i.e. baseline drinking, site, primary and secondary matching hypotheses) were enclosed successively as covariates. With the above-described model, ‘TSF clients reported higher PDA than CBT ones, MET fell in the middle and was not different from the other two’. The difference between TSF and cognitive–behaviour therapy (CBT) was not found when baseline drinking was enclosed as a covariate. However, if site, treatment type and site × treatment type were also entered as covariates, the variables PDA and drinks for drinking days (DDD) both had significant results. Performing post-hoc tests, the authors found TSF outcomes superior to CBT, but TSF was not superior to motivational enhancement therapy (MET). The Project Match Research Group therefore performed a logistic regression analysis with a dichotomous abstinence outcome variable in which abstinence is defined as not drinking at all during months 37–39. In this case TSF clients obtained higher rates of abstinence; 36% in TSF versus 24% in CBT and 27% in MET. The Project Match Research Group authors themselves define the results as a ‘possible slight advantage’, and we described them as ‘no significant differences’, meaning that there is a difference, but this difference may be small. Therefore, in a commentary on the study by the Project Match authors , differences among the three studied interventions were summarized as follow: ‘Project MATCH demonstrated that outcomes from TSF are as favourable as those for other well-tested approaches in the treatment of alcohol problems’ (p. 33).
We believe that the conclusions regarding implications for practice simply flow from our results. In fact, we say that: ‘It should also be underlined that in the available studies all the interventions appeared to improve at least some of the outcomes considered’.
It is not the aim of this letter to discuss the case for the adoption of non-experimental studies in Cochrane Systematic Reviews in which methods can be consulted on the web (http://www.cochrane.org/reviews/revstruc.htm) and we recognize that good-quality evidence may come from observational studies as well, and guide the process of developing clinical guidelines , particularly when analysing the occurrence of adverse effects or other rare events. However, it has been shown recently that observations speak for themselves and randomized controlled trials are only unnecessary when the strength of the association between two variables is sufficient to identify the treatment effect signal from the noise from bias and confounding. Nevertheless, our review concluded by proposing that it is possible that a well-designed qualitative study may provide us with further hypotheses for future research. We therefore recognize that there are aspects of problematic alcohol use which can be difficult to study in experimental conditions.
In conclusion, we thank Dr Kaskutas for devoting her attention to our review and we will take her suggestions into consideration when updating it (Cochrane Reviews are updated on average every 2 years), although we reject the accusation of having provided misleading conclusions. We wish to remind readers that Cochrane Systematic Reviews should be considered as the best source of available evidence, but not recommendations for clinical practice.
- 5Comments on Project MATCH: matching alcohol treatments to client heterogeneity. Addiction 1999; 94: 31–69.