How best to measure change in evaluations of treatment for substance use disorder
Article first published online: 6 OCT 2010
© 2010 The Authors, Addiction © 2010 Society for the Study of Addiction
Volume 106, Issue 2, pages 294–302, February 2011
How to Cite
Marsden, J., Eastwood, B., Wright, C., Bradbury, C., Knight, J. and Hammond, P. (2011), How best to measure change in evaluations of treatment for substance use disorder. Addiction, 106: 294–302. doi: 10.1111/j.1360-0443.2010.03143.x
- Issue published online: 5 JAN 2011
- Article first published online: 6 OCT 2010
- Submitted 23 December 2009; initial review completed 10 March 2010; final version accepted 30 July 2010
- Jacobson & Truax;
- regression to the mean;
- reliable change index;
- substance use disorder treatment effectiveness
Aims To compare the performance of the Jacobson & Truax (JT) reliable change index (RCI) with three alternative methods, using data from individuals receiving treatment for substance use disorders.
Design English National Treatment Outcome Monitoring Database for publicly funded specialist community pharmacological and psychosocial interventions.
Participants New adult admissions to treatment across England (1 January–31 December 2008), with in-treatment clinic progress review conducted after an average of 122.8 days for 18 163 individuals.
Measurements Self-reported days using heroin, crack, cocaine powder and alcohol during the 4 weeks before admission and clinical review, recorded using the Treatment Outcomes Profile and analysed using a multi-level, mixed-linear model, with both observed and true scores to estimate the effect of regression to the mean (RTM). Differences in performance among the JT RCI and the alternative methods were assessed by the proportion assigned to a reliably ‘improved’, ‘unchanged’ or ‘reliably deteriorated’ category; level of agreement; difference in effect size for observed and true scores; and receiver operating characteristic parameters.
Findings When compared to the alternative methods, the JT RCI was more conservative in assigning individuals to the improved category, and it showed no evidence of inferiority on any measure. For each method, all individuals categorized as reliably deteriorated and the majority of those categorized reliably improved had outcome scores which fell beyond that expected by RTM. Substituting true scores for observed scores moderated the size of the change effect associated with reduced use of the four substances, but this remained statistically significant.
Conclusions The Jacobson & Truax Reliable Change Index appears to be the optimal measure of change for evaluations of treatment for substance use disorder, in that it is the most conservative for assessing improvement and at least as accurate on all other criteria. Any evaluation of change needs to take account of regression to the mean.