SEARCH

SEARCH BY CITATION

In his commentary ‘Is this the path to effective prevention?’[1], Dr Midford expresses his ‘unease about the paradigm that underpins effort’ in the area of universal preventive intervention research. He raises general issues about the role of universal preventive interventions vis-à-vis other strategies that potentially mitigate the consequences of substance misuse, as well as issues about methods for evaluating intervention efficacy. He supports his critical view of the research on universal prevention with a critique of our paper [2], taking issue with one-tailed significance tests and presenting ‘re-analyses’ of our data. We commend critical reviews of research on preventive interventions that highlight important issues to consider; we take exception, however, to what we view as a misleading articulation of issues regarding our paper and the questionable reasoning used in conducting re-analyses of our data. In addition, treatment of more general issues fails to adequately consider the range of relevant literature.

A major issue concerns Midford's assertion that our use of an intent-to-treat analysis magnifies effects by inflating sample size. Intent-to-treat analyses are considered widely to be the gold standard for initial testing of intervention efficacy, avoiding intervention self-selection biases and other potential confounds that may occur in subgroup analyses [3]. Moreover, intent-to-treat analyses are assumed generally to be more conservative, often deflating observed intervention effects by including data from individuals who did not receive an intervention in the analyzed intervention group.

Midford's ‘as-treated’ re-analyses do not incorporate information on who did or did not actually receive an intervention, instead estimating the number of attendees from intervention participation tables. As such, his analyses cannot be described legitimately as ‘as-treated’. Also, he assumes that the proportion of substance users in his reduced sample would be the same as that observed in our full sample. This assumption is not defensible because the justification for conducting ‘as-treated’ analyses assumes that only actual participants in the intervention would show intervention benefits. To be consistent with this justification, the proportion of users in the ‘not-treated’ subgroups in the intervention conditions in study 1 (not presented in Midford's tables) would be comparable to the proportion of users as the control group; for study 2, LST (Life Skills Training)-only rates of drug misuse should be applied to the portion of the SFP (Strengthening Families Program): 10–14 + LST group who did not attend SFP: 10–14 (Midford ignores the fact that SFP: 10–14 non-attendees actually received LST). Although we maintain that intent-to-treat analyses are the most appropriate in our case, we demonstrate how a more appropriately informed approximation of ‘as-treated’ analyses shows intervention results considerably stronger than reported by Midford (see Tables 1 and 2).

Table 1.  Re-calculated results for estimated ‘as-treated’ analysis for study 1 informed by not-treated rates, adapted from Midford commentary, Table 1.
 12th grade past year misuse of prescription narcoticsYoung adult life-time misuse of prescription narcoticsYoung adult life-time misuse of prescription barbiturates
Yes (%)No (%)Total (%)P-value two-tailed Fisher's exact testYes (%)No (%)Total (%)P-value two-tailed Fisher's exact testYes (%)No (%)Total (%)P-value two-tailed Fisher's exact test
  1. The number of users for the ‘not-treated’ intervention subgroups is estimated by applying the proportion of users in the control group to the number of individuals in the intervention groups that did not attend an intervention (‘not-treated’ subgroup users = control group user proportion × the number of intervention group individuals not actually attending). The number of users in the ‘treated’ intervention subgroups is then computed as the observed number of users in the entire intervention groups minus the estimated number of users among the ‘not-treated’ subgroups, with negative results rounded to 0. ISFP: Iowa Strengthening Families Program; PDFY: Preparing for Drug Free Years.

Midford results
 ISFP01171170.039611161170.005201171170.0760
 PDFY11231240.138551191240.152021221240.7028
 Control6151157 14147161 5156161 
Re-calculated results
 ISFP (not treated)1.30 34 4.61 53 1.64 53 
(3.82)   (8.70)   (3.11)   
 ISFP (treated)01171170.039601171170.000101171170.0760
 PDFY (not treated)0.96 25 2.45 28 0.87 28 
(3.82)   (8.70)   (3.11)   
 PDFY (treated)01241240.036051191240.152011231240.2376
 Control group6151157 14147161 5156161 
(3.82)(96.18)(100) (8.70)(91.30)(100) (3.11)(96.89)(100) 
Table 2.  Re-calculated results for estimated ‘as-treated’ analysis for study 2, adapted from Midford commentary Table 2.
 11th grade life-time prescription drug misuse12th grade life-time prescription drug misuse
Yes (%)No (%)Total (%)P-value two-tailed Fisher's exact testYes (%)No (%)Total (%)P-value two-tailed Fisher's exact test
  • *

    The number of prescription drug users for LST + SFP: 10–14 ‘not-treated’ subgroup is estimated by applying the proportion of users in the LST group to the number of individuals in the LST + SFP: 10–14 group that did not attend the SFP: 10–14 intervention (i.e. the ‘not- treated’ subgroup). The number of users in the LST + SFP: 10–14 ‘treated’ subgroup is then computed as the observed number of users in the entire LST + SFP: 10–14 intervention group minus the estimated number of users for the LST + SFP: 10–14 ‘not-treated’ subgroup. LST: Life Skills Training; SFP: Strengthening Families Program.

Midford results
 LST314434740.5238353904250.3167
 LST + SFP: 10–1451321370.1202101271370.3090
 Control35417452 36307343 
Re-calculated results*
 LST (treated)*314434740.5238353904250.3167
(6.54)(93.46)(100) (8.24)(91.76)(100) 
 LST + SFP: 10–14 (not treated)24.8 379 31.2 379 
(6.54)   (8.24)   
 LST + SFP: 10–14 (treated)*01371370.000131341370.0015
 Control35417452 36307343 

Independent of methodological problems with Midford's re-analyses, it is important to note that such analyses are also misguided on theoretical grounds. They ignore published research discussing mechanisms of long-term intervention effects among non-participants in a community, occurring via peer and/or parent social networks [4]. Based on this work, reductions in substance use among those directly receiving interventions could be expected, over time, to produce reductions in substance use among non-participating peers, further justifying analyses incorporating the full sample.

A more minor issue concerns our use of one-tailed significance tests. Negative intervention effects may be conceived as a possibility; that conception, however, would be unreasonable because of the preponderance of evidence for the specific interventions examined. Results in numerous refereed journal reports ([5], http://www.ppsi.iastate.edu) clearly justify directional hypotheses and, therefore, one-tailed significance tests. In any event, our use of directional hypotheses is transparent; any reader who disagrees with this usage could readily convert the reported P-values accordingly. Also, it is simply inaccurate to state that an alpha was ‘set at 0.10’ (p. 1169 [1]). Our reports follow the social science standard of reporting P values associated with a range of alpha levels, clearly labeling findings at or below 0.10 as ‘marginally significant’ (e.g. p. 1166 [2]).

We believe that the critical reader would want to assess two central but distinct questions about our paper: ‘What are the most valid and reasonable conclusions to be drawn about intervention efficacy from study observations?’ and ‘What is the practical significance of those findings?’. Concerning the first question, the paper describes studies that are designed to address threats to internal, construct, and statistical conclusion validity, along with study limitations. Pertaining to practical significance, our paper presents relevant interpretive guidelines. Overall, readers are in a good position to draw their own conclusions.

Concerning the more general issues noted in the opening paragraph, Midford draws upon a subset of critical papers, failing to represent the full range of reviews of the evidence base for preventive interventions (the lead author recently found more than 24 reviews of relevance (e.g. [6–8]). Midford fails to address adequately the rationale for universal interventions, and his conclusions ignore relevant literature describing the most appropriate application of universal interventions, in conjunction with selective and indicated ones [9]. We support strongly an open dialogue among scientists with varying positions upon the important issues raised by Midford, but believe they should be guided by full consideration of the relevant literatures and follow accepted standards for preventive intervention science (see http://www.preventionscience.org).

References

  1. Top of page
  2. References