Reason for withdrawal from publication
Possible errors in this review have been identified through the Feedback mechanism. These concern: data extraction, presentation and statistical analysis (see below). The review will be re-published following investigation, revision and further peer review, at which time all points raised will be addressed. As soon as the revised version of this review has been approved by an independent editorial group, a response to feedback will appear at this link: (http://www.bris.ac.uk/Depts/CochraneBehav/).
Article: Home-based support for disadvantaged teenage mothers
Date of Submission: 20-Sep-2007
Name: David Olds
Email Address: email@example.com
Personal Description: Occupation Professor of Pediatrics
(References are numbered and the reference list appears at the end of the Feedback).
Feedback: Given our high regard for the ideals and methods of the Cochrane Collaboration, we were surprised and seriously concerned by the publication of two recent reviews of home visiting for disadvantaged mothers produced by Geraldine MacDonald and colleagues (1;2).
One review focuses on randomized trials of programs for teenage mothers(2) and the other focuses on trials of programs for adults.(1) These reviews are flawed in three fundamental ways. First, the authors' attempts to synthesize evidence from trials of heterogeneous programs conceal important differences among the programs and their effects (3;4). Second, the authors have failed to meet basic methodological standards for conducting such reviews,(5) which has led to significant omissions and inaccuracies. Third, the approach the reviewers have taken with this literature (outcomes selected, separation of trials by ages of mothers, and consideration of moderated program effects) further obfuscates this literature. Not surprisingly, these flawed efforts have resulted in inaccurate conclusions.
We should acknowledge that we are not unbiased observers. Each of us has been involved in research with a program of home visiting for disadvantaged parents known today as the Nurse-Family Partnership (NFP), and have authored our own reviews of this literature. While this has the potential for influencing our interpretation of these reviews, it also gives us unique insight into variations in the interventions themselves, the challenges of conducting research on these interventions, and what the evidence does and does not say about this method of delivering services.
Attempting to Summarize across Heterogeneous Interventions Obscures the Literature
The Cochrane Handbook for Systematic Reviews of Interventions states "If studies are clinically diverse then a meta-analysis may be meaningless, and genuine differences in effects may be obscured." (page 99)(3). The studies in the current reviews are clinically diverse. They differ in length (ranging from 6 months to 5 years); target populations (e.g., pregnant women or primiparous women going through the transition to parenthood); goals (such as whether they attempt to improve prenatal health and prevent rapid successive pregnancy); and structures (including visitor backgrounds and the degree to which program goals and objectives are operationalized in visit-by-visit guidelines and visitors are trained in behavioral change strategies). We believe that the success of interventions can be attributed to the constellation of program attributes such as these. Notwithstanding the reviewers' acknowledgments of these challenges,(1;2) their attempts to aggregate findings across different program models mask important differences among home visiting programs and their effects that deserve to be highlighted rather than buried.
Flawed Execution of Reviews Harms the Field of Study and the Stature of Cochrane
Reviews of the literature must be held to the same high standards as the primary research they synthesize.(5) The current reviews contain numerous errors, omissions, and inappropriate analytic approaches that distort findings from individual trials. The errors reflect a pervasive lack of attention to the most fundamental aspects of the studies they review. They fail, for example, to follow their inclusion criteria in determining whether studies belong in the teenage or adult reviews. They are inconsistent in their report of allocation concealment in the Denver trial. P-values are reported inaccurately (e.g., p=0.70 rather than p=0.07). A 4-year follow-up of the Denver trial of the NFP published in Pediatrics in 2004 is not included in the review, even though the authors reran their literature searches in March 2006 (6). Across both reviews, results are omitted, reported inaccurately, and distorted. A table of errors that we have noted is attached [sent separately to review authors, and being addressed by them - Ed].. The errors and presence of editorial notes in the results and discussion sections, such as "WHAT IS THIS" in the review of programs for teenage mothers,(2) suggest a hurried, inattentive process.
Flawed Approach to this Literature
Distortions of the Evidence through Selection of Outcomes. The Cochrane Handbook for Systematic Reviews of Interventions states, "In general, Cochrane reviews should include all reported outcomes that are likely to be meaningful to people making a decision about the healthcare problem the review addresses" (page 60)(3). The choice of outcomes to include or exclude in these reviews violates this guidance and is incongruent with the literature. The reviewers selected some outcomes of questionable clinical significance (e.g., 'parents reports of general stress', 'parenting stress'), which the majority of primary investigators did not measure. At the same time, they excluded others that have undeniable relevance for those who must decide about whether to invest in home visiting programs: indicators of prenatal health (such as prenatal tobacco use), rates and timing of subsequent pregnancies, maternal employment, and use of welfare services. These important outcomes were examined in at least 3 of the trials covered in these reviews (7;8). The reviewers' decision not to evaluate programs in relation to these important outcomes reflects a failure to follow Cochrane guidelines and to place research on home visiting in proper clinical, policy, and scientific context.
Distortions of the Evidence by Separating Trials on the Basis of Maternal Age.
The authors chose to conduct two reviews, separating the literature into programs that focused primarily on teenage mothers(2) as opposed to those that served primarily adults(1). This fundamental decision, which was not explained or justified, excluded studies that enrolled samples that were neither predominantly teenage nor adult by the reviewers' definitions. Investigators did not always report data in accordance with the reviewers' criteria, forcing the reviewers either to use data that did not conform to their classification schemes or to request investigators to perform extra unfunded work. The reviewers' efforts to classify trials on the basis of the ages of the parents has confused the literature and reduced the number of studies of programs serving disadvantaged parents that might be evaluated jointly.
Selection of Some Sample Characteristics that Might Moderate Program Impact is Poorly Justified.
Within each of their reviews, the authors indicated that they were interested in learning about whether home visiting programs had greater or reduced impact for minorities and for mothers with learning disabilities (1;2). By asking this kind of question, they indicate an appreciation for the possibility that the effects of such programs may vary depending upon the configuration of population risks, but their focus on learning disabilities raises questions about the reviewers' understanding about how epidemiology should be used to guide intervention research. We know of no empirical basis for claiming that parents with learning disabilities (as opposed to low intelligence) are at increased risk for compromised parenting or that home visiting programs might reduce risks associated with learning disabilities.
Misrepresenting the NFP
The poor conduct of these reviews prompts us to reply to some of the more significant distortions regarding research on the NFP. Problems may exist with other studies, which we did not examine as closely.
The NFP Program Has Been Tested with Different Racial and Ethnic Groups.
The reviewers were particularly interested in the effects of home visiting programs for minorities and yet failed to report the systematic way the trials of the NFP have addressed this issue. In trials of the NFP, we enrolled samples that were constituted incrementally from one trial to the next to examine NFP impact first with whites,7-13 then blacks,14-16 and finally with a sample that included a large portion of Hispanics (6;17). This strategy enabled us to determine the extent to which NFP findings replicated with new racial and ethnic groups as the program of research grew(7;8). This systematic approach to designing trials is nearly unprecedented in health and human service intervention research and was not addressed in these reviews.
NFP Program Effects on Parenting and Child Outcomes are More Pronounced for Groups at Greater Risk.
The reviewers expressed interest in understanding participant moderators of home visiting program effects, but failed to adequately portray the coherent patterns of participant effect moderation in trials of the NFP. The effects of the program examined in the first trial in Elmira were more pronounced among mothers and children at greater socio-demographic risk (e.g., where the mothers had combinations of being poor, unmarried, and teenage). The effects of the program in Elmira on injuries and child maltreatment were further concentrated among women who had limited sense of control over their lives. These patterns of results from Elmira led us to focus the subsequent trials in Memphis and Denver on populations that exhibited concentrated social disadvantage, and to hypothesize that the effects of this program on care-giving and child outcomes would be even greater among mothers who are at greater risk for failing to care competently for their children because of mothers' limited sense of mastery, mental health, and intellectual functioning, i.e., they had low 'psychological resources.' There is now consistent evidence from trials of the NFP that program effects on care-giving and children are more pronounced among disadvantaged families where mothers have low 'psychological resources' (6;17). This important pattern of results is disregarded in these reviews.
The NFP and Background of Visitors.
The Denver trial examined the relative impact of the NFP when delivered by nurses versus paraprofessional visitors by randomly assigning families to arms of a trial in which visitor background was varied systematically by design (6;17). Findings through child age 4 show that nurses produced clinically and statistically significant effects consistent with earlier trials of the NFP and that paraprofessional effects were rarely statistically or clinically significant. It is the only study in the home visiting literature that randomized participants into arms of a trial in which visitor background was varied systematically. Given that the reviewers professed interest in sorting out whether the background of visitors matters, it is remarkable that their summarization of the Denver trial virtually ignores results for the paraprofessional visitors.
Inaccurate Reporting of Findings.
One consequence of the reviewers' inaccurate review procedures is that there are major misrepresentations of findings for the NFP, including failure to report important significant effects, distorted reporting of results, incorrect summaries of study findings, as well as unwarranted comments about this literature. These inaccuracies in reports of findings and discussions of the literature are included in the table of errors.
We are concerned that the Cochrane Collaboration has sanctioned the publication of these reviews. Reviews that offer complete and accurate summaries of progress and failures in health care will contribute to informed policy, practice, and scientific advancement. Those that attempt to summarize across heterogeneous interventions or fail to summarize the literature accurately have the potential to damage both the field they review and the review process itself. Cochrane review procedures represent an enormously important step in the direction of ensuring rigorous summaries of the literature. The application of these procedures cannot substitute, however, for a nuanced understanding of the interventions studied, the challenges of measurement in complex service settings, and basic principles of research design and analysis. The publication of these reviews highlights the need for reviewers to fully comprehend the literature they examine and to present their work with accuracy, quality, and a deep sense of responsibility.
David L. Olds, Ph.D.
Dennis W. Luckey, Ph.D.
Nancy S. Donelan-McCall, Ph.D.
John Holmberg, Psy.D.
Robin A. Tutt, BA
University of Colorado
Harriet Kitzman, R.N, Ph.D.
Robert Cole, Ph.D.
Elizabeth Anson, M.S.
University of Rochester
Charles R. Henderson, Jr.
John Eckenrode, Ph.D.
Kimberly Sidora-Arcoleo, Ph.D., M.P.H.
University of Arizona
1. Bennett C, Macdonald G, Dennis J, Coren E, Patterson J, Astin M, Abbott J. Home-based support for disadvantaged adult mothers. Cochrane Database Syst Rev 2007;(3):CD003759.
2. Macdonald G, Bennett C, Dennis J, Coren E, Patterson J, Astin M, Abbott J. Home-based support for disadvantaged teenage mothers. Cochrane Database Syst Rev 2007;(3):CD006723.
3. Higgins, J. P. T.; Green, S. Cochrane Handbook for Systematic Reviews of Interventions 4.2.6. Chichester, UK: John Wiley & Sons, Ltd.; 2006.
4. Olds DL. What can we conclude from meta analyses of early interventions. Prevention & Treatment; 2003;6.
5. Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchison BG, Milner RA, Streiner DL. Agreement among reviewers of review articles. J Clin Epidemiol 1991;44(1):91-8.
6. Olds DL, Robinson J, Pettitt L, Luckey DW, Holmberg J, Ng RK, Isacks K, Sheff K. Effects of home visits by paraprofessionals and by nurses: age-four follow-up of a randomized trial. Pediatrics 2004;114:1560-8.
7. Olds DL. Prenatal and infancy home visiting by nurses: from randomized trials to community replication. Prevention Science 2002 Sep;3(3):153-72.
8. Olds DL. Preventing crime with prenatal and infancy support of parents: the Nurse-Family Partnership. Victims and Offenders 2007;2:205-25.
9. Olds DL, Eckenrode J, Henderson CR Jr, Kitzman H, Powers J, Cole R, Sidora K, Morris P, Pettitt LM, Luckey D. Long-term effects of home visitation on maternal life course and child abuse and neglect. Fifteen-year follow-up of a randomized trial. JAMA 1997 Aug 27;278(8):637-43.
10. Olds DL, Henderson CR Jr, Kitzman H. Does prenatal and infancy nurse home visitation have enduring effects on qualities of parental caregiving and child health at 25 to 50 months of life? Pediatrics 1994 Jan;93(1):89-98.
11. Olds DL, Henderson CR Jr, Chamberlin R, Tatelbaum R. Preventing child abuse and neglect: a randomized trial of nurse home visitation. Pediatrics 1986 Jul;78(1):65-78.
12. Olds D, Henderson CR Jr, Cole R, Eckenrode J, Kitzman H, Luckey D, Pettitt L, Sidora K, Morris P, Powers J. Long-term effects of nurse home visitation on children's criminal and antisocial behavior: 15-year follow-up of a randomized controlled trial. JAMA 1998 Oct 14;280(14):1238-44.
13. Interview with Dr. David Olds [homepage on the Internet]. Nurse-Family Partnership; c2006 [cited 2007 Sep 5]. Available from: http://www.nursefamilypartnership.org/resources/files/PDF/DavidOldsinterview1-24-06.pdf.
14. Kitzman H, Olds DL, Henderson CR Jr, Hanks C, Cole R, Tatelbaum R, McConnochie KM, Sidora K, Luckey DW, Shaver D, et al. Effect of prenatal and infancy home visitation by nurses on pregnancy outcomes, childhood injuries, and repeated childbearing. A randomized controlled trial. JAMA 1997 Aug 27;278(8):644-52.
15. Kitzman H, Olds DL, Sidora K, Henderson CR Jr, Hanks C, Cole R, Luckey DW, Bondy J, Cole K, Glazner J. Enduring effects of nurse home visitation on maternal life course: a 3-year follow-up of a randomized trial. JAMA 2000 Apr 19;283(15):1983-9.
16. Olds D, Kitzman H, Cole R, Robinson J, Sidora K, Luckey D, Henderson C, Hanks C, Bondy J, Holmberg J. Effects of nurse home visiting on maternal life-course and child development: age-six follow-up of a randomized trial. Pediatrics 2004;114:1550-9.
17. Olds DL, Robinson J, O'Brien R, Luckey DW, Pettitt LM, Henderson CR Jr, Ng RK, Sheff KL, Korfmacher J, Hiatt S, et al. Home visiting by paraprofessionals and by nurses: a randomized, controlled trial. Pediatrics 2002 Sep;110(3):486-96.
Submitter agrees with default conflict of interest statement:
I certify that I have no affiliations with or involvement in any organization or entity with a financial interest in the subject matter of my feedback.