This report describes an innovative pilot project to involve consumers and consumer advocates in the process of refereeing systematic reviews in preparation by members of the Cochrane Collaboration's Pregnancy and Childbirth Group. The Cochrane Collaboration is an international not-for-profit organization that was established in 1993 to prepare, maintain, and ensure the accessibility of systematic reviews about the effects of specific health care interventions.

In contrast to more conventional and less formally conducted “narrative” reviews of scientific reports, systematic reviews assess the available evidence about the effects of health interventions through a rigorous and explicit process that minimizes bias and allows prospective users to understand and assess the methods involved. This process includes the following: specifying populations, interventions, outcomes, and study designs that will and will not be included; designing and carrying out a thorough search for reports meeting these criteria; eliminating methodologically inadequate reports from among the relevant reports that are identified, and summarizing results of the relevant higher quality studies—if feasible, through metaanalysis. Within the Cochrane Collaboration, preference is given to controlled trial designs. If done well, a systematic review provides the best possible answers, within the scope of currently available research, to questions about the safety and effectiveness of health interventions.

After proposing and registering a review title, authors of Cochrane Collaboration systematic reviews develop a “protocol” or formal plan outlining the rationale for the review and steps they will take to carry it out. Submitted draft protocols are refereed and then revised by authors. After acceptance by the editorial team of the Pregnancy and Childbirth Group or another of approximately 50 topic-based review groups, a protocol is entered into The Cochrane Database of Systematic Reviews. This database is issued quarterly and is available by subscription as a component of both The Cochrane Library (1) and Evidence-Based Medicine Reviews (2). After the plan for a review is carried out, the draft review is again refereed and revised. Following acceptance by the pertinent editorial team, the systematic review is published in The Cochrane Database of Systematic Reviews. For further information about the Cochrane Collaboration and its work, see the Cochrane Collaboration brochure (3), leaflet (4), and website (5).

Since its inception, Cochrane Collaboration leaders and enacted policies have viewed full partnership with “consumers” as essential for the success of this work (6). Although consumers are welcome to serve as authors of reviews, and some do so, most reviews are prepared by academics, clinicians, and other health professionals. Consumers are understood as offering complementary perspectives of those who use services and often have personal experience with health conditions and interventions. The organization emphasizes consumer involvement to help ensure that completed reviews are relevant and comprehensible to end users, and to increase the likelihood that both consumers and professionals will use the reviews.

To encourage consumer participation, two of the 12 positions on the Cochrane Collaboration's Steering Group are reserved for consumer representatives. A Consumer Network, which is open to all without charge, provides general support through training programs, a newsletter, an Internet website (7), stipends for attending meetings, and other avenues. In addition, the groups that prepare reviews in specific topic areas and other official Cochrane entities are all mandated to seek consumer input and feedback.

In 1998 the Cochrane Pregnancy and Childbirth Group developed a structure and process for a Consumer Panel, which began to function in January 1999. Its principal work is the refereeing of all drafts of protocols for systematic reviews and drafts of completed reviews as they have gone through the editorial process. This work occurs in tandem with more conventional professional refereeing. The Consumer Panel has also contributed in other ways, such as participating in applied research to strengthen the process of involving consumers and providing consumer input into spin-off projects that are not formally within the auspices of the Cochrane Collaboration. This report focuses on the work of the Cochrane Pregnancy and Childbirth Group's Consumer Panel in refereeing Cochrane protocols and prepublication reviews about the effects of maternity care interventions.

Process of Involving Consumers

  1. Top of page
  2. Process of Involving Consumers
  3. Experience of Involving Consumers
  4. Future Evolution of the Pregnancy and Childbirth Group's Consumer Panel
  5. References

About 250 individuals throughout the world work as members of the Cochrane Pregnancy and Childbirth Group to prepare systematic reviews about the safety and effectiveness of maternity care interventions. A Coordinating Editor and team of several additional editors have responsibility for planning and monitoring the group's work. The Review Group Coordinator manages the day-to-day activities of the group and the editorial office, which is located in Liverpool, United Kingdom, and the Trials Search Coordinator helps reviewers identify relevant studies. A Principal Consumer Coordinator takes the lead in consumer involvement with the review group, coordinates input from European consumers, and collates and summarizes all consumer feedback for editors and reviewers. Consumer Coordinators for North America and Australasia involve consumers from these areas of the world.

When an author submits a draft protocol for a review by the Pregnancy and Childbirth Group, the editorial team solicits evaluations from three peer referees: a group editor and group reviewer who is not an editor, and a content expert who is not a group member. In addition, the protocol is forwarded to the Consumer Coordinators, who ask Consumer Panel members if they have an interest in evaluating the specific protocol. As limited resources were available to support the work of the Consumer Panel in its first two years, most members who served on the panel and evaluated draft protocols and reviews were recruited from the formal and informal networks of the three Consumer Coordinators.

The Pregnancy and Childbirth Group has welcomed evaluations from all consumers who have volunteered for each of the protocols. The Consumer Coordinators send all volunteers a copy of the protocol and a checklist for evaluating the protocol. The checklist covers all sections of the protocol, beginning with the title. In addition to yes/no/don't know ratings, the checklist asks evaluators to provide details and suggestions about any areas of concern. To facilitate participation, any form of consumer feedback is welcome, including: completed checklists, comments inserted on protocols (either within electronic versions or handwritten on printed versions), separate narrative evaluations, and verbal feedback by telephone. Moreover, consumer feedback is welcomed and forwarded to the responsible editor and review author at any time. In practice, most consumers use the checklist form and complete the evaluation within the requested two-week time period.

The Principal Consumer Coordinator receives all consumer evaluations for each protocol, summarizes and collates all feedback, and provides pertinent biographical information about participating consumers. In a cover note to the lead review author and the editor responsible for this protocol, she offers to discuss any of the feedback. She then sends the collated consumer feedback to the consumers who commented on the protocol. With all consumer and professional evaluations in hand, the editor works with the lead review author to strengthen any shortcomings. The editorial team copy edits accepted protocols and forwards them for publication in the Cochrane Database of Systematic Reviews. Consumers receive copies of any published protocols on which they commented.

Authors of accepted protocols are encouraged to carry out their reviews rigorously and expeditiously. After the editorial office receives a draft of the completed review, the same set of professional and consumer referees who commented on the protocol are asked to evaluate the completed draft review. Consumers receive a copy of the draft review and a review evaluation checklist. Although any type of feedback is welcome, most use the checklist and provide supplemental comments. Again, the Principal Consumer Coordinator summarizes the consumer feedback, collates all feedback, and offers further dialog and assistance to the review author and responsible editor. As with protocols, the responsible editor considers the professional and consumer feedback and makes recommendations to the author(s) for strengthening the review. After accepting and copy editing the review, the editorial team submits it for publication in the Cochrane Database of Systematic Reviews.

New Consumer Panel members are given two documents to help them understand and participate in the referee process: consumer guides to assessing Cochrane protocols and reviews, respectively. They discuss the reason for conducting systematic reviews and the principles involved, the process of preparing a protocol and review, and the purpose, objectives, and expectations for each section of protocols and reviews.

Experience of Involving Consumers

  1. Top of page
  2. Process of Involving Consumers
  3. Experience of Involving Consumers
  4. Future Evolution of the Pregnancy and Childbirth Group's Consumer Panel
  5. References

Over 50 consumers and consumer advocates have participated in the first two years of the Pregnancy and Childbirth Group's Consumer Panel. Although most participants live in the United Kingdom and United States, consumers from Australia, Canada, New Zealand, and Norway have also provided evaluations. Collectively, they have evaluated 46 protocols, and 40 prepublication reviews, with many others currently in process.

The number of consumers who have evaluated specific protocols has ranged from one to seven. At a minimum, the Principal Consumer Coordinator provides an assessment. Consumers have expressed the greatest interest in evaluating protocols in three general areas: (1) routine or widely used interventions that have an impact on large numbers of women and infants, (2) forms of care about which consumers have concerns or doubts, and (3) topics of particular interest to individual consumers due to personal experience with the condition or intervention that is addressed.

Participating consumers have clearly welcomed the opportunity for partnership and involvement, although panel members rarely have any institutional or economic support to take on this work. They volunteer to provide evaluations and nearly always honor the tight turnaround times for keeping the review development process on track. It is clear that many deeply appreciate the opportunity and become thoroughly engaged in the work. Some provide lengthy and detailed narrative assessments, in addition to completing the checklists. Even in the case of obscure or technically sophisticated review topics, invitations from the Consumer Coordinators almost always yield volunteers.

The initial pilot experience of the Consumer Panel affirms the Cochrane Collaboration's expectation that consumers can make substantive multidimensional contributions that improve the quality and relevance of systematic reviews. Moreover, the focus of many consumer concerns suggests that their involvement in fact extends and complements the perspectives of professionals. Some consumer concerns clearly present the “lay” perspective—for example, panel members often identify the need for explanation of terms and for consideration of additional outcomes that are of special interest to families. In addition, they have made many technically sophisticated contributions. For example, several evaluations questioned whether intrapartum co-interventions that have not been considered in the analyses may have affected the results. Due to their closeness to clinical settings, professionals may take the existence of other interventions for granted and may be less likely to raise this important methodological consideration.

We have been impressed by the range and substance of this feedback, and its considerable potential for strengthening the protocols and reviews. For a published example of the range of issues that Consumer Panel members may raise and the potential level of ensuing consumer-professional dialogue, see Hodges (8). Common concerns of consumers include the following:

  • Language of the protocols and reviews, for example, the need to:

    • Simplify and avoid jargon

    • Explain professional terms

    • Avoid wording that may be insensitive to consumers

    • Provide greater precision

    • Clarify apparent contradictions

  • Rationale for the reviews, for example, the need to:

    • Use high levels of evidence, including other Pregnancy and Childbirth Group reviews and books (9,10), to describe background considerations

    • Clarify the magnitude of the problem being addressed and its impact on mothers and babies

    • Distinguish conceptually different concerns and divide overly broad questions into two or more reviews

    • Distinguish between circumstances in more and less developed countries

    • Avoid prejudgments about the safety and effectiveness of the intervention under consideration

    • Avoid, in general, routine or widespread use of interventions among well women and infants

  • Interventions that should be included in the search and in future research, for example, the need to:

    • More systematically consider the impact of dietary and other lifestyle interventions

    • Establish appropriate comparison groups

  • Outcomes that should be included in the search and in future research, for example, the need to:

    • More systematically consider potential maternal and infant risks of interventions (both physical and psychosocial), and risks for further interventions

    • Consider mothers' attitudes and experiences

    • Consider implications for mother-infant relationships

    • Consider longer term outcomes

  • Review topic priorities, for example, the need to:

    • Give greater emphasis to understanding how to prevent problems

    • Give greater emphasis to understanding low-technology, low-risk solutions to problems

    • Give greater emphasis to understanding indications for interventions.

Future Evolution of the Pregnancy and Childbirth Group's Consumer Panel

  1. Top of page
  2. Process of Involving Consumers
  3. Experience of Involving Consumers
  4. Future Evolution of the Pregnancy and Childbirth Group's Consumer Panel
  5. References

We plan to refine the tools used to obtain consumer feedback. Oliver et al have conducted research to explore effective ways of involving consumers in the assessment of Cochrane Collaboration protocols and systematic reviews (Oliver S, Owen C, Oakley A, Unpublished research, 1998). The Consumer Panel will adapt and use several documents that this team developed: Guidelines for Consumers Interested in Peer-reviewing Research, Guidelines for Consumers Peer-reviewing Research, and Check-list for Relevance of Cochrane Reviews to Consumers.

The Pregnancy and Childbirth Group prepared and published many systematic reviews before its Consumer Panel was established. We have begun to extend the panel's work to include providing feedback on those earlier reviews. We are soliciting and providing consumer evaluations within the established process for updating the reviews. Authors prepare Cochrane reviews with the understanding that they are also expected to keep them up-to-date and take any expressed concerns or criticism into consideration. Present Cochrane Collaboration policy requires that that an update be completed within two years of the first or most recent date of publication.

We would like to involve a broader spectrum of consumers. When the pilot project began, resources did not permit this. We give priority to consumer input from less developed nations, and we are seeking one or more Consumer Coordinators who can help us reach this goal. Within more developed nations, we need to improve the job of obtaining the perspectives of women across the full socioeconomic spectrum and women of varying ethnic backgrounds. We are faced with the challenge that many who would be interested in commenting and have important perspectives and insights are hard for us to reach and have life circumstances that make participation difficult.

Two important changes in 2000 are enabling expansion. First, the Pregnancy and Childbirth editorial office funded the part-time position of the Principal Consumer Coordinator. This position allows support of more consumers, development of more sophisticated tracking systems, and provision of better recognition to the work than previously.

Second, in 1999 the Maternity Center Association of New York City established Maternity Wise™, a long-term program to promote evidence-based maternity care. Through a collaborative agreement with the Pregnancy and Childbirth Group, the Maternity Center Association now sponsors the North American branch of the Consumer Panel. The new Maternity Wise website (11) will soon make Consumer Panel resources available to Internet users throughout the world, with the potential of involving a much broader range of individuals. For example, we can create in advance a roster to help ensure our access to consumers with an interest in relatively esoteric maternity care topics and input for each protocol and review from individuals who have experienced the conditions and interventions that it addresses. Although hard-to-reach women often disproportionately lack access to the Internet, the Consumer Panel's Internet presence nonetheless may help us expand the demographic profile of participants.

Expanded outreach will require more explicit guidelines about eligibility for Consumer Panel participation. To date, we have been somewhat flexible about the question of who is a consumer, a consumer advocate, and a professional. For example, some childbirth educators and labor support companions (doulas) are interested in participating as consumers with personal experiences and as consumer advocates. As awareness of opportunities for participation grows, we will need to establish guidelines to determine whether to refer interested individuals to the Consumer Panel or the professional referee roster.

In addition, we would like to conduct a formal evaluation to assess the process and impact of the Pregnancy and Childbirth Group's Consumer Panel. For example, how do consumer evaluations of systematic reviews compare with evaluations provided by professionals? How do the professional authors of protocols and systematic reviews experience receiving feedback from consumers? What consumer feedback does and does not lead to changes in protocols and reviews? Do authors who have received feedback from the Consumer Panel develop a more general understanding of, and appreciation for, consumer concerns that influences their future research and other professional work? What is the impact of Consumer Panel participation on the panel members themselves? In what areas does Consumer Panel participation improve the quality of Pregnancy and Childbirth Group reviews?

The Cochrane Collaboration has identified the following 10 fundamental and worthy principles (3): fostering collaboration; building on the enthusiasm of individuals; avoiding duplication; minimizing bias; keeping up to date; ensuring relevance; ensuring access; continually improving the quality of work; ensuring continuity; enabling wide participation.

Through consumer involvement, the Pregnancy and Childbirth Group has expanded its team and strengthened collaboration, benefited from consumers' energy and enthusiasm, helped to reduce bias through broader input that contributes to “checks and balances,” helped to ensure that reviews address needs and interests of childbearing families and are conceptually accessible, incorporated important quality improvement mechanisms, and created a more open and inclusive process for carrying out its work.

We encourage other investigators to consider this experience and establish similar consumer partnerships—beginning with the identification of priority research questions and proposal development—to strengthen the quality, relevance, and impact of their work.


  1. Top of page
  2. Process of Involving Consumers
  3. Experience of Involving Consumers
  4. Future Evolution of the Pregnancy and Childbirth Group's Consumer Panel
  5. References