SEARCH

SEARCH BY CITATION

A response to ‘The effectiveness of public health nursing: the problems and solutions in carrying out a review of systematic reviews’ by L. Elliot, I.K. Crombie, L. Irvine, J. Cantrell & J. Taylor (2003) Journal of Advanced Nursing45(2), 117–125

  1. Top of page
  2. A response to ‘The effectiveness of public health nursing: the problems and solutions in carrying out a review of systematic reviews’ by L. Elliot, I.K. Crombie, L. Irvine, J. Cantrell & J. Taylor (2003) Journal of Advanced Nursing45(2), 117–125
  3. Scoping a systematic review: the review question
  4. Critically appraising systematic reviews
  5. Qualitative findings as evidence
  6. Conclusion
  7. References

In their methodological paper, ‘The effectiveness of public health nursing: the problems and solutions in carrying out a review of systematic reviews’, Elliot et al. accurately describe the difficulties faced in systematically reviewing research evidence — whether that be in the form of individual primary research reports or completed systematic reviews. Their general observations on the systematic review process itself will ring true with other reviewers. Anyone who has conducted a review will concur with the observations made on the difficulty of refining the search strategy so that it balances the need to maximize the results and minimize bias with the need to screen out irrelevant and poor quality papers. So, too, would any reviewer recognize the enormity of conducting a critical appraisal of over 300 systematic review reports!

Elliot et al. recognize the limitations of the process they followed in their review and these are well articulated in their paper. A number of issues arising out of their paper represent areas for debate, and these are identified and discussed in this JAN Forum response.

Scoping a systematic review: the review question

  1. Top of page
  2. A response to ‘The effectiveness of public health nursing: the problems and solutions in carrying out a review of systematic reviews’ by L. Elliot, I.K. Crombie, L. Irvine, J. Cantrell & J. Taylor (2003) Journal of Advanced Nursing45(2), 117–125
  3. Scoping a systematic review: the review question
  4. Critically appraising systematic reviews
  5. Qualitative findings as evidence
  6. Conclusion
  7. References

Ciliska et al. 2001) state that ‘A systematic review…is a rigorous summary of all the research evidence that relates to a specific question…’ (p. 100). The review question – although not specifically presented in Elliot et al.'s paper – is, as the authors acknowledge, extremely broad and covers 14 ‘health topics’ (p. 3). Generally speaking, although health professionals often want answers to very broad questions, the fact is that the narrower a question is, the easier it is to conduct a review. It is now well accepted that a sound, systematic review rests on a well defined review question. If, in this case, the reviewers were interested in establishing the relationship between the role of the nurse in improving the public's health, it may have been more effective and feasible to have conducted a series of reviews based on specific, focused questions (related to, for example, defined client groups, health outcomes or public health nursing roles) rather than a broad, all-encompassing review that included different populations, interventions and outcomes.

However, Elliot et al. are reporting on a commissioned review where the scope of the task was largely predetermined by those who commissioned it. The increasing acceptance of the systematic review process as a way of informing policy and service planning by governments, policy makers and planners in health systems across the world has led to a rise in the commissioning of projects that apply the systematic review process to broad and complex questions. But are the existing orthodoxies of the systematic review process – as it is currently conceived – appropriate to such broadly conceived questions?

Critically appraising systematic reviews

  1. Top of page
  2. A response to ‘The effectiveness of public health nursing: the problems and solutions in carrying out a review of systematic reviews’ by L. Elliot, I.K. Crombie, L. Irvine, J. Cantrell & J. Taylor (2003) Journal of Advanced Nursing45(2), 117–125
  3. Scoping a systematic review: the review question
  4. Critically appraising systematic reviews
  5. Qualitative findings as evidence
  6. Conclusion
  7. References

The authors describe the difficulties they encountered in critically appraising the reviews they found from their search of the literature, stating that ‘(j)udging the quality of reviews was often very difficult…’ and, further, that ‘(t)here was always the tension between the need to focus on high quality studies and the danger of an overzealous application of quality criteria that could lead to the exclusion of relevant reviews’(p. 5). The critical appraisal process is, of course, one of the most difficult components in a systematic review of the available evidence. Many checklists have been developed for use by appraisers and the major aim of critical appraisal is to establish the validity of the reported results (Cileska et al. 2001 and Joanna Briggs Institute 2004). Reviewers include in their report the criteria they used to appraise the studies included in their review and readers therefore can take these into account when making their own judgement of the review's findings. However, reviewers rarely describe the degree to which the criteria are applied when making decisions about what to include or exclude; and, to some extent, there is always an element of judgement involved when these decisions are being made. Should greater explication of how these judgements are made be part of a systematic review report?

Systematic Reviews seek to minimize bias, rather than exclude it entirely, and the strength of the approach is that reports are transparent by presenting an account of the process. Currently, many reviewers develop appraisal instruments for each specific review. Would credibility and transferability benefit from a standardization of criteria across reviews?

Qualitative findings as evidence

  1. Top of page
  2. A response to ‘The effectiveness of public health nursing: the problems and solutions in carrying out a review of systematic reviews’ by L. Elliot, I.K. Crombie, L. Irvine, J. Cantrell & J. Taylor (2003) Journal of Advanced Nursing45(2), 117–125
  3. Scoping a systematic review: the review question
  4. Critically appraising systematic reviews
  5. Qualitative findings as evidence
  6. Conclusion
  7. References

Elliot et al. discuss the possibility that systematic reviews, as they are currently conducted, may ‘miss’ evidence that is either generated or appraised using qualitative techniques. Knowledge acquired from qualitative approaches to research is largely rendered invisible in current approaches to systematic reviews. Questions such as ‘What is evidence?’ and ‘What are acceptable research results in terms of generating knowledge that amounts to evidence for the purpose of informing practice?’ inspire conflicting views. These differences generally align with the various positions that characterize the longstanding debate between qualitative and quantitative researchers. This is clearly not an easily resolved argument, but it is vitally important in terms of ascertaining the value of research-generated ‘evidence’ to nursing and healthcare practice.

I have argued for a pluralistic approach when considering ‘what counts as evidence’ for health care practices (Pearson 1999) and Evans & Pearson 2001) proposed that reviews that include both (or either) qualitative evidence and quantitative evidence are of importance to most practitioners. They go on to suggest that ‘… optimal methods for reviewing qualitative research are still evolving’ (p. 1). There are signs, however, that the Evidence Based Practice movement is beginning to adopt a more comprehensive view of what counts as evidence. Research initiatives are attempting to construct approaches to assessing and synthesizing the results of interpretive and critical research, so that these forms of evidence can become an integral part of systematic reviews and, thus, inform practice (Popay & Roen 2003, Pearson 2004). I have alluded to this work in a recent Guest Editorial in JAN (Pearson 2003).

The well-established approach to Systematic Reviews developed by the Cochrane Collaboration is vital to our continuing efforts to improve health outcomes by applying the best evidence generated through the randomised controlled trial and other designs that attempt to objectively measure the relationship between interventions and outcomes. However, nurses, medical practitioners and allied health professionals are concerned with more than ‘cause and effect’ questions and this is clearly the central problem described by Elliot et al. Their interest lay in establishing the role of the nurse in public health programmes; a question that could only ever be partly addressed using quantitative research approaches. The diverse origins of problems in public health require a broad interpretation of what counts as valid evidence for practice, and the utilization of a diverse range of research methodologies is needed to generate appropriate evidence.

Conclusion

  1. Top of page
  2. A response to ‘The effectiveness of public health nursing: the problems and solutions in carrying out a review of systematic reviews’ by L. Elliot, I.K. Crombie, L. Irvine, J. Cantrell & J. Taylor (2003) Journal of Advanced Nursing45(2), 117–125
  3. Scoping a systematic review: the review question
  4. Critically appraising systematic reviews
  5. Qualitative findings as evidence
  6. Conclusion
  7. References

Elliot et al.'s paper illuminates the complexities surrounding the systematic review of evidence in health care. It raises issues worthy of further debate. Evidence based practice will be enhanced and strengthened by vigorous debate on how the methods for Systematic Review can be further developed. Much more investment needs to go into the task of developing new and more effective methods of extracting the evidence from all of the research that has been carried out in nursing and health care.

References

  1. Top of page
  2. A response to ‘The effectiveness of public health nursing: the problems and solutions in carrying out a review of systematic reviews’ by L. Elliot, I.K. Crombie, L. Irvine, J. Cantrell & J. Taylor (2003) Journal of Advanced Nursing45(2), 117–125
  3. Scoping a systematic review: the review question
  4. Critically appraising systematic reviews
  5. Qualitative findings as evidence
  6. Conclusion
  7. References