SEARCH

SEARCH BY CITATION

Keywords:

  • number needed-to-treat;
  • randomised controlled trials;
  • meta-analysis;
  • methodology

Abstract

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information

Aim: In translating clinical research into practice, the summarization of data from randomized trials in terms of measures of effect to be readily appreciated by the point-of-care clinicians is important. In this context, the body of literature highlighted the ‘number needed to treat’ as a useful measure. The objectives of our study were to assess how meta-analyses described number needed to treat and corresponding 95% CI, and to explore issues related to reporting number needed to treat in the selected meta-analyses.

Method: For an illustration, we searched for the Cochrane systematic reviews and non-Cochrane systematic reviews. Two-stage selection was done to identify eligible studies. First, we fixed a date and then, we searched meta-analyses in PUBMED available on the date fixed. Secondly, we purposively selected five Cochrane systematic reviews and three non-Cochrane systematic reviews, according to our inclusion criteria. The critical appraisal of meta-analyses identified for the current study was done with the 5-item quality checklist introduced to the current analysis.

Results: A total of 8 systematic reviews, 5 Cochrane systematic reviews and 3 non-Cochrane systematic reviews/meta-analyses, were identified for the present study. Of these 8 meta-analyses, some (50%; 4/8) described number needed to treat in the method session of the study. However, the majority (87.5%; 7/8) reported number needed to treat in the results. For the details, 80% in Cochrane reviews and 66.5% in non-Cochrane reviews reported number needed to treat in the results. Only two studies (25%; 2/8) reported susceptibility to publication bias, provided simplified interpretation or discussed number needed to treat.

Conclusion: Although the Cochrane handbook for systematic reviews of interventions suggests the reviewers to include number needed to treat in reporting effect estimations, there still is a need to improve.


Introduction

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information

A pivotal step in translating clinical research into practice is the summarization of data from randomized trials in terms of measures of effect that can be readily appreciated by the point-of-care clinicians (1). As such, the most commonly encountered effect measures used in clinical trials with dichotomous data are risk ratio (relative risk, RR), odds ratio (OR), risk difference (or absolute risk reduction, ARR), and number needed-to-treat (NNT) (2, 3). The former two represents relative measures, while the latter two are absolute measures. As a matter of fact, each effect measure has its merits and limits of statistical property. Of these, RR, OR, and ARR are used extensively in both clinical and epidemiological investigations. RR and OR are particularly useful when considering trade-off between likely benefits and likely harms of intervention (3), albeit with existence of debate between RR and OR, and is beyond the scope of this study. However, in clinical decision making, it is more meaningful to use the measure ‘NNT’. This measure is simply calculated on the reciprocal of the ARR, or 1/ARR (or 100/ARR if percentages are used rather than proportions) (1, 2, 4), for any trial, which has reported a binary outcome. When the treatment effect is not statistically significant, the 95% CI for the ARR spans zero, and one limit of the CI for the NNT will be negative. In this case the inverse of ARR is often termed as the number needed-to-treat-to-harm (NNH). Hence, in the case of positive 95% CI, it is better to be termed as number needed-to-treat-to-benefit (NNB) (1, 2). The 95% CI is simply by taking reciprocals of the values defining the CI for the ARR (2, 3) and it is recommended to be used in reporting effect measures in meta-analyses. For simplicity sake, we would prefer the term NNT, encompassing both NNB and NNH, which is often described in the published reviews. NNH reflects detrimental events indicating deterioration in outcome, while NNB is beneficial events of improvements in outcomes. The NNT is defined as the expected numbers of people who need to receive the experimental intervention rather than the comparator for one additional person to either incur or avoid an event in a given time frame (3). Computing NNT by converting from RR, OR, and ARR to NNT is described in detail elsewhere (1-3, 5). The illustration of this study is mainly concerned with conversion of ARR to NNT on the fixed follow-up periods. NNT on time-specific follow-ups is beyond the scope of this work. It is important to note the two unique characteristics of NNT in studies with dichotomous outcomes: (1) since the NNT is derived from the ARR, it is still a comparative measure of effect (ie, experimental versus control) and not a general property of a single intervention, and (2) the NNT is an expected value.

For dichotomous outcomes, the results should be, where possible, expressed as NNT with 95% CI and the baseline risk to which it applies (6). Moreover, the body of literature suggests NNT is applicable in assessing the likelihood of publication bias (ie, susceptibility to publication bias). This is because of the number of participants in studies with zero effect (relative benefit of one) would be needed to give a NNT too high to be clinically relevant (7).

Taking together, the objectives of our study were to assess how studies on meta-analyses described NNT and corresponding 95% CI, and to explore issues related to reporting NNT in the selected meta-analyses.

Methods

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information

Computation framework

Table 1 provides a computation framework of NNT, while Table 2 illustrates imputation of NNT using hypothetical data. In order to assess the susceptibility to publication bias, how many data (trials, participants) would be required both to be unpublished and to have zero treatment effect (relative risk/benefit 1) to make any result clinically irrelevant was estimated, using the NNT of predetermined utility. Table 3 demonstrates the stepwise calculation of susceptible publication bias using arbitrary NNT value (ie, predetermined utility value).

Table 1.  Illustration of the computation framework of NNT in meta-analyses
DescriptionExperimental drugaComparator or placebobRR (95% CI)ARR (95% CI)NNT (95% CI)
  1. RR = risk ratio; ARR = absolute risk reduction; NNT = number needed to treat. ni= group size in the study 1, 2, …,6.

  2. aNumber of participants with events/total participants in experimental group.

  3. bNumber of participants/total participants in control group.

Study 1A/n1E/n6   
Study 2B/n2F/n7   
Study 3C/n3G/n8   
Study 4D/n4H/n9   
Study 5E/n5I/n10   
Study 6E/n6I/n12   
Summary measures in meta-analysesX = (A + B + C + D + E)/ (n1+ n2+ n3+ n4+ n5+ n6)Y = (E + F + G + H + I)/(n7+ n8+ n9+ n10+ n11+ n12)X/YX – Y1/ (X – Y)
Table 2.  Illustration of number needed to treat in meta-analyses
 Experimental drug (Oral etoricoxib 120 mg)Comparator drug (Placebo)RR (95% CI)ARR (95% CI)NNTa (95% CI)
  1. Data source: Ref. (8).

  2. aNNTs in this case is NNB (‘number needed to treat to benefit) as derived from positive value of ARR.

  3. bNNT 1.7 implies that it is expected that almost one in two participants treated with etoricoxib 120 mg experienced this level of pain relief who would not have done so if treated with placebo.

Study 173/1003/25   
Study 273/9718/46   
Study 361/766/49   
Study 437/506/50   
Study 562/1001/50   
Study 626/8010/175   
Summary measures332/50334/2955.70.541.8b
   (4.18–7.93)(0.48–0.59)(1.5–2)

Selection of studies

Two-stage selection was done to identify eligible studies. First, we fixed a date (with regard to the convenient schedule of the authors of this study) and then, we identified meta-analyses in PubMed on the date fixed. In order to obtain the eligible studies on meta-analyses, we applied free search terms, ‘meta-analysis’ and ‘randomized controlled trials’. Second, we purposively selected five Cochrane systematic reviews and three non-Cochrane systematic reviews. The selection criteria were systematic reviews which included (1) quantitative synthesis; (2) sufficient data to calculate RR and ARR; (3) individual RCTs with pharmacological interventions; (4) only one review from each Cochrane group entity published in 2012; (5) only one article from each journal publication in 2012; (6) free access to the full articles; and (7) published in English.

Analyses

Two authors independently assessed the reporting of NNT in the selected meta-analyses. The critical appraisal of meta-analyses identified for this study was done with the 5-item quality checklist, based on literature review and consultations with the researchers. The 5-item scale is as follows: (1) Does the meta-analysis describe NNT appropriately in methods? (2) Is the meta-analysis reported NNT in results? (3) If reported, is the susceptibility to publication bias based on NNT included? (4) Is the NNT interpreted in simplification? (5) Does the meta-analysis comment NNT in discussion? We gave ‘yes’, ‘unclear’ (ie, incomplete), or ‘no’ to each item. For any discrepancy between the two authors, we resolved with discussion and by taking advice from the senior author. If NNT was described, we checked the reported NNT, following the process described in Table 2 and 3, using an example dataset (8). For this study, both data entry and data analysis were done with Excel spreadsheet. The findings of this study were reported according to the statement of strengthening the reporting of observational studies in epidemiology (STROBE; supporting material) (9).

Table 3.  Detection of zero treatment effect trials
StepActionDataEnter data
  1. Data source: Ref. (8).

  2. aPredetermined utility.

  3. bCalculating how many patients in zero treatment effect trials would make the NNT exceed predetermined utility.

1How many patients in the data set (RCT or meta- analyses)?A798
2What was the NNT obtained in the RCT or meta-analyses?B1.8
3What NNT value would be the limit of clinical utility or acceptability?C12a
   Calculation b
4Divide step 1 by step 2A/B = D434.7
5Multiply step 4 by step 3D × C = E5217
6Subtract step 1 from step 5E – A = F4419
(Extra patients needed for the NNT to move from calculated NNT to the set NNT)

Results

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information

A total of eight systematic reviews, comprising of five Cochrane systematic reviews (8, 10-13), and three non-Cochrane systematic reviews/meta-analyses (14-16), were identified for this study. Table 4 shows the description of the selected meta-analyses. Of eight meta-analyses, the majority (87.5%, 7/8) reported NNT in the results although only four studies (50%) described NNT in the method session. For the details, 80% (4/5) in Cochrane reviews and 66.7% (2/3) in non-Cochrane reviews reported NNT in the results. Only two studies (25%, 2/8) reported susceptible to publication bias, provided simplified interpretation or discussed NNT.

Table 4.  Characteristics of the selected studies
TypeStudy, year (experimental drug)Method reported (described correctly) (1)Result included ‘susceptible to publication bias’Simplified interpretation (4)Discussion (5)
(2)(3)
  1. Cochrane = Cochrane systematic review for intervention studies; rHuFVIIa = Recombinant human activated factor VII; UFH = unfractionated heparin; LMWH = low molecular weight heparin.

Cochrane(8) (oral etoricoxib)yesyesyesyesyes
Cochrane(10) (oral varenicline)noyesyesnoyes
Cochrane(11) (UFH, LMWH)yesyesnonono
Cochrane(12) (inositol)yesyesnonono
Cochrane(13) (rHuFVIIa)nonononono
Non-Cochrane(14) (rivaroxaban)noNononono
Non-Cochrane(15) (prophylatic antibiotics)noyesnoyesyes
Non-Cochrane(16) (lidocaine/tetracaine patch)yesyesyesunclearno

Discussion

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information

Although the Cochrane Handbook for Systematic Reviews of Interventions suggests the reviewers to compute NNT in reporting effect estimations, there still is space for improvement. On the other hand, we were aware of a merit of two reviews identified for this study (12, 16), reporting NNT in their abstracts. Information in abstract would reach a greater number of consumers as well as stakeholders. In the context of evidence-based medicine, with NNT increasingly used in comparative effectiveness studies of therapies, its accurate estimation and correct interpretation are crucial (17). In the current analysis, the Cochrane review of Cahill (10) had highlighted the important factors in imputation of NNT; “The number needed to treat to benefit to achieve each additional successful quitter can be derived from the pooled difference between placebo and treatment quit rates. However, absolute quit rates vary considerably between trials, according to the definition of cessation, length of follow-up, the population treated, and the extent of the counseling and follow-up support given.”

Of note is that if any meta-analysis did not compute NNTs, the reason might be due to varying follow-up times. For any meta-analysis computing NNTs, it must be based on fixed follow-up times. This was an important precaution in summarizing data in meta-analyses when individual trials with unequal follow-up times were identified. In intervention studies of pharmacology discipline, drug effects could accumulate overtime, describing the likelihood of changes of the effect. Studies have highlighted that the computation of the NNTs can, however, be inaccurate and its interpretation misleading in trials with varying follow-up times (17, 18). Despite intuitively appealing measure of the effect of a treatment, computation of NNT must be performed with care in trials with varying follow-up times (18).

Regarding the reporting of NNT, it was found that not all meta-analyses included in this study reported the corresponding 95% CIs of NNT. As a general principle, the CI describes the uncertainty inherent in estimates, and within which we can be reasonably sure that the true effect actually lies (3). If it is in fixed-effect model, its 95% CI addresses the question, “if there is a single population treatment effect across all the trials, what is the best estimate (and uncertainty) for this common treatment effect?” (19). If it is from random-effects model, it addresses the question, “what is the best estimate of the average effect?”

Regarding the interpretation of NNT, not all studies provided simplified interpretation of NNT. An NNT of 9, as an example, can be interpreted as “it is expected that one additional person will incur an event for every nine participants.” NNT = 9 does not imply that one additional event will occur in each and every group of nine people. In essence, a large treatment effect, in the absolute scale, leads to a small number NNT (2). A treatment that is expected to save one life for every nine patients treated is better than a competing treatment that saves one life for every 68 treated. In general, where efficacy is high, the requirement is for small numbers of patients. Where efficacy is low, the requirement is for large numbers of patients. With regard to unwanted effects, the NNH should be as large as possible when assessing the safety of the intervention.

When we assessed the strength of evidence for each outcome, the total number of participants contributing data, the methodological quality of studies and the degree of heterogeneity are crucially important criteria. It is also to be considered the number of additional participants needed in studies with zero effect (relative benefit of one) required to change (7), based on the arbitrary value of NNT. None of the meta-analyses reported susceptibility to publication bias, based on clinical utility of NNT, except the Clarke Cochrane Review (8). With regard to the NNT of predetermined utility (arbitrary NNT based on experts in some case), it was typically taken to be an NNT of 8 or greater in meta-analyses of pain related conditional analysis (7), assuming 10 for this study. We found that, in this study information on publication bias based on NNT was provided only in the Clarke review (8); “We carried out a comprehensive search for relevant studies, and investigated the potential influence of publication bias by examining the number of participants in studies with zero effect (relative risk of 1.0) needed for the point estimate of the NNT to increase beyond a clinically useful level. In this case, we chose a clinically useful level as eight. For the primary outcome of at least 50% pain relief with etoricoxib 120 mg, more than 5000 participants (ie, 5217, see Table 3) would have to have been involved in unpublished trials with zero treatment effects for the NNT to increase above this threshold. It is highly unlikely that this amount of missing data exists.” In the context of knowledge translation, understandable information would be valuable for the knowledge consumers, who are also point-of-care clinicians.

Overall, the findings of this study suggest that (1) if not all, many meta-analyses report NNT, (2) even if reported, information on NNT was inadequate, and (3) even if reported adequately, discussion on NNT was not done adequately. We acknowledge the caveats of our study. Being a cross sectional design, generalizability is limited. It is likely to miss meta-analyses with proper reporting of NNT. Further studies with larger samples are recommended.

Conclusion

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information

Our findings serve as timely reminder that NNT should be simply computed in equal follow-up times and its corresponding 95% CIs. NNT provide magnitude of the clinical effect which is important for point-of-care clinicians, and the healthcare decision makers in the context of the best utilization of resources

Acknowledgements

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information

We are grateful to the participants and the researchers of the primary studies used for the present analysis. We wish to thank the International Medical University (IMU), Malaysia for allowing us to perform this study.

References

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information
  • 1
    Cook RJ, Sackett DL. The number needed to treat: a clinically useful measure of treatment effect. British Medical Journal 1995; 310: 45254.
  • 2
    Altman DG, Gore SM, Gardner MJ, Pocock SJ. Statistical guidelines for contributors to medical journals. In: Gardner MJ, Altman DG, editor(s). Statis-tics with confidence. London : British Medical Journal 1989; 83100.
  • 3
    Higgins JPT, Green S. Principles of meta-analysis. In: Higgins JPT, Green S, editor(s). Cochrane handbook for systemic reviews of interventions. Cochrane Book Series. John Wiley and Sons Ltd, West Sussex. 2009.
  • 4
    McQuay H, Moore R. An Evidence-Based Resource for Pain Relief. Oxford : Oxford University Press, 1998.
  • 5
    Schechtman E. Odds ratio, relative risk, absolute risk reduction, and the number needed to treat-which of these should we use Value In Health 2002; 5: 43136.
    Direct Link:
  • 6
    McQuay HJ, Moore RA. Placebo. Postgraduate Medical Journal 2005; 81: 15560.
  • 7
    Moore RA, Barden J, Derry S, McQuay HJ. Managing potential publication bias. In: McQuay HJ, Kalso E, Moore RA, editor(s). Systematic reviews in pain research: methodology refined. Seattle : IASP Press, 2008: 1524.
  • 8
    Clarke R, Derry S, Moore RA. Single dose oral etoricoxib for acute postoperative pain in adults. Cochrane Database of Systematic Reviews 2012;(4). Art. No.: CD004309. doi: 10.1002/14651858.CD004309.pub3.
  • 9
    von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Bulletin of the World Health Organization 2007; 85: 86772.
  • 10
    Cahill K, Stead LF, Lancaster T. Nicotine receptor partial agonists for smoking cessation. Cochrane Database of Systematic Reviews 2012;(4). Art. No.: CD006103. doi: 10.1002/14651858.CD006103.pub6.
  • 11
    DiNisio M, Porreca E, Ferrante N, Otten HM, Cuccurullo F, Rutjes AWS. Primary prophylaxis for venous thromboembolism in ambulatory cancer patients receiving chemotherapy. Cochrane Database of Systematic Reviews 2012;(2). Art. No.: CD008500. doi: 10.1002/14651858.CD008500.pub2.
  • 12
    Howlett A, Ohlsson A, Plakkal N. Inositol for respiratory distress syndrome in preterm infants. Cochrane Database of Systematic Reviews 2012;(3). Art. No.: CD000366. doi: 10.1002/14651858.CD000366.pub2.
  • 13
    Martí-Carvajal AJ, Karakitsiou DE, Salanti G. Human recombinant activated factor VII for upper gastrointestinal bleeding in patients with liver diseases. Cochrane Database of Systematic Reviews 2012;(3). Art. No.: CD004887. doi: 10.1002/14651858.CD004887.pub3.
  • 14
    Antonio Gómez-Outes A, Ana Isabel Terleira-Fernández AI, Suárez-Gea ML, Emilio Vargas-Castrillón E. Dabigatran, rivaroxaban, or apixaban versus enoxaparin for thromboprophylaxis after total hip or knee replacement: systematic review, meta-analysis, and indirect treatment comparisons. British Medical Journal 2012; 344: e3675 doi: 10.1136/bmj.e3675
  • 15
    Kun Jiang, Wei Huang, Xiao-Nan Yang, Qing Xia. Present and future of prophylactic antibiotics for severe acute pancreatitis. World Journal of Gastroenterology 2012; 18: 27984.
  • 16
    Kim WO, Song BM, Kil HK. Efficacy and safety of a lidocaine/tetracaine medicated patch or peel for dermatologic procedures: a meta-analysis. Korean Journal of Anesthesiology 2012; 62: 43540.
  • 17
    Suissa D, Brassard P, Smiechowski B, Suissa S. Number needed to treat is incorrect without proper time-related considerations. Journal of Clinical Epidemiology, 2012; 65(1): 426.
  • 18
    Suissa S. Calculation of number needed to treat. New England Journal of Medicine 2009; 361(4): 4245.
  • 19
    Riley RD, Higgins JP, Deeks JJ. Interpretation of random effects meta-analyses. British Medical Journal 2011; 3(42): d549. doi: 10.1136/bmj.d549.

Supporting Information

  1. Top of page
  2. Abstract
  3. Introduction
  4. Methods
  5. Results
  6. Discussion
  7. Conclusion
  8. Acknowledgements
  9. References
  10. Supporting Information

Strengthening the Reporting of Observational studies in Epidemiology (STROBE).

FilenameFormatSizeDescription
JEBM_12002_sm_SuppMat.docx20KSupporting info item

Please note: Wiley Blackwell is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.