SEARCH

SEARCH BY CITATION

The practice of evidence-based medicine requires physicians to be familiar with the most relevant research published in the medical literature.[1] Individual randomized clinical trials are well suited to provide compelling evidence of an intervention's therapeutic benefit.[2, 3] However, it has become very difficult (arguably impossible) for physicians to read every publication of relevance to their particular specialty. Systematic reviews and meta-analyses have therefore become increasingly important. Systematic reviews are descriptive in nature, and “collate, compare, discuss, and summarize the current results” in a particular field.[4] Meta-analysis goes a step further, providing us with a statistical technique to combine results from multiple individual trials and then use this dataset to conduct a new analysis that we could not conduct on the basis of any of the individual trial's datasets.

Before starting our discussions, two general points are noteworthy. First, while the term meta-analysis is typically used in the literature to refer to the entire process of conducting such an analysis, we believe it does not adequately capture and emphasize the need for methodological rigor in the full array of actions required. Certainly, an analysis is conducted, and the term meta-analysis is entirely appropriate when discussing that segment of the process. However, determining the study reports from which we construct our new dataset, choosing the appropriate analytical model, and presenting our results with scientific and clinical decorum are also critically important. Indeed, the mathematical calculations are the easiest part of the overall process, exemplifying very well that the discipline of statistics is much more than mere number crunching.[5] We therefore believe that the term meta-methodology meaningfully describes the entire process, one part of which is conducting a meta-analysis.[6]

Second, as we have noted previously, it is unfortunate that, concurrently with the publication of their results in a high-profile medical journal, some meta-analysts disseminate their findings in the mass media “with a bravado that markedly departs from calm, scientific and clinical discourse, and seemingly with the expectation that the nation's physicians will change their practice of medicine overnight.”[6] As Turner and colleagues[7] noted, “In the era of sensationalist, sound-bite coverage, clinical science sadly falls very low on the list of points to be covered in the allotted 30 seconds of television coverage,” a point not unknown to meta-analysts who deliberately participate in this circus. Fortunately, many more meta-analysts provide physicians, and hence their patients, with appropriately presented information.

The Fundamentals of Meta-Methodology

  1. Top of page
  2. The Fundamentals of Meta-Methodology
  3. Two Case Studies
  4. Concluding Comments
  5. Disclosures
  6. References

Meta-methodology allows us to make a quantitative evaluation of the evidence provided by two or more individual trials that have addressed the same research question. It commonly involves the statistical combination of summary statistics, ie, mean treatment effects and the variances associated with those estimates, from various trials (study-level data), but it also refers to analyses performed on participant-level data, in which the raw data from each participant in each trial are used to create the new dataset. While the latter is always preferable, it can be challenging to access such data for a variety of reasons, including availability, the proprietary nature of the data and, in some cases, obtaining approval from an institutional review board. However, a recent paper by Persu and colleagues[8] addressing an interventional procedure, sympathetic renal denervation (RDN) for the treatment of drug-resistant hypertension, did report a meta-analysis of participant-level data: we review that analysis as case study 2 in due course.

Consider a typical scenario of wishing to employ meta-methodology to provide a more precise evaluation of a drug's treatment effect than is given by a single trial. The basic steps include the following:

  1. Establishing rules for whether the data from an identified study report will be incorporated into the new dataset.
  2. Identification of all pertinent studies in the literature (and, if possible, datasets that may have not been published for a variety of reasons).
  3. Data extraction (obtaining the treatment effect and its variance for each study to be included in the analysis).
  4. A quantitative test of the homogeneity of treatment effects among studies.
  5. Data analysis.
  6. Evaluation of the robustness of the results.
  7. Dissemination of results, interpretations, and conclusions.

The fourth step is necessary since the statistical theory underpinning meta-analysis assumes that the study-specific treatment effects are relatively homogeneous. A certain degree of heterogeneity among treatment effects and study characteristics can be addressed by choosing the appropriate statistical model, as discussed shortly. However, a higher degree of heterogeneity indicates that the researchers should pause for thought regarding the appropriateness of conducting a meta-analysis.

The sixth step involves conducting the analysis without the data from the largest study or studies to see if the result remains qualitatively similar. If so, the result of the primary analysis (including all data) is deemed robust. If not, it may not be appropriate to present the primary analysis alone and make potentially influential statements based on it.

Establishing the Studies and Data To Be Included

One straightforward approach is to include every study identified. A counterargument is that, almost certainly, some studies will be “better” than others, and that “less good” studies should perhaps not be included. In the latter case, strict a priori inclusion and exclusion criteria that operationally define entry into the analysis must be stated in advance of searching for studies.

Data Analysis: Fixed-Effects and Random-Effects Models

The goal of meta-methodology is to obtain a single estimate of the treatment effect and its variance on the basis of all data included in the meta-analysis. Two items of data are obtained from each study report: (1) a measure of the treatment effect in that study; and (2) the variance associated with the treatment effect, often operationalized as a two-sided 95% confidence interval (CI) placed around the treatment effect.

At the outset we must decide whether to employ a fixed-effects or a random-effects analysis model. These differ in the degree of influence each individual study's treatment effect is allowed to exert on the newly calculated treatment effect: this influence is operationalized by the weight assigned to each treatment effect. Both models use the precision of each study's treatment effect (higher precision, which is given more weight, is conveyed by narrower CIs) when assigning its weight. However, and very importantly, the random-effects model also uses an estimate of how different from each other the studies are in various characteristics such as the nature of the study population, number of participants in each treatment group, length of treatment periods, concomitant illnesses, and quality of measurements made during the trial. The greater the degree of difference between the studies incorporated in the analysis, the more important it becomes for us to employ the random-effects model.

The random-effects model tends to generate wider CIs around the treatment effect, indicating less precision. If a fixed-effects model is employed when there are considerable differences between the studies included, the CI will be narrower than it would have been had the random-effects model been used, and thus the confidence placed in the result of the analysis will be greater than it should be. Narrower CIs also make it easier to achieve a statistically significant result, an outcome used by some meta-analysts to ascribe more gravitas to their results than they deserve: consideration of a result's clinical significance is equally (and arguably more) important, and the clinical assessment of a treatment effect is “a completely separate assessment” from its statistical significance.[9]

Two Case Studies

  1. Top of page
  2. The Fundamentals of Meta-Methodology
  3. Two Case Studies
  4. Concluding Comments
  5. Disclosures
  6. References

The first of our case studies, conducted by DiNicolantonio and colleagues,[10] is a pharmaceutical study-level analysis comparing carvedilol with four β1 selective β-blockers. As part of the overall meta-methodological approach, 3 trials including 644 participants with acute myocardial ischemia and evaluating relative reductions in all-cause mortality were combined for meta-analysis. Of particular interest here is that the authors reported results using both a fixed-effects model and a random-effects model. For the fixed-effects model, the result was as follows:

  • display math

This result can be interpreted in this manner:

The result from this meta-analysis indicates a statistically significant reduction in all-cause mortality associated with carvedilol in the general population. The result is compatible with a reduction as great as 68% and as small as 6%, and our best estimate is a reduction of 45%.

The result for the random-effects model was as follows:

  • display math

For this model, the result is interpreted in this manner:

The result from this meta-analysis does not indicate a statistically significant reduction in all-cause mortality associated with carvedilol in the general population. The result is compatible with a reduction as great as 74% but also compatible with an increase as great as 12%. Our best estimate is a reduction of 44%.

The important point made by this example is that the best estimates of the truth in the general population provided by the fixed-effects and random-effects models are essentially the same (relative reductions in all-cause mortality of 45% and 44%), but the statements of the statistical significance of the results, something given far too much weight by some researchers and many in the media when the focus should be on clinical significance, are completely different as a result of the widths of the respective CIs. For the fixed-effects model, the lower and upper limits of the CI (0.32 and 0.94, respectively) both fall below 1, hence the attainment of statistical significance. For the random-effects model, the lower and upper limits (0.26 and 1.12, respectively) of this wider CI lie on either side of 1, hence the failure to attain statistical significance. We give credit to the authors for presenting the results generated by both analysis models.

The second case study is a participant-level analysis involving RDN, an intervention for drug-resistant hypertension currently attracting much attention in the hypertension literature.[11-13] One key question concerns which methodology, clinic blood pressure measurement (CBPM) or ambulatory blood pressure monitoring (ABPM), is more appropriate for evaluating the intervention's efficacy: Doumas and colleagues[14] and Turner and O'Brien[15] have reviewed several (relatively small) studies addressing this question. It is therefore of considerable interest that Persu and colleagues[8] recently published “the first subject-level meta-analysis of the 6-month responses of both office and ambulatory blood pressure to RDN in carefully selected patients in whom secondary hypertension was excluded and who had resistant hypertension confirmed by ambulatory monitoring.” Their participant-level (subject-level) meta-analysis employed a random-effects model (centers were the random factor) and was conducted using data from 109 individuals who had undergone RDN at 1 of 10 expert centers involved in the European Network COordinating Research on Renal Denervation (ENCOReD). The results are presented in the Table. Baseline data and respective decreases in blood pressure are shown for CBPM, 24-hour ABPM, daytime ABPM, and nighttime ABPM. The results of the meta-analysis therefore showed that the intervention's efficacy was considerably less when expressed in terms of reductions in ABPM than when expressed in CBPM terms.

Table 1. BP Baselines and Reductions at 6 Months[8]
BP Measurement MethodologyBaseline: SBP/DBP, mm HgDecrease at 6 Months: SBP/DBP, mm Hg (All Decreases Significant at P≤.03)
  1. Abbreviations: ABPM, ambulatory blood pressure (BP) monitoring; CBPM, clinic blood pressure measure.

CBPM174.5/9817.6/7.1
24-H ABPM156.7/91.55.9/3.5
Daytime ABPM160.8/94.96.2/3.4
Nighttime ABPM147/83.54.4/2.5

Concluding Comments

  1. Top of page
  2. The Fundamentals of Meta-Methodology
  3. Two Case Studies
  4. Concluding Comments
  5. Disclosures
  6. References

As is true across all research methodology, if rigorous meta-methodology has led to the acquisition of optimal quality data, implementation of the appropriate meta-analytical strategy, and transparent reporting, the resulting publication will be of optimal quality and benefit to physicians and their patients. The intellectual honesty of meta-analysts lies in their adherence to strict methodological and statistical rigor, appropriate interpretation of results, and adoption of the appropriate degree of restraint needed to disseminate conclusions in a responsible manner in the best interests of both individual patients and public health. Given all of these considerations, meta-methodology must be undertaken carefully, diligently, and responsibly.

Disclosures

  1. Top of page
  2. The Fundamentals of Meta-Methodology
  3. Two Case Studies
  4. Concluding Comments
  5. Disclosures
  6. References

The authors report no specific funding in relation to the preparation of this paper. No editorial support was used.

References

  1. Top of page
  2. The Fundamentals of Meta-Methodology
  3. Two Case Studies
  4. Concluding Comments
  5. Disclosures
  6. References
  • 1
    Bailar JC, Hoaglin DC, eds. Medical Uses of Statistics, 3rd ed. Hoboken, NJ: John Wiley & Sons; 2009.
  • 2
    Turner JR. The 50th anniversary of the Kefauver-Harris Amendments: efficacy assessment and the randomized clinical trial. J Clin Hypertens (Greenwich). 2012;14:810815.
  • 3
    Turner JR, Hoofwijk TJ. Clinical trials in new drug development. J Clin Hypertens (Greenwich). 2013;15:306309.
  • 4
    Matthews JNS. Introduction to Randomized Controlled Clinical Trials, 2nd ed. Boca Raton, FL: Chapman & Hall/CRC; 2006.
  • 5
    Turner JR. New Drug Development: An Introduction to Clinical Trials, 2nd ed. New York, NY: Springer; 2010.
  • 6
    Turner JR. Editor's commentary: additional associate editors, new submission category, and meta-methodology. Drug Inf J. 2011;45:221227.
  • 7
    Turner JR, Satin LZ, Callahan T, Litwin J. The science of cardiac safety. Appl Clin Trials. 2010;19(11 suppl):48, 14.
  • 8
    Persu A, Jin Y, Azizi M, et al; on behalf of the European Network COordinating research on REnal Denervation (ENCOReD). Blood pressure changes after renal denervation at 10 European expert centers. J Hum Hypertens. 2013 Sep 26. [Epub ahead of print]
  • 9
    Durham TA, Turner JR. Introduction to Statistics in Pharmaceutical Clinical Trials. London: Pharmaceutical Press; 2008.
  • 10
    DiNicolantonio JJ, Lavie CJ, Fares H, et al. Meta-analysis of carvedilol versus beta 1 selective beta-blockers (atenolol, bisoprolol, metoprolol, and nebivolol). Am J Cardiol. 2013;111:765769.
  • 11
    Grassi G, Seravalle G, Brambilla G, Mancia G. The sympathetic nervous system and new nonpharmacologic approaches to treating hypertension: a focus on renal denervation. Can J Cardiol. 2012;28:311317.
  • 12
    Gosain P, Garimella PS, Hart PD, Agarwal R. Renal sympathetic denervation for treatment of resistant hypertension: a systematic review. J Clin Hypertens (Greenwich). 2013;15:7584.
  • 13
    Gulati V, White WB. Review of the state of renal nerve ablation for patients with severe and resistant hypertension. J Am Soc Hypertens. 2013Aug 14; [Epub ahead of print].
  • 14
    Doumas M, Anyfanti P, Bakris G. Should ambulatory blood pressure monitoring be mandatory for future studies in resistant hypertension: a perspective. J Hypertens. 2012;30:874876.
  • 15
    Turner JR, O'Brien E. Diagnosis and treatment of resistant hypertension: the critical role of ambulatory blood pressure monitoring. J Clin Hypertens (Greenwich). 2013;15:868873.