Overview of methods for comparing the efficacies of drugs in the absence of head-to-head clinical trial data

Authors

  • Hansoo Kim,

    Corresponding author
    1. Melbourne EpiCentre, Department of Medicine (Royal Melbourne Hospital), The University of Melbourne, Parkville, VIC, Australia
    • Correspondence

      Mr Hansoo Kim MSc, Melbourne EpiCentre, Department of Medicine (RMH), The University of Melbourne, VIC 3010, Australia.

      Tel: +61 [0] 3 9342 8772

      Fax: +61 [0] 3 9342 8760

      E-mail: h.kim14@pgrad.unimelb.edu.au

    Search for more papers by this author
  • Lyle Gurrin,

    1. Centre for Molecular, Environmental, Genetic and Analytic Epidemiology, The University of Melbourne, Parkville, VIC, Australia
    Search for more papers by this author
  • Zanfina Ademi,

    1. Melbourne EpiCentre, Department of Medicine (Royal Melbourne Hospital), The University of Melbourne, Parkville, VIC, Australia
    2. Department of Epidemiology and Preventive Medicine, Monash University, Parkville, VIC, Australia
    Search for more papers by this author
  • Danny Liew

    1. Melbourne EpiCentre, Department of Medicine (Royal Melbourne Hospital), The University of Melbourne, Parkville, VIC, Australia
    Search for more papers by this author

Abstract

In most therapeutic areas, multiple drug options are increasingly becoming available, but there is often a lack of evidence from head-to-head clinical trials that allows for direct comparison of the efficacy and/or safety of one drug vs. another. This review provides an introduction to, and overview of, common methods used for comparing drugs in the absence of head-to-head clinical trial evidence. Naïve direct comparisons are in most instances inappropriate and should only be used for exploratory purposes and when no other options are possible. Adjusted indirect comparisons are currently the most commonly accepted method and use links through one or more common comparators. Mixed treatment comparisons (MTCs) use Bayesian statistical models to incorporate all available data for a drug, even data that are not relevant to the comparator drug. MTCs reduce uncertainty but have not yet been widely accepted by researchers, nor drug regulatory and reimbursement authorities. All indirect analyses are based on the same underlying assumption as meta-analyses, namely that the study populations in the trials being compared are similar.

Introduction

In most therapeutic areas, multiple drug options are increasingly becoming available, but there is often a lack of evidence from head-to-head clinical trials that allows for direct comparison of the efficacy (and/or safety) of one drug vs. another. The situation arises partly from drug registration in many worldwide markets being only reliant on demonstrated efficacy from placebo-controlled trials. Furthermore, trials with active comparators, especially those designed to show non-inferiority or equivalence of one drug vs. another, generally need large sample sizes, and hence are expensive to mount.

However, knowledge about the relative efficacies between (or among) various drugs is needed for decision making in clinical practice, as well as in public health and health policy.

An example is the treatment of type 2 diabetes mellitus (T2DM). The introduction of several new drug classes (notably glucagon-like peptide-1 [GLP-1] analogues and dipeptidyl peptidase 4 [DPP4] inhibitors) over the past several years has resulted in added complexity to therapeutic choice. In particular, it is not clear what the relative efficacies of the various agents are and hence where they fit in the overall treatment pathway. Very few GLP-1 analogues and DPP4 inhibitors have been compared in head-to-head studies. This uncertainty poses a challenge for clinicians, patients and health policy makers.

In the absence of head-to-head clinical trial data, several statistical methods have been developed over the past 15 years to allow for indirect comparison between or among various drugs (or other interventions). This manuscript will provide an overview of the four main methods, which are summarized in Figure 1 (in addition to head-to-head comparison). First, we briefly describe naïve direct comparisons, followed by a more detailed description of adjusted indirect comparisons, which are the most commonly used methods in drug registration and health technology assessment. These rely on comparisons against common comparators. Finally, two more complex methods that involve networks of drugs will be discussed: multiple adjusted indirect comparisons and mixed treatment comparisons.

Figure 1.

Conceptualization of the methods to compare the relative efficacies of drugs (or other interventions)

Naïve direct comparisons

Naïve direct comparison between two drugs refers to an assessment or analysis where clinical trial results for one drug are directly compared with clinical trial results for another drug. There is no attempt to adjust for any discordance in comparators between/among the trials. An example of naïve direct comparison is the meta-analysis by Sherifali et al. [1], who directly compared various oral anti-diabetic agents with respect to glycaemic control. The authors concluded that sulphonylureas and thiazolidinediones were the most efficacious drugs, but because no formal adjustment for the level of glycaemic control obtained by the control treatment was performed, the results should to be interpreted cautiously.

As illustrated in the example, the major limitation of the naïve direct comparison is that it is not possible to determine if any differences noted between the efficacy measures of different drugs can solely be attributable to the drugs themselves. Instead, the differences may reflect differences in other aspects of the various clinical trials, such as populations, comparators and outcomes. Conversely, the finding of similar efficacies may also be inaccurate, as variations in the trials may have masked a true difference in efficacies. As claimed by Bucher et al. [2], who initially proposed the methods for adjusted indirect comparisons (see below), naïve direct comparisons of results across different trials ‘break’ the original randomization and are subject to significant confounding and bias because of systematic differences between or among the trials being compared. This is despite the fact that each trial itself may have reduced confounding and bias through rigorous design. Therefore, naïve direct comparisons of randomized trials provide no more robust evidence than naïve direct comparisons of observational studies [2].

Adjusted indirect comparisons

Bucher et al. [2] presented a statistical method for making indirect comparisons of treatment effects that preserves the randomization of the originally-assigned patient groups. The method compares the magnitude of the treatment effect between two treatments relative to a common comparator, which serves as a link between the two treatments.

Assume two hypoglycaemic drugs, A and B, were compared with respect to reduction in blood glucose relative to a common comparator C. Drug A was compared with drug C in a head-to-head clinical trial and drug B with drug C in another. In an adjusted indirect comparison, using C as a common comparator, the difference between A and B would be estimated by comparing the difference between A and C with the difference between B and C. Hypothetical results are summarized in Table 1. In addition, the results of a naïve direct comparison of the same two trials are also included for comparison.

Table 1. Adjusted indirect comparison and naive direct comparison: hypothetical example using continuous outcomes. Two hypoglycaemic drugs, A and B, have been assessed with respect to reduction in blood glucose. Drug A was compared with drug C in a head-to-head clinical trial and drug B with drug C in another
 Clinical trial 1Clinical trial 2
ACBC
Observed change in blood glucose−3 mmol l−1−2 mmol l−1−2 mmol l−1−1 mmol l−1
Adjusted indirect comparison: A vs. B[(−3 mmol l−1) − (−2 mmol l−1)] − [(−2 mmol l−1) − (−1 mmol l−1)] = (−1 mmol l−1) − (−1 mmol l−1) = 0 mmol l−1
Naïve direct comparison: A vs. B(−3 mmol l−1) − (−2 mmol l−1) = −1 mmol l−1

The adjusted indirect comparison shows that there is no difference between A and B in terms of change in blood glucose after adjustment for the common comparator C. In contrast, the naïve direct comparison, which ignores the treatment effect relative to the comparator, shows a difference between A and B, and hence overestimates the treatment effect.

Adjusted indirect comparisons can also be undertaken when trial outcomes are binary (yes/no). Assume that the two hypothetical trials had also included attaining glycosylated haemoglobin (HbA1c) < 7.0% as an outcome of interest. Table 2 summarizes hypothetical results of the individual trials, as well as the results of adjusted indirect and naïve direct comparisons between the two trials.

Table 2. Adjusted indirect comparison and naive direct comparison: hypothetical example using binary outcomes. Two hypoglycaemic drugs, A and B, have been assessed with respect to reduction in blood glucose. Drug A was compared with drug C in a head-to-head clinical trial and drug B with drug C in another
 Clinical trial 1Clinical trial 2
ACBC
Patients reaching HbA1c < 7.0%30%15%20%10%
Adjusted indirect comparison: A vs. B (relative risk for reaching HbA1c < 7.0%)(30% /15%) / (20%/10%) = 2.0/2.0 = 1.0
Naïve direct comparison: A vs. B (relative risk for reaching HbA1c < 7.0%)30%/20% = 1.5

The naïve direct comparison shows that patients treated with drug A have a 50% higher chance of reaching HbA1c < 7.0% (relative risk = 1.5). However, an adjusted indirect comparison shows no difference between A and B given that the relative difference of both drugs compared with C is the same.

Adjusted indirect comparison is a method accepted by drug reimbursement agencies such as the Australian Pharmaceutical Benefits Advisory Committee (PBAC) [3], the UK National Institute of Clinical Excellence (NICE) [4] and the Canadian Agency for Drug and Technologies in Health (CADTH) [5]. CADTH has even made available simple software for undertaking adjusted indirect comparisons [6]. However, among leading drug regulatory bodies, only the US Food and Drug Administration (FDA) [7] specifically mentions adjusted indirect comparisons in its guidelines.

The main disadvantage of adjusted indirect comparisons is the uncertainty with which they are associated. This is because the statistical uncertainties of the component comparison studies are summed. A graphical example is provided in Figure 2, which describes the results (with uncertainties) of the two hypothetical head-to-head trials illustrated in Table 1. The differences in pairwise comparisons in terms of change to blood glucose are as follows: A vs. C: −1 mmol l−1 (variance = 1 mmol l−1) and B vs. C: −1 mmol l−1 (variance = 1 mmol l−1). That is, the uncertainty of each trial was ± 1 mmol l−1 as measured by the variance. An adjusted indirect comparison between A and B using C as a common comparator would find no difference in the point estimate [(−1 mmol l−1) – (−1 mmol l−1)] = 0 mmol l−1, but the uncertainty would be greater, these being the sum of the variance in the individual head-to-head trials, ± 2 mmol l−1. Hence the result of the adjusted indirect comparison of A vs. B would be 0 mmol l−1 (variance = 2 mmol l−1).

Figure 2.

Illustration of uncertainty in a hypothetical adjusted indirect comparison. Two hypoglycaemic drugs, A and B, have been assessed with respect to reduction in blood glucose. Drug A was compared with drug C in a head-to-head clinical trial and drug B with drug C in another. Point estimates and uncertainty ranges of the differences in blood glucose change are illustrated. In the adjusted indirect comparison of A vs. B using C as a common comparator, the uncertainties in the pairwise comparisons of A vs. C and B vs. C are additive on the mmol l−1 scale

When no common comparator can be identified between two drugs of interest, a series may be constructed whereby the two drugs are linked indirectly via two or more comparators. The scheme is illustrated in Figure 1 (fourth panel). Drug A has been directly compared with drug C in a randomized trial, and drug B with drug D. Drugs C and D have also been directly compared in a randomized trial. In this situation, an indirect comparison of A vs. B can be undertaken via their direct links to C and D, respectively, as well as the direct link between C and D.

Kim et al. [8] performed a multiple adjusted indirect comparison in order to compare sitagliptin with insulin in T2DM with respect to change in HbA1c. As sitagliptin had only been compared with placebo and insulin had only been compared with exenatide, a connecting trial comparing exenatide with placebo was required. The results of this multiple indirect comparison are presented in Table 3.

Table 3. Example of a multiple adjusted indirect comparison Kim et al. [9]). In order to compare sitagliptin with insulin in T2DM with respect to change in HbA1c, links were made via sitagliptin vs. placebo, exenatide vs. placebo and insulin vs. exenatide
 Sitagliptin vs. placeboExenatide vs. placeboInsulin vs. exenatide
  1. CI, confidence interval.
Results of the head-to-head clinical trials: (absolute change in HbA1c)0.57 (95% CI: 0.45, 0.72)0.52 (95% CI: 0.32, 0.92)0.40 (95% CI: 0.27, 0.60)
Adjusted indirect comparison: sitagliptin vs exenatide using placebo as common comparator0.30 (95% CI: 0.17, 0.53)
Adjusted indirect comparison: sitagliptin vs insulin using exenatide as common comparator0.74 (95% CI: 0.40, 1.37)

With multiple indirect comparisons, uncertainty accumulates at every link. In the case of the analysis by Kim et al. [8], uncertainty was additive for both the adjusted indirect comparison of sitagliptin vs. exenatide and of sitagliptin vs. insulin. Therefore multiple adjusted indirect comparisons are generally associated with significant uncertainty, the extent of which correlates with the number of links in the series.

The key underlying assumption with adjusted indirect comparisons is that subjects recruited into the different studies being ‘linked’ via the common comparator are similar enough to be ‘pooled’. This is akin to the pooling of the results of different studies in a meta-analysis.

Reasons for not performing adjusted indirect comparisons can be based on differences in treatment populations, as was the case in the analysis of metastatic renal cell carcinoma by Mills et al. [9], in which one intervention (temsirolimus) was excluded as patients prescribed this agent differed from the others.

Mixed treatment comparisons

The concept and method for mixed treatment comparison (MTC) was introduced by Lu & Ades in 2005 [10], using a family of statistical models called Bayesian models. Figure 1 (fifth panel) provides a graphical example of a MTC. The concept is that any comparison that includes either one of two drugs being compared contains information that can be used to describe the link between the pair. For example, to establish the relative efficacy between drug A and drug B, any trial which included either A or B will provide information for the comparison between A and B, regardless of what the comparators were.

As an example, Ribeiro et al. [11] used various methods to investigate the impact of statin dose (high, intermediate or low) on major cardiovascular events using evidence from 47 trials. The clinical trials had compared particular dose categories of statin (low, intermediate and high) with either placebo or another dose category. The results for each of the dose categories were calculated using random effect models and meta-analyses showed low heterogeneity. Direct comparisons (except for intermediate dose vs. low dose), adjusted indirect comparisons and MTCs were all undertaken.

Figure 3 conceptualizes the comparisons and the results are summarized in Table 4. The only comparison without direct evidence was intermediate dose vs. low dose statins. The results from MTCs are different from those of adjusted indirect comparisons because the former incorporate data from any study that involves either of the pair being compared. For example, the MTC of high vs. intermediate dose statin incorporated not only the direct data between the high and intermediate doses, but also indirect data involving other comparisons with high and intermediate doses.

Figure 3.

Conceptualization of a mixed treatment comparison undertaken by Ribeiro et al. [11], who investigated the impact of various statin doses (high, intermediate or low) and placebo on major cardiovascular events using evidence from 47 trials. Lines indicate the intervention groups that had been directly compared in the trials

Table 4. Results of a direct comparison, adjusted indirect comparison and mixed treatment comparison of various dose categories of statins and the relative risk of stroke: key results from Ribeiro et al. [11])
 High dose statins vs. low dose statinsHigh dose statins vs. intermediate dose statinsIntermediate dose statins vs. low dose statins
  1. CI, confidence interval.
Direct comparison1.00 (95% CI: 0.56, 1.79)0.86 (95% CI: 0.77, 0.96)Not available
Adjusted Indirect comparison0.87 (95% CI: 0.66, 1.14)1.02 (95% CI: 0.78, 1.35)0.85 (95% CI: 0.82, 0.87)
Mixed treatment comparison0.83 (95% CI: 0.68, 0.99)0.91 (95% CI: 0.80, 1.04)0.91 (95% CI: 0.76, 1.09)

The major advantage of MTCs over adjusted indirect analyses is that they accommodate direct as well as indirect information about the treatment effects of drugs. That is, they consider the totality of evidence. Furthermore, as a result of additional data being incorporated and because of the underlying assumptions of Bayesian theory (which are beyond the scope of this review), uncertainty associated with MTCs is generally less than that of adjusted indirect comparisons. Hence even if two drugs are ‘connected’ only remotely via a long string of linking comparators, their relative efficacies can be estimated without significant uncertainty.

The major disadvantage of MTCs lies in the complexity of Bayesian models, which involve advanced statistical methods and obviously sufficient theoretical and technical know-how. At present, relatively limited expertise exists in MTC, but this is growing. The Medical Research Council Biostatistics Unit from Cambridge University is among the leaders in the field of Bayesian statistics, and has developed free software called ‘WinBUGS’ that can be used for undertaking MTCs [12]. Other software packages such as ‘R’, ‘Splus’ and ‘Stata’ can also be used to perform MTCs.

Furthermore, MTCs, like all pooled analyses, should only combine the results of similar trials. Non-statistical heterogeneity (e.g., differences in study settings) cannot be quantified and hence investigators need to make subjective assessments of which studies to pool.

There is no universal acceptance of the MTC method (yet), which reflects that it is relatively new. Indeed, none of the drug regulatory bodies around the world, including the US FDA and the European Medicines Agency (EMEA), mention MTCs in their guidelines. Drug reimbursement agencies around the world have different views. CADTH [6] and NICE [4] both accept MTCs as part of drug reimbursement submissions, whereas the PBAC [3] remains uncertain. However, the results of MTCs are commonly published in leading journals.

Conclusion

Different methods are available for comparing drugs in the absence of head-to-head clinical trial evidence. Naïve direct comparisons are in most instances inappropriate. Adjusted indirect comparisons are currently the most commonly accepted method, and use links through one or more common comparators. MTCs use Bayesian statistical models to incorporate all available data for a drug, even data not pertaining to the specific pairwise comparison of interest. MTCs reduce uncertainty but have not yet been widely accepted by researchers, nor drug regulatory and reimbursement authorities. All indirect analyses are based on the underlying assumption that the study populations in the trials being compared are similar enough to be pooled, akin to meta-analyses.

Competing Interests

All authors have completed the Unified Competing Interest form at http://www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare HK had scholarship support from the Australian National Health and Medical Research Council (NHMRC) for the submitted work, the authors have no financial relationships with any organizations that might have an interest in the submitted work in the previous 3 years and the authors have no other relationships or activities that could appear to have influenced the submitted work.

Ancillary