Assessing the performance of population adjustment methods for anchored indirect comparisons: A simulation study

Standard network meta‐analysis and indirect comparisons combine aggregate data from multiple studies on treatments of interest, assuming that any factors that interact with treatment effects (effect modifiers) are balanced across populations. Population adjustment methods such as multilevel network meta‐regression (ML‐NMR), matching‐adjusted indirect comparison (MAIC), and simulated treatment comparison (STC) relax this assumption using individual patient data from one or more studies, and are becoming increasingly prevalent in health technology appraisals and the applied literature. Motivated by an applied example and two recent reviews of applications, we undertook an extensive simulation study to assess the performance of these methods in a range of scenarios under various failures of assumptions. We investigated the impact of varying sample size, missing effect modifiers, strength of effect modification and validity of the shared effect modifier assumption, validity of extrapolation and varying between‐study overlap, and different covariate distributions and correlations. ML‐NMR and STC performed similarly, eliminating bias when the requisite assumptions were met. Serious concerns are raised for MAIC, which performed poorly in nearly all simulation scenarios and may even increase bias compared with standard indirect comparisons. All methods incur bias when an effect modifier is missing, highlighting the necessity of careful selection of potential effect modifiers prior to analysis. When all effect modifiers are included, ML‐NMR and STC are robust techniques for population adjustment. ML‐NMR offers additional advantages over MAIC and STC, including extending to larger treatment networks and producing estimates in any target population, making this an attractive choice in a variety of scenarios.


FIGURE B2
Coverage zip plots for the ( ) contrast estimate for scenario a. One of the two effect modifiers was not adjusted for. Sample size is varied between 100, 500, and 1000. The 95% confidence/credible intervals are coloured as coverers (green) or non-coverers (purple), and the colour change should occur at the 95th centile (i.e. nominal coverage). The horizontal dashed lines are 95% Monte Carlo confidence intervals for the coverage.

B.2 Scenario b FIGURE B3
Coverage zip plots for the ( ) contrast estimate for scenario b. Strength of effect modification is varied from weak (0.1 change in log odds ratio per covariate standard deviation in the study) to strong (0.5 change in log odds ratio per covariate standard deviation in the study). Each method (other than Bucher) adjusts for the full set of effect modifiers. The 95% confidence/credible intervals are coloured as coverers (green) or non-coverers (purple), and the colour change should occur at the 95th centile (i.e. nominal coverage). The horizontal dashed lines are 95% Monte Carlo confidence intervals for the coverage.  FIGURE B4 Bias (a) and standard errors (b) for the population-average contrast estimates for scenario b, along with 95% Monte Carlo confidence intervals. Strength of effect modification is varied from weak (0.1 change in log odds ratio per covariate standard deviation in the study) to strong (0.5 change in log odds ratio per covariate standard deviation in the study). One of the two effect modifiers was not adjusted for. The points are coloured by contrast, with lighter shades for the population and darker for the population.

FIGURE B5
Coverage zip plots for the ( ) contrast estimate for scenario b. Strength of effect modification is varied from weak (0.1 change in log odds ratio per covariate standard deviation in the study) to strong (0.5 change in log odds ratio per covariate standard deviation in the study). One of the two effect modifiers was not adjusted for. The 95% confidence/credible intervals are coloured as coverers (green) or non-coverers (purple), and the colour change should occur at the 95th centile (i.e. nominal coverage). The horizontal dashed lines are 95% Monte Carlo confidence intervals for the coverage.

B.3 Scenario c FIGURE B6
Coverage zip plots for the ( ) contrast estimate for scenario c. Each method (other than Bucher) adjusts for the full set of effect modifiers. The shared effect modifier assumption is broken, so that treatment is subject to weak effect modification whilst treatment is subject to strong effect modification and vice versa. The 95% confidence/credible intervals are coloured as coverers (green) or non-coverers (purple), and the colour change should occur at the 95th centile (i.e. nominal coverage). The horizontal dashed lines are 95% Monte Carlo confidence intervals for the coverage.    FIGURE B7 Bias (a) and standard errors (b) for the population-average contrast estimates for scenario c, along with 95% Monte Carlo confidence intervals. One of the two effect modifiers was not adjusted for. The shared effect modifier assumption is broken, so that treatment is subject to weak effect modification whilst treatment is subject to strong effect modification and vice versa. The points are coloured by contrast, with lighter shades for the population and darker for the population.

FIGURE B8
Coverage zip plots for the ( ) contrast estimate for scenario c. One of the two effect modifiers was not adjusted for. The shared effect modifier assumption is broken, so that treatment is subject to weak effect modification whilst treatment is subject to strong effect modification and vice versa. The 95% confidence/credible intervals are coloured as coverers (green) or non-coverers (purple), and the colour change should occur at the 95th centile (i.e. nominal coverage). The horizontal dashed lines are 95% Monte Carlo confidence intervals for the coverage.

B.4 Scenario d FIGURE B9
Joint covariate distributions in the and study, as the correlation between covariates is varied. The true overlap is defined as the proportion of the joint density contained within the 95% HDR of the joint density, calculated using numerical integration.

FIGURE B10
Coverage zip plots for the ( ) contrast estimate for scenario d. The correlation between covariates is varied between 0, 0.25, and 0.5. Each method (other than Bucher) adjusts for the full set of effect modifiers. The 95% confidence/credible intervals are coloured as coverers (green) or non-coverers (purple), and the colour change should occur at the 95th centile (i.e. nominal coverage). The horizontal dashed lines are 95% Monte Carlo confidence intervals for the coverage.

FIGURE B11
Bias (a) and standard errors (b) for the population-average contrast estimates for scenario d, along with 95% Monte Carlo confidence intervals. The correlation between covariates is varied between 0, 0.25, and 0.5. One of the two effect modifiers was not adjusted for. The points are coloured by contrast, with lighter shades for the population and darker for the population.

FIGURE B12
Coverage zip plots for the ( ) contrast estimate for scenario d. The correlation between covariates is varied between 0, 0.25, and 0.5. One of the two effect modifiers was not adjusted for. The 95% confidence/credible intervals are coloured as coverers (green) or non-coverers (purple), and the colour change should occur at the 95th centile (i.e. nominal coverage). The horizontal dashed lines are 95% Monte Carlo confidence intervals for the coverage.

TABLE B8
Simulation results for scenario d, only adjusting for one of two effect modifiers. The correlation between covariates is varied between 0, 0.25, and 0.5. Monte Carlo standard errors for each statistic are shown in brackets.

Method
Contrast     FIGURE B14 Bias (a) and standard errors (b) for the population-average contrast estimates for scenarios e and f, along with 95% Monte Carlo confidence intervals. One of the two effect modifiers was not adjusted for. The between-study overlap and covariate-outcome relationship are varied jointly. The points are coloured by contrast, with lighter shades for the population and darker for the population.

FIGURE B15
Coverage zip plots for the ( ) contrast estimate for scenarios e and f. One of the two effect modifiers was not adjusted for. The between-study overlap and covariate-outcome relationship are varied jointly. The 95% confidence/credible intervals are coloured as coverers (green) or non-coverers (purple), and the colour change should occur at the 95th centile (i.e. nominal coverage). The horizontal dashed lines are 95% Monte Carlo confidence intervals for the coverage.

FIGURE B16
Bias in the population-average contrast estimates for scenarios g, h, and i, along with 95% Monte Carlo confidence intervals. Each method (other than Bucher) adjusts for the full set of effect modifiers. The covariate distributions and correlation structures in each study population are varied jointly. The points are coloured by contrast, with lighter shades for the population and darker for the population.

FIGURE B17
Empirical and model standard errors for scenarios g, h, and i, along with 95% Monte Carlo confidence intervals. Each method (other than Bucher) adjusts for the full set of effect modifiers. The covariate distributions and correlation structures in each study population are varied jointly. The points are coloured by contrast, with lighter shades for the population and darker for the population.

FIGURE B18
Joint covariate distributions in the and study, as the covariate distributions and correlation between covariates are varied. The true overlap is defined as the proportion of the joint density contained within the 95% HDR of the joint density, calculated using numerical integration.

FIGURE B19
Coverage zip plots for the ( ) contrast estimate for scenarios g, h, and i. Each method (other than Bucher) adjusts for the full set of effect modifiers. The covariate distributions and correlation structures in each study population are varied jointly. The 95% confidence/credible intervals are coloured as coverers (green) or non-coverers (purple), and the colour change should occur at the 95th centile (i.e. nominal coverage). The horizontal dashed lines are 95% Monte Carlo confidence intervals for the coverage.

FIGURE B20
Bias in the population-average contrast estimates for scenarios g, h, and i, along with 95% Monte Carlo confidence intervals. One of the two effect modifiers was not adjusted for. The covariate distributions and correlation structures in each study population are varied jointly. The points are coloured by contrast, with lighter shades for the population and darker for the population.

FIGURE B21
Empirical and model standard errors for scenarios g, h, and i, along with 95% Monte Carlo confidence intervals. One of the two effect modifiers was not adjusted for. The covariate distributions and correlation structures in each study population are varied jointly. The points are coloured by contrast, with lighter shades for the population and darker for the population.

FIGURE B22
Coverage zip plots for the ( ) contrast estimate for scenarios g, h, and i. One of the two effect modifiers was not adjusted for. The covariate distributions and correlation structures in each study population are varied jointly. The 95% confidence/credible intervals are coloured as coverers (green) or non-coverers (purple), and the colour change should occur at the 95th centile (i.e. nominal coverage). The horizontal dashed lines are 95% Monte Carlo confidence intervals for the coverage.