Prioritizing investments in new vaccines against epidemic infectious diseases: A multi‐criteria decision analysis

J Multi‐Crit Decis Anal. 2019;26:153–163. Abstract Background: In 2016, the Coalition for Epidemic Preparedness Innovations (CEPI) launched a call for proposals (CfP) for vaccine development against Lassa, MERS, and Nipah. CEPI is faced with complex decisions that involve confronting trade‐offs between multiple objectives, diverse stakeholder perspectives, and uncertainty in vaccine performance. Objective: This study reports on a multi‐criteria decision analysis (MCDA) and its testing on CEPI decisions. Methods: Consultations with CEPI's Scientific Advisory Committee (SAC) and document reviews helped identify and structure the criteria against which to evaluate proposals. Forty four subject‐matter experts assessed performance of 18 proposals on multiple criteria. SAC preferences were elicited via a survey employing an adapted swing‐weighting technique and were incorporated into measures of value and cost‐ to‐value. A Monte Carlo simulation estimated overall value and ranking probabilities by value and by cost‐to‐value for each proposal. Results: Reviewer assessments and SAC preferences varied significantly. Despite this uncertainty, 14 preferred proposals emerged from the analysis and SAC recommendations on the basis of value and cost‐to‐value. In some cases, SAC recommendations deviated from the analysis because of: less emphasis on cost‐to‐value if budgets seemed underestimated by applicants, more emphasis on the likelihood of generating vaccines for target pathogens versus platform potential against unknown pathogens, and emphasis on funding a diversity of platforms per pathogen. Conclusions: Despite vaccine performance uncertainty and stakeholder preference heterogeneity, MCDA distinguished between options in a way that broadly corresponded to decisions. Divergence between the MCDA and the SAC point to potential updates needed to the model such as platform diversity trade‐offs.


| INTRODUCTION
Following the successful vaccine research and development (R&D) response to the 2014 West-African Ebola epidemic (Grobusch & Goorhuis, 2017), the World Health Organization (WHO) prioritized 11 epidemic infectious diseases (EIDs) most likely to cause severe outbreaks in the near future (WHO, 2016). Vaccines can prevent EID outbreaks from becoming humanitarian crises (CEPI, 2016a;Kieny et al., 2016). However, market incentives have failed to sustain R&D efforts in this area (Plotkin, 2016). A new entity, the Coalition of Epidemic Preparedness Innovations (CEPI), was set up in 2016 with a US$1 billion investment target to support the development of vaccines, contributing to the world's preparedness for EID outbreaks (CEPI, 2016a).
One of CEPI's first business plan targets was to advance the development of two to three vaccine candidates against priority EIDs from preclinical through to end of early clinical safety and immunogenicity testing (Phase IIa) by 2022 (CEPI, 2016a). By doing so, CEPI aimed to address the "just-in-case" R&D preparedness gap associated with lack of Phase IIb/III-ready EID vaccines in advance of epidemic outbreaks.
Just like any funder of pharmaceutical R&D, CEPI is faced with the challenge of prioritizing limited resources in order to meet inherently risky R&D targets (Aurentz, Kirschbaum, & Thunecke, 2011). Evidence suggests that the average probability of successfully advancing a vaccine candidate from preclinical through to end of Phase II is less than 10% (Pronker, Weenen, Commandeur, Claassen, & Osterhaus, 2013). In addition, the large costs (DiMasi, Grabowski, & Hansen, 2016; and long timelines (Russell & Gronwall, 2012) involved in developing vaccines make investment decisions in this space tremendously complex. This complexity is compounded by the fact that where commercial objectives are lacking, such as in the field of EID vaccines, commercial value alone is an insufficient criterion to making pharmaceutical R&D investment decisions (Antonijevic, 2015;Cioffe, 2011;Perez-Escobedo, Azzaro-Pantel, & Pibouleau, 2012;Phillips & Bana e Costa, 2007).
To address its business plan targets in line with mission and scope, CEPI launched a competitive call for proposals (CfP) in late January 2017 to support the development of vaccine R&D in three priority EIDs: Lassa Virus, Middle East respiratory syndrome coronavirus (MERS-CoV), and Nipah virus. The rationale was to invest in vaccine R&D projects that would improve the likelihood of generating vaccines relevant for use in response to these EIDs; as well as improve the likelihood that the platform technologies supporting these vaccines would be suitable for use in vaccine development against newly or unexpectedly emerging EIDs.
Evaluating proposals received in response to the call faces several challenges. First, pharmaceutical R&D portfolio management involves considering multiple criteria, including organizational capabilities, technical and manufacturing feasibility, development timelines and costs, and alignment with target product profiles (TPPs; Aurentz et al., 2011;Bode-Greuel & Nickisch, 2008;Seget, 2005). For instance, the WHO has been advocating for use of ideal TPPs, or preferred product characteristics (PPCs) tailored to EID outbreak preparedness needs, to determine use potential of vaccines and to guide R&D priorities in the field (WHO, 2018).
Second, stakeholder opinions varied on the relative importance of different objectives. CEPI's Board has the ultimate decision-making authority on all CEPI R&D investments. An independent, multimember Scientific Advisory Committee (SAC) advises the CEPI Board and Secretariat on R&D investments and makes technical recommendations for project funding. The composition of the SAC is diverse, and at the time of the deliberations described here included nine representatives of governments and regulators, seven industry members, eight academics, and four representatives of non-profit R&D organizations (CEPI, 2016b).
In this context of multiple trade-offs and heterogeneous stakeholder perspectives, multi-criteria decision analysis (MCDA) has the potential to improve the quality of decision making Viergever, Gouglas, & Tromp, 2017). MCDA has become increasingly popular in health valuation Thokala et al., 2016) and its applications are numerous across a variety of areas in health (Adunlin, Diaby, & Xiao, 2015) and associated decision problems (Drake, de Hart, Monleon, Toro, & Valentim, 2017;Marsh et al., 2017;Thokala et al., 2016). MCDA can offer a rational and transparent approach to priority setting, simultaneously considering all relevant criteria to avoid ad hoc decisions (Baltussen & Niessen, 2006). Where there are multiple stakeholders with diverse perspectives, MCDA can make relevant conflicts explicit, helping decision makers understand them and consider their impact on decisions (Phillips & Bana e Costa, 2007;Timmis, Black, & Rappuoli, 2017).
The use of MCDA has been increasingly advocated in vaccine R&D (Barrochi, Black, & Rappuoli, 2016;Timmis et al., 2017). However, to our knowledge, only one other MCDA framework has been applied to support the prioritization of vaccine R&D (see for instance Phelps et al., 2014;Kloeber, 2011;Madhavan et al., 2012;Madhavan et al., 2013;Madhavan et al., 2015). This framework places emphasis on different attributes of burden of disease, which are difficult to apply in the CEPI context, given the sporadic and unpredictable nature of EIDs.
Moreover, it does not lend itself easily to the estimation of value of vaccine R&D, which is adjusted for the probability of success (PoS) of early stage, risky vaccine candidates; nor does it assume sources of preference and constraints that are relevant to the CEPI decision context. An MCDA framework was developed in accordance with ISPOR Good Practice guidelines  to inform the prioritization of EID vaccine R&D proposals and support CEPI CfP decisions and was tested against the SAC recommendations. This study reports on the application of the MCDA framework. Results are anonymized because of confidentiality restrictions associated with ongoing contract negotiations between CEPI and developers of selected proposals for funding.

| METHODS
The analysis focused on 18 full proposals that were selected by the CEPI SAC for an extended review following on an initial review of 33 preliminary proposals (CEPI, 2017a). The 18 proposals had a combined budget of over US$700 million and were reviewed by CEPI between March and May 2018. Seventeen proposals were at the preclinical development phase and one proposal was at clinical phase 1, with the aim that CEPI funding would advance them to the end of clinical phase 2. Proposals covered three different types of diseases: Lassa, MERS, and Nipah. Proposal budgets ranged from US$22 million to US$68 million, with a median cost of US$35 million. Proposal timeframes through to end of clinical phase 2 ranged from 4 to 6 years, with a median timeframe of 5 years. Due to confidentiality restrictions, individual proposal budgets and timeframes are not reported here; however, it is these budgets and timeframes that have been used to generate values in the framework presented below. Proposal names and disease classifications have been anonymized throughout the remainder of this manuscript. Proposals have been labelled as P1 to P18 and platform types are labelled 1-3. Seven proposals covered disease 1; seven proposals covered disease 2; and four proposals covered disease 3.
The goal was to undertake a quantitative valuation and ranking of the 18 proposals against criteria that were of interest to the SAC. It was assumed that not more than 14 proposals could be funded, given the resources available. The remainder of this section provides a step-by-step overview of the modelling approach adopted (more details provided in Data S1).
Step 1 To narrow down the list of criteria, and combine the criteria into a value framework, members of the SAC and CEPI secretariat staff were asked first in an email survey and then in a group discussion in January 2017 to determine: whether all factors relevant to CfP decisions had been captured by the criteria; the relationship between the criteria, and whether any of the criteria should be removed or re-grouped if overlapping, or irrelevant. Following this engagement, overall value (Vi) was estimated as described in Equation (1).
Where:  Table 1. Each of these criteria is defined as a probability on a measurement scale 0-100%. where: C1 i = experience and track-record: Likelihood that the applicant is sufficiently competent to deliver on the proposed activities of the project, for a given proposal i.
C2 i = feasibility: Likelihood that the development of the candidate vaccine through phase II is technically feasible, for a given proposal i.
C3 i = manufacturing scalability and speed: Likelihood that the vaccine candidate is manufacturable and scalable in timeframes and volumes to respond to outbreaks, for a given proposal i.
C4 i = use potential for CfP target pathogens: Should a vaccine candidate be successfully developed and manufactured, the likelihood that it will meet CEPI's Target Product Profile and will be relevant for use in an emergency, for a given proposal i.
C5 i = use potential for new pathogens: Should a vaccine candidate be successfully developed and manufactured, the likelihood that the platform technology supporting the candidate vaccine will be suitable for use in vaccine development against newly emerging pathogens for a given proposal i.
The value framework presented in Equations 1-3 was presented to and approved by the SAC in February 2017, together with proposed criteria descriptions, measurement scales, and appraisal questions for reviewers (see Table 1).  Overall likelihood that the project will generate a vaccine that is relevant for use in response to one of the CfP target pathogens (0-100%) • Probability of successful vaccine development from preclinical through phase II (criteria C1 to C3) times the probability of use for CfP target pathogens (C4) O2. Likelihood that the platform technology will be suitable for vaccine development against new pathogens Overall likelihood that the platform technology supporting the vaccine will be suitable for use in response to newly emerging and/or unexpected pathogens (0-100%) • Probability of successful vaccine development from preclinical through phase II (criteria C1 to C3) times the probability of use for new pathogens (C5) does not have the potential to support vaccine development against a pathogen (C5)-so these criteria were combined multiplicatively.
Step 2 For each criterion C1-C5, reviewers were asked to define the most likely worst-case and best-case performance of proposals on a scale of 0-100% (see Data S1 for details). In order to determine the degree of homogeneity in the assessments provided by the different reviewers, an inter-reviewer assessment variability test was conducted. Specifically, for each criterion C1-C5, and for each performance estimate (worst-case, most likely, best-case), the following steps were undertaken. First, the performance mean across all reviewers assessing a given proposal was calculated. Second, the difference between this mean and each reviewer's performance estimate on the given proposal was calculated. Third, steps 1 and 2 were repeated for all proposals.
Fourth, for each reviewer, the average deviation of his or her performance estimate from the performance mean across all of his or her assessed proposals was estimated. Fifth, on the basis of Cicchetti's (1994) classification, reviewer variability was determined as good if this average deviated less than 20% from the performance mean, and excellent if it deviated less than 10% from the performance mean across all of his or her assessed proposals. Seven of the 44 reviewers were found to have at least one average worst-case, most likely, or best-case performance estimate against C1-C5 that deviated more than 20% from the equivalent performance mean across all their assessed proposals. In total, the estimates for which such deviations were observed accounted for only 3% of the total number of worstcase, most likely, and best-case performance estimates collected.
The impact of removing these results from the analysis was tested and found to not substantially change the performance of proposals.
Step 3. Estimating partial values (PV O1 , PV O2 ) Partial value functions were elicited for O1 and O2 from each SAC member using an online survey (24 respondents out of 29 survey recipients). The functions were defined using a mid-value splitting method-a widely-used decomposed scaling technique also known as the bi-section method (Von Winterfeldt & Edwards, 1986)-by eliciting the value mid-point on a 10%-60% performance range (point a in Figure 1).
In the questioning procedure applied for the elicitation of partial values on each of O1 and O2, SAC members answered up to six pairwise choice questions that iteratively approached this value mid-point. For instance, for O1 the first question was: "Consider the following two proposals, each with different starting likelihoods of generating a vaccine that will be relevant for use in response to one of the CfP target pathogens. Imagine you are given the opportunity to improve the performance of one of these proposals. Which of the following options would you prefer?
• Option A: Improve Proposal A so that the likelihood that it generates a vaccine that will be relevant for use in response to one of the CfP target pathogens increases from 10% to x% • Option B: Improve Proposal B so that the likelihood that it generates a vaccine that will be relevant for use in response to one of the CfP target pathogens increases from x% to 60%

• Indifferent between options A and B″
In the first question, x was set as the mid-point in the performance range (35%). If a respondent was indifferent, the partial value function was considered linear, and no further questions were asked. If a respondent chose option A or option B, the value of x was updated according the logic defined in Section 2 of the Data S1.
The pairwise choice questions identified a to be within a range.
It was assumed that a was the mid-point in this range, on the basis of which partial value functions could then be defined (see Section 2 of Data S1).
Step 4. Estimating weights (W O1 , W O2 ) Weights were elicited for O1 and O2 using the trade-off method (von Winterfeldt and Edwards, 1986), in each case for a range of performance of 10%-60%, from each SAC member using an online survey (24 respondents out of 29 survey recipients). An iterative pairwise comparison was used to identify the value of b, such that respondents would be indifferent between improving O1 from 10% to y% and improving O2 from 10% to 60%. Specifically, the following question was asked via an online survey: FIGURE 1 Partial value function of likelihood of generating a vaccine for one of the CfP target pathogens: an illustration "Considering the following two proposals, which of these would you prefer?
• Proposal A ○ Likelihood of generating a vaccine that will be relevant for use in response to one of the CfP1 target pathogens = y% ○ Likelihood that the technology will be suitable for use in vaccine development against newly emerging/unexpected pathogens = 10% • Proposal B ○ Likelihood of generating a vaccine that will be relevant for use in response to one of the CfP1 target pathogens = 10% ○ Likelihood that the technology will be suitable for use in vaccine development against newly emerging/unexpected pathogens = 60%" The initial value of y was set at 35%, and varied depending on responses as defined in Section 2 of the Data S1. After six questions, the value of b was defined within a range defined in Section 2 of Data S1. It was assumed that the value of b was the mid-point in this range.
Section 2 of the Data S1 provides more details on how b was used to estimate weights for O1 and O2.
Step 5. Eliciting time preference (r) Time preference was estimated using a choice exercise designed to identify the value of c, such that SAC members were indifferent between a z% chance of successfully delivering a proposal within 5 years, and a 100% chance of doing so within 10 years. The following question format was implemented with SAC members in an online survey: "Considering the following two proposals, which of these would you prefer?
• The value of z in the first question was set at 55%, and then varied depending on responses in a manner described in Section 2 of Data S1. After up to six questions, the value of c was identified within a range described in Section 2 of Data S1. It was assumed that c took the value of the mid-point in this range. Section 2 of the Data S1 provides more details on how c was used to estimated to estimate the discount rate.
Step 6 Dealing with uncertainty Both reviewer performance inputs and SAC preferences were subject to significant variations. This uncertainty was incorporated into the MCDA via Monte Carlo simulation. The model was run 10,000 times, each time drawing from the different inputs, as follows: • Performance inputs: For criteria C1 to C5, each iteration randomly selected one reviewer and randomly selected a performance estimate from their performance distribution.
• SAC preferences: Each iteration randomly drew the partial value, weights, and time preference of a single SAC member's distributions.
The mean and 95% confidence intervals of performance on C1-C5, O1, O2, and V were estimated for each iteration of the simulation.
Comparison of proposals within each iteration allowed a ranking of proposals, which, when analysed across all iterations, allowed the estimation of the rank probability of a proposal.

| RESULTS
Criteria performances of the 18 vaccine R&D proposals are presented in Table 2. The uncertainty in performance means that there is substantial overlap in the confidence intervals around most proposals' performance on: the likelihood of generating a suitable vaccine for one of the CfP target pathogens (O1) and on the likelihood that the platform technology will be suitable for vaccine development against new pathogens (O2).   Assuming that not more than 14 projects can be selected, Figure 3 plots the likelihoods of proposals ranking in the top 1-14, on the basis of discounted value versus cost-to-value. The consideration of budgets did not affect most of these ranking outputs in the analysis,    for funding (proposals 1, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 17, and 18   Second, the SAC placed less emphasis on cost-to-value, as in some cases proposed budgets were considered unrealistic. One implication of this was that some proposals that had small budgets but whose overall value was otherwise low were not prioritized. Third, the SAC's deliberation pointed towards structural implications for the MCDA. A novel combined multiplicative-additive model structure was adopted. Few MCDA applications in healthcare have multiplicative components  despite concerns that health technology assessment does not meet the analytical requirements of additive models (Marsh, Sculpher, Caro, & Tervonen, 2018).
The multiplicative component of the model implicitly gave equal weight to criteria C1-C5, whereas the SAC deliberation seemed to emphasize some of these criteria more than others (e.g., C2 and C4).
Fourth, the SAC's recommendations could imply alternative weights to those used in the model; specifically that an even greater weight was given to O1 than what was elicited through the survey.
Weights in the model were elicited using an iterative comparison of improvements in pairs of criteria. This method was chosen due to the small sample size providing insufficient power for a discrete choice experiment; and a desire to elicit ordinal data in a survey format (Tervonen et al., 2017). The result was that, on average, more weight was given to O1. Though there was also significant variation in SAC member weights. Given this variation, one possibility that would reconcile the SAC recommendations with the result of the MCDA is that SAC members who gave a higher weight to O1 were more influential in the deliberation.
In conclusion, the analysis reported in this study demonstrates that it is possible to use a MCDA to support the prioritization of vaccine R&D investments in a complex decision context characterized by outcomes uncertainty, variance in expected performance of vaccines, and heterogeneity of stakeholder preferences. With the intention to aid, rather than replace deliberative stakeholder processes or prescribe decisions, the findings illustrate how MCDA can help differentiate investments, and support decision making.