Risk modelling in blood safety – review of methods, strengths and limitations
Brian Custer, MPH, PhD, Blood Systems Research Institute / University of California, San Francisco, 270 Masonic Ave, San Francisco, CA, USA
Risk modelling studies in blood safety play an important but occasionally misunderstood role. These studies are intended to quantify and contrast risks and benefits. This information is critical for policy development and intervention decision-making. The limitations of risk modelling should be considered alongside the results obtained.
The goal of this manuscript and presentation is to review current risk modelling techniques used in blood safety and to discuss the pros and cons of using this information in the decision-making process. The types of questions that can be answered include the extent of a risk or threat; implications of action or inaction; identification of effective strategies for risk management; or whether to adopt specific interventions. These analyses can be focused on a risk alone but are often combined with economic information to gain an understanding of feasible risk interventions given budgetary or other monetary considerations. Thus, analyses that include risk modelling provide insights along multiple lines. As important, the analyses also provide information on what is not known or uncertain about a potential hazard and how much that uncertainty may influence the decision-making process.
Specific examples of the range of risk analyses in which the author has participated will be reviewed and will include ongoing process improvement in testing laboratories such as error identification/eradication, estimation of the risk of malaria exposure based on the specific locations of travel, evaluation of blood supply and demand during an influenza pandemic, cost-utility analyses of screening interventions for infectious diseases in countries with different human development indices, and insurance against emerging pathogen risk. Each of these analyses has a different purpose and seeks to answer different questions, but all rely on similar methods.
The tool kit for risk analysis is broad and varied but does have limitations. The chief limitation of risk modelling is that risk analyses are not scientific experiments or otherwise controlled studies. Consequently, the analyses are more apt to be influenced by assumptions. These assumptions may be necessary to structure a problem in a way that will allow the question of interest to be answered or may result from incomplete or missing information. Another potential limitation is that commissioners of such studies, those who undertake them, and the intended audience, such as regulatory agencies, may have distinct and differing interpretations of the results.
Risk modelling is a set of techniques that can be used to inform and support decision-making at all levels in transfusion medicine. Advances in risk modelling techniques allow for continued expansion in the scope of possible questions that can be analysed. Expanded use also improves the acceptance of the utility of these studies in blood safety and transfusion medicine.
Any effort to define the existence, scope, or consequences of an extrinsic or intrinsic threat (known as a hazard) to the safety of the blood supply is a form of risk assessment. Risk assessment is accomplished through an iterative process of defining and distilling a problem to the core pieces of information that characterize the hazard or the possible consequences of actions that can mitigate it, coupled with a quantitative evaluation. Efforts to advance blood safety while seeking to improve patient outcomes following transfusion rely on effective and timely risk management. Yet, risk management rarely occurs with a full understanding of all aspects of a hazard. Risk assessment and management are therefore accomplished through risk modelling.
Models are simplifications of larger or more complex processes or problems. The purpose of modelling in health care is to develop a simulation of a real medical or public health issue that is condensed to its most important aspects. The process of modelling can also be used to help define which aspects are most important; a tension exists between providing the most comprehensive model and a model that addresses the key factors relevant to a hazard.
The types of questions that can be answered include the extent of a hazard; implications of action or inaction; identification of effective strategies for risk management and related efforts to support decision-making between interventions. These analyses can be focused on a hazard alone but are often combined with economic information to gain an understanding of effective and affordable interventions given budgetary or other monetary considerations. Linking risk assessment and economic data into a single analysis provides an evaluation of risks and benefits in terms of morbidity/mortality and also costs. The goal of this manuscript is to review the methods used in risk modelling in blood safety/transfusion medicine. Focus will be given to recent risk modelling efforts across a range of relevant issues in blood safety. Following the examples, the primary strengths and limitations of risk modelling will be discussed.
Overview of scope of risk modelling and methods
Risk modelling is conducted by many people in many different disciplines and may not be explicitly recognized as risk modelling. When choices have to be made, even something as specific and technical in nature as an evaluation of which antibody assay performs best in a given setting or country is a form of risk modelling. An example is a recent effort to evaluate the performance of serological assays for Trypansoma cruzi . Under the assumption that good quality clinical and laboratory data are available, the results of analyses such as the T. cruzi test assessment tend to lead to conclusions with relatively little uncertainty. Other risk modelling efforts can include a truly expansive set of hazards, such as the recent effort to prioritize emerging pathogen risks with potential to impact the blood supply . Most risk modelling falls between these extremes and involves a combination of a perceived hazard for which there is sufficient information to define the important aspects coupled with a goal of informing policy. Examples are truly wide ranging and include assessing the risk of infection in donated blood based on changes in donor deferral such as malaria-related travel or modifying the deferral for men who have sex with men, to appropriate testing strategies for West Nile virus (WNV) given other constraints, to cost-benefit analyses of specific interventions such as bacterial culture for platelets compared to use of pathogen reduction technology [3–5].
Methods for risk modelling and related studies in blood safety have recently been reviewed in other publications [6,7]. Risk assessment is typically focused on a single threat. A four-step process for risk assessment has been developed; (i) identification – establishing the existence of the hazard, (ii) dose–response assessment and related evaluation – whether the hazard leads to morbidity or mortality, (iii) exposure assessment – whether the hazard is present or could be present in the population of interest, and (iv) risk modelling – the process of synthesizing multiple lines of information into an overall assessment .
Risk modelling can stand alone and serve to define the scale or scope of a specific hazard. However, risk managers typically need more information on the consequences of managing a risk through different strategies or interventions. Contrasting the consequences and benefits of different strategies leads to a broader set of methods where interventions are compared within the same model framework using various forms of decision analysis. Decision analysis is a comparative evaluation seeking to help provide insights into the most effective approaches for risk management. A decision analysis that includes both risk assessment and economics jointly is a cost-effectiveness or cost-utility analysis depending on how outcomes are assessed . Risk management is a process that can require multiple different decisions, with subsequent decisions dependent on the preceding ones. The complexity of the available underlying methods continues to expand, allowing for the inclusion of multiple decisions in one evaluation . However, the use of newer methods such as Markov decision processes is uncommon in blood safety.
Rarely is available data complete, and risk modellers use statistical distributions or ranges of values for factors that are not known with certainty; this uncertainty is carried through the analysis so that results include a central point estimate such as a mean with an associated confidence interval around the point estimate .
A component of risk modelling is sensitivity analysis, which seeks to determine how important uncertainty is for each particular data element or model parameter and for the overall analysis. Not all model parameters can be defined by underlying probability distributions, and some parameters are likely to be deterministic with sensitivity analysis conducted over a range of plausible values. Regardless of the risk model, the assessment of the influence of uncertainty and related techniques are crucial and informative. Probabilistic sensitivity analysis provides the greatest insights and also most comprehensive consideration of uncertainty [10,11]. Sensitivity analysis also serves another function; it identifies areas where additional knowledge about an aspect of the hazard may be critical for better risk management. The process of modelling and sensitivity analysis identifies gaps in current knowledge and suggests directions for future research.
Examples of the use of risk modelling
One of the blood safety hazards that has received the most extensive risk modelling in the last 20 years is vCJD. This disease poses such novel and difficult questions that a tremendous amount of risk modelling has been conducted on virtually every aspect of the overlap between the biology of prions and blood transfusion. Jurisdictions and blood operators worldwide have sought to evaluate the threat that vCJD may represent for blood recipients [12–16], how that threat might be reduced through donor deferral [17–21], and the value of technological interventions in mitigating risk [22–26]. These efforts continue today.
Another example of risk modelling in blood safety is an analysis of WNV minipool nucleic acid testing (NAT) data in the United States  During most of the year, a minipool NAT format is used for testing WNV, but concern about the potential for infections missed by minipool testing prompted consideration of the use of individual donation NAT. Although no longer the case today, in 2004 the capacity for individual donation testing was limited. Individual donation trigger strategies were developed based on counts of repeat reactive donations and/or weekly repeat reactive rates, together with strategies for the discontinuation of individual donation testing. The approach of targeted individual donation testing based on ongoing monitoring of minipool yield is now standard practice for this infection in the United States and Canada during the high season for WNV. In subsequent years, this approach has been modified and greatly expanded to include consideration of probabilistic risks and to gain on additional understanding of the risks posed by infections that could be missed by minipool NAT . The analyses for WNV risk show a progression in the sophistication of the modelling techniques used. The more recent modelling efforts identified the importance of the size of the geographic area used for monitoring the decision to trigger individual donation testing and also showed the difficulty in determining which criteria to use to stop individual donation testing. Such modelling efforts are vital to understanding the implications of different policy options.
A further example of risk modelling that is relevant in blood safety is defining the impact of modification of deferral policies on the risk of potentially infectious blood donations. For example, in settings where selective serological testing is not used for malaria, the risk management strategy is non-specific and so to prevent the possible donation from a small number of donors that could have acquired malaria but may not be symptomatic at the time of donation, thousands of donors are deferred representing a large loss of donated blood and donors who never return [28,29]. A recent study compared the risk that a donor presents to donate while infected with malaria under current (up to 1-year) and alternative deferral periods . The model assessed the risk of malaria infection following travel to different locations and included the distribution of incubation periods for each malaria species. In this analysis, travel to Africa was shown to represent 1000 times greater risk of malaria than travel to parts of Mexico that are considered endemic for malaria, yet in the United States, travel to Mexico leads to 10 times as many deferred donors. The risk modelling techniques used in this analysis relied on available data for all US travellers to foreign destinations and then projected the observed occurrence of malaria on to the donor population. Subsequent analyses of this topic sought to examine locations of travel within Mexico for US travellers who are blood donors and showed that risk is extraordinarily low for the most popular travel destinations reported by US donors.
The ISBT Working Party on Transfusion-Transmitted Infectious Diseases (WP-TTID) recently supported a new risk modelling effort that seeks to provide a portal for risk assessment and cost-utility analyses of screening interventions for HIV, HCV and HBV in countries with different human development indices. The goal of this project is to expand the availability and use of these methods through a web interface that guides users through the entry of the core data necessary for these analyses and then provides results that can be specific to a country or can be compared to similar results for other countries.
Comparing analyses across countries is inherently difficult because of differences in the epidemiology of disease, health care infrastructure and available resources. Limitations of previous analyses in blood safety include incomparability of models and analysis assumptions [31,32]. The WHO has developed several publications that seek to describe methods under the CHOosing Interventions that are Cost-Effective framework [33–35]. The goal of these methods is to structure and conduct analyses in such a way that the interventions considered are relevant in a specific setting and across settings.
In the ISBT-WP-TTID-funded project, we have sought to use the same approach and have also identified the most important data that must be obtained for an analysis to have specific relevance for a country. The current work builds off of the previous work of van Hulst and colleagues . The countries included in this study were the Netherlands, Thailand and Ghana, thus showing the applicability of common underlying methods to different countries. Importantly, the effort to improve access to the analysis was facilitated by a web-based user interface developed by Hubben and colleagues that allowed for entry of country-specific data in a simple and consistent manner .
In the ISBT-WP-TTID project, we have now constructed a format for structured risk assessment and cost-utility analysis of HIV, HCV and HBV screening strategies that will be useful for two groups; persons that may not have the expertise in risk modelling in specific countries or settings but want to conduct such analyses, and persons who are interested in comparative analyses across settings. This updated web interface (http://www.bloodsafety.isbt-web.org) is currently being used to assess the cost-utility of different HIV, HCV and HBV testing strategies in six countries (Brazil, Ghana, Netherlands, South Africa, Thailand and USA). The risk modelling uses common analytical assumptions and underlying disease progression models for each infection. However, country-specific data on the incidence and prevalence of infection, percentage of first time and repeat donors, cost of different testing methods, average age of transfusion recipients, transfusion survival and related parameters are necessary. In an innovative response to the recognition that data for every variable in the risk model may not be available, when country-specific data are not available, data are used from a country with a similar human development index. This project also provides two types of results; results are expressed in (i) terms of the number infected blood donations interdicted using different screening strategies and (ii) as cost per disability-adjusted life year gained ($Int/DALY). The choice of this metric over quality-adjusted life years was made to provide a standard outcome across settings. This risk assessment and cost-utility analysis project has provided an important advance in the availability of risk modelling to persons who have internet access anywhere in the world but may not have the training to conduct a risk analysis.
Risk modelling in placed in perspective
Each of the risk modelling examples reviewed here has a different purpose and seeks to answer different questions, but all rely on similar methods. As a group, these analysis techniques have both advantages and disadvantages.
Risk modelling techniques allow for the analysis of an incredibly broad range of topics and questions. Notably, these studies are not dependent on having complete data on all aspects of a hazard or related information for the results obtained to have meaning. The results each analysis must always be placed in the context of what is known about the hazard at the time the analysis was conducted. An analysis from 10 years ago may not be applicable to the same hazard today. Risk modelling is an iterative process and can be conducted early during the emergence of a new hazard and reconducted throughout the time the hazard exists. The history of vCJD analyses demonstrates this progression of understanding. Invariably successive analyses are improved because of the availability of new data and often reduced uncertainty. At the same time that the quality of the data is improving, an understanding of the most relevant aspects of the hazard that need to be included in the risk model undergoes refinement. Aspects of a hazard that seemed important at one time may become less relevant. A further aspect of risk modelling that may be under-appreciated by many in the blood safety and transfusion medicine discipline is that risk modelling can also be used to help suggest research directions. The use of risk modelling allows risk managers to understand how important a given model variable and its specific numerical value is and this can be used to conduct an analysis to assess the importance of pursuing more precise data through a technique called value of information analysis .
The tool kit for risk analysis is broad and varied but does have limitations. The methodologies can be complex and difficult for persons unfamiliar with them to follow. The complexity of the methods may create unease on the part of risk managers; if something seems too complex to understand it is difficult to use such information for decision-making. The risk modeller’s goal should be an analysis that is no more complex than it needs to be to answer the specific question at hand. However, it is just as important to incorporate new analysis methods or approaches as these usually lead to a deeper understanding of the hazard or the most effective interventions against a hazard. Finding the balance between using established methods and including new methods in a risk modelling project will be context and hazard specific.
The chief limitation of risk modelling is that risk analyses are not scientific experiments or otherwise controlled studies. Consequently, the analyses are more apt to be influenced by assumptions. These assumptions may be necessary to structure a problem in a way that will allow the question of interest to be answered or may result from incomplete or missing information. Risk modelling requires an assessment of the quality of available data or previous research as part of the process of selecting which data to use in a new analysis. Risk modelling requires that the persons involved serve as ‘honest brokers’ of the information. Analyses can be unintentionally biased by the way a question is posed and then analysed or by the choices of which data are included. In a sense, a risk analysis is an argument regarding aspects of a hazard, and the more thorough and rigorous the evaluation is, the broader the acceptance of the findings is likely to be.
Another potential limitation is that commissioners of studies that include risk modelling, those who undertake them, and the intended audience, such as regulatory agencies, may have distinct and differing interpretations of the results. Risk models provide insights into hazards but do not provide interpretations of results or make decisions. These techniques are intended to support decision-making. Interpretation of findings is context specific.
The purpose of risk modelling in health care is to develop a version of a real medical or public health issue that is condensed to the most important aspects of the hazard. Risk modelling is a set of techniques that can be used to inform and support decision-making at all levels in transfusion medicine. The primary limitation of risk modelling largely lies in the fact that such efforts are not experiments or controlled studies. The primary strength of risk modelling is the broad scope of informative analyses that can be conducted. Advances in risk modelling techniques allow for continued expansion in the scope of possible questions that can be analysed. Expanded use also improves the acceptance of the utility of these studies in blood safety and transfusion medicine.