Risk-Based Decision Making for Terrorism Applications

Authors


*Address correspondence to Robin L. Dillon, 418 Old North, Washington, DC 20057, USA; tel: 202-687-5398; fax: 202-687-4031; rld9@georgetown.edu.

Abstract

This article describes the anti-terrorism risk-based decision aid (ARDA), a risk-based decision-making approach for prioritizing anti-terrorism measures. The ARDA model was developed as part of a larger effort to assess investments for protecting U.S. Navy assets at risk and determine whether the most effective anti-terrorism alternatives are being used to reduce the risk to the facilities and war-fighting assets. With ARDA and some support from subject matter experts, we examine thousands of scenarios composed of 15 attack modes against 160 facility types on two installations and hundreds of portfolios of 22 mitigation alternatives. ARDA uses multiattribute utility theory to solve some of the commonly identified challenges in security risk analysis. This article describes the process and documents lessons learned from applying the ARDA model for this application.

1. INTRODUCTION

Anti-terrorism efforts, that is, defensive measures used to reduce the vulnerability of individuals and property to terrorist acts,(1) are receiving significant government funding since the September 11 attacks on the United States. Unfortunately, with the nearly infinite number of attack scenarios and the persistent nature of the threat, choosing the “best” anti-terrorism efforts is a difficult challenge.

The solution advocated at the highest level of government is to base one's decisions on an assessment of the risks. For example, in 2005 Senate confirmation hearings, the future Secretary Michael Chertoff states, “DHS [Department of Homeland Security] must base its work on priorities driven by risk.”(2) Furthermore, the Government Accounting Office (GAO)(3) states: “A comprehensive risk management process can be an effective foundation for allocating antiterrorism resources.” In the same report, the GAO provides a definition of risk management as “a systematic, analytical process to determine the likelihood that a threat will harm individuals or physical assets and to identify actions to reduce risk and mitigate the consequences of a terrorist attack.”

The problem, often overlooked, in recommending a risk-based approach is that decisions require preferences and, more specifically, decisions regarding risk require thresholds that define acceptable levels of risk. The tools of risk assessment can objectively support the identification of attack scenarios, linkages among initiating events and different damage states, probabilities of occurrence, and possible mitigation measures and benefits.(4,5) Although many policymakers would like to advocate that decisions can be made objectively based on risk, as we will show and discuss later, these decisions require preferences and subjective input, and this goes beyond the objective tools of risk assessment.

In Section 2, we detail many of the challenges that exist for security risk analysis and terrorism applications. Significant research has been ongoing to address these issues,(6) but more work needs to be done. In Section 3, we describe the anti-terrorism risk-based decision aid (ARDA) for risk-based decision making. Our model has the advantage of applying consistent rules, treating model parameters systematically, and documenting subjective preferences and assumptions. In Section 4, we discuss the general results of the application of ARDA. In Section 5, we detail the many lessons learned from this application. We conclude in Section 6.

2. SUMMARY OF CHALLENGES FOR SECURITY RISK ANALYSIS

The formal recognition of risks is not new. The desire to manage risks in a concerted way is at least as old as the insurance pools formed by groups of Phoenician ship owners in ancient times(7) or by the Venetians in the 15th century. Also, the research applying risk analysis to terrorism events since September 11 is vast; however, several important challenges remain. We describe five major challenges in no particular order.

First, terrorism risk is dynamic. Defenders are constantly making investments to reduce vulnerabilities, and attackers can constantly alter preferences for targets.(8,9) Therefore, some standard for estimating and monitoring change is needed. This same standard for monitoring change could also be used for estimating the rate of return on previous anti-terrorism investments. The Congressional Research Service(2) has challenged Congress to draft a national impact assessment to quantify the return on investment of the approximately $12 billion (through FY2008) allocated through the Department of Homeland Security's Grant Program. Without clear risk measures or scores, one cannot benchmark improvements or progress to reduce the overall risk level.

Second, there are not enough resources to eliminate all risks or even comply with all the standards and requirements currently in place. The U.S. Navy currently funds 60–70% of the security manpower Navy policies require to protect U.S. installations. (Note, those policies are under review at this time, as recent analyses suggest that policies overestimate the manpower requirement.) Assuming that this staffing shortfall continues in the future, the question is how much risk are they accepting by staffing at the current levels? Additionally, several studies look at resource allocation.(10–12) A counterintuitive result of prior research is that resource allocation to the greatest risk may not be the most cost-effective option. If there are smaller risks that can be entirely eliminated for a small amount of money, these investments, even if directed at small risks, could be the most cost-effective option from a benefit-cost ratio perspective.

Third, the large number of potential attack scenarios and security options makes it difficult to assess and include portfolio effects of mitigation measures. As will be discussed, several simplifying steps are used in this study to examine thousands of scenarios composed of 15 attack modes against 160 facility types on two installations and hundreds of portfolios of 22 mitigation alternatives.

Fourth, the historical data on terrorism (especially in the United States) are limited. Additionally, there are enormous uncertainties in estimates of threats, vulnerabilities, and consequences. Consequences are generally the best understood because scientific models exist that can quantify impacts from specific attack scenarios. For example, the Department of Defense (DoD) has models that estimate the destruction to a building based on the quantity of explosives, distance, building material, etc.(13) The most uncertain of the three components is the threat piece.(14)

Finally, risk is an inherently subjective assessment. Although most researchers agree that the risk of terrorism is some function of threat, vulnerability, and consequences, many competing theories exist on how to consider these components. One way is to quantitatively assess threat, vulnerability, and consequences, multiply the three factors to obtain an expected loss risk measure, and make comparisons based on the expected loss. Although there is no agreement on a better way, there is general agreement that an expected value approach is not appropriate for low-probability, high-consequence events.(15,16) There is also general consensus that the consequences from a terrorist attack are multidimensional. For example, Ayyub et al.(17) discuss casualties, economic impacts, mission disruption, and recuperation time but do not provide a mechanism for aggregating these measures. Aggregating different types of consequences requires value preference tradeoffs, which are, by definition, subjective. At the most fundamental level, there is no clear means to get answers to the truly subjective questions: “How much risk mitigation is enough?” or “What is an acceptable level of risk?”

Although we do not solve all of these challenges entirely, incorporating multiattribute utility theory and risk analysis into the ARDA model does, at least, recognize and contribute to the resolution of each of these major challenges. However, as will be described later, the implementation of the ARDA model was not without its own challenges.

3. ANTI-TERRORISM RISK-BASED DECISION AID

At a conceptual level, risk-based decision making is a straightforward process: define the system to be analyzed, assess the baseline risks to that system as defined, assess the improvements to the system from available mitigation alternatives, assess the costs of those mitigation systems, and then prioritize the mitigation alternatives to maximize risk reduction based on cost-benefit tradeoffs. Although seemingly very intuitive, there are many challenges in the details. Our overall approach is shown in Fig. 1, and the remainder of this section describes the specifics of implementing the ARDA model. The examples and illustrations discussed here focus on protecting U.S. Navy assets (e.g., bases, ships, aircraft, etc.), but the model can be applied to any decision-making situation that involves multiple potential targets and various mitigation alternatives.4

Figure 1.

Risk-based decision making.

We worked primarily with two decisionmakers on this study. The first, the project sponsor, was responsible for advising the highest levels of Navy leadership on the value of Navy investments. For this effort, he guided the framing of the analysis objectives and helped focus the effort on the most pertinent questions Navy leadership has regarding force protection decisions. The second decisionmaker had oversight for programs that protect Navy installations from terrorist risk and provide law enforcement capabilities. Because of his close interaction with Navy senior leadership in managing and protecting Navy installations and his past experience in advising senior leaders on installation protection, this decisionmaker was chosen to provide our assessments, as will be detailed in Section 3.

In our application, we do not explicitly consider mitigation costs. Intuitively, our decisionmakers know that budgets are limited; however, identified priorities could influence the budget process, so at this stage, we focus on examining the effectiveness of portfolios of risk mitigation alternatives rather than on prioritizing options based on cost-benefit considerations. It is assumed that if, for example, waterside barriers are identified as a necessary security measure to effectively reduce risk, a future study would determine how to install waterside barriers in a cost-effective manner. We will discuss some cost-risk tradeoffs that we did examine in Section 5.

3.1. Define the System

The first step is to define the system consistent with the scope and objectives of the analysis. This requires gathering four pieces of information: (1) the targets, such as specific facilities (or facility types) and war-fighting assets at the facilities, (2) the attack modes, (3) the attack scenarios (attacks paired against facilities), and (4) the mitigation alternatives for the appropriate attack scenarios. The more facilities and/or facility types identified, the greater the data assessment task and the more detailed the information available to the decisionmaker to prioritize investments at an installation level. Important things to consider in determining the right list of targets are the different types of facilities that exist at different installations (i.e., administration buildings, airfields, etc.), the criticality of those facilities to the mission of the installation, and vulnerability issues associated with where the facilities are located (e.g., off-base, on-base, or enclaved on-base, i.e., within an interior perimeter inside the main perimeter of the installation). Additionally, some facilities at military installations (generally airfields, piers, and maintenance facilities) will have war-fighting assets to include in the mission and economic value of the facility. We specifically focus on 38 different types of facilities, for example, piers, airfields, maintenance facilities, administration buildings, hospitals, utility equipment, communications equipment, etc. We also include one facility type representing the entire installation because of the chemical, biological, and radiological scenarios. Depending on the installation, these 39 facilities are classified as critical or significant and are denoted as off-base, on-base, or enclaved on-base, as appropriate for each type of facility. These variations resulted in a database of about 160 different types of facilities. For notation, let F be the set of possible facility types f.

The second step is to characterize the attack modes. Fig. 2 shows the 15 attack modes considered in our analysis sorted by relative technical ease.5 Technical ease specifically refers to the technical difficulty for an adversary to carry out one attack mode versus another and is a relative assessment provided in our case by a subject matter expert from Naval Intelligence. For example, our expert assessed the small arms attack as the easiest attack mode to conduct (technical ease = 1.0), and the other attack classes are considered relative to this attack mode (e.g., technical ease of radiological attack = 0.001, i.e., 1,000 times harder to execute than a small arms attack). This relative technical ease is a subjective assessment meant to capture the ease of acquiring necessary material, capability of handling and employing material, and ability to maneuver material to a position from which to launch an attack. Technical ease does not consider the vulnerability or consequences associated with an attack scenario, but instead is an important component in the threat submodel described later. We did not consider a nuclear attack in our set of attack modes because the enormity of the consequences makes it impossible to evaluate it in comparison to the other scenarios considered. For notation, let A be the set of possible attack modes a.

Figure 2.

Possible attack modes.

Next, we determine a plausible set of (attack mode, facility) pairs—called attack scenarios—by mapping attack modes to facilities. For example, water-based attacks can only be launched against assets located on or near water (e.g., ships and/or piers). The set of all conceivable attack scenarios is given as A×F. Using the concept of susceptibility from Ayyub et al.,(17) a facility is only considered susceptible (s = 1) to attack modes that can cause damage to that type of facility. Thus, the set of attack scenarios can be defined as X ={A×F | s = 1}. Also, if any attack scenario is judged too insignificant by the decisionmaker, it can be removed from X.

Fourth, mitigation alternatives and the issues associated with applying portfolios of alternatives need to be identified. Mitigation alternatives are generally different security upgrades or consequence mitigation measures. The Navy uses the term “capability area” to specifically describe technological anti-terrorism alternatives (as compared with procedural or personnel changes). Some of the mitigation alternatives considered in this study include technology to improve security surveillance, access control, waterside protection, medical response, inspections, and regional operations centers.6 For notation, let M be the set of possible mitigation alternatives m. A mitigation alternative is considered capable (c = 1) for attack modes that can be improved (i.e., likelihood or consequence reduced) with the application of that option. Thus, the set of mitigation alternatives can be defined as Y ={M×A | c = 1}. Also, if any mitigation alternative is considered too costly by the decisionmaker, it can be removed from Y.

The definition of the system and the choice of facility types, attack modes, and mitigation alternatives will expand the problem exponentially. We consider 160 facility types, 15 attack modes, and 22 mitigation alternatives with some simplifying assumptions and algorithms that will be described later in this article.

3.2. Assess Baseline Risk

A standard risk score is needed to compare the risk level of one attack scenario x to another in the set X and measure the effectiveness of mitigation alternatives yY. We use a derived risk score that is a function of a facility's vulnerability to an attack scenario, the consequences of the attack scenario if the attack is successful, and the relative threat likelihood of the attack scenario. We discuss each of these components in this section. Fig. 3 provides an overview of these risk model components.

Figure 3.

Risk scoring and prioritization model.

3.2.1. Determine Facility Vulnerability to Relevant Attacks

The standard technical definition of the vulnerability of a facility is the probability that an attack against that facility will succeed, given the attack was initiated.(20) A facility's (or system's) vulnerability is also called the system effectiveness.(13) Vulnerability assessment methods commonly look at the defender's ability to detect the attack, delay the adversary while marshalling a response (such as with barriers, doors, locks, vaults, etc.), and respond to the attack in a way to interrupt and neutralize the adversary.(13) The DoD uses a Detect, Assess, Warn, Defend, Recover (DAWDR) construct to assess protection of DoD assets, where DAWD functions occur before an attack and Recover functions occur after an attack. Table I summarizes the definitions of the DAWDR components. An attack is defeated if all DAWD functions are performed. The Navy divides the Defend function into an Interdict subfunction and a Neutralize subfunction. The attack succeeds if any of the DAWD functions fail. Because vulnerability is the probability of a successful attack, given the attack was initiated (V = p(Success | Attack)), the vulnerability can be calculated as one minus the probability of detecting, assessing, warning, interdicting, and neutralizing the adversary:

image(1)

In most cases, the vulnerability data need to be assessed by subject matter experts. The subject matter experts who provided vulnerability and risk mitigation data for our study were anti-terrorism and force protection capability area systems engineers and coordinators who worked for Naval Facilities Engineering Command (NAVFAC). A reasonable assumption for military installations that we adopt in our data assessment is that vulnerabilities are determined by attack classes (groups of similar attacks) and facility location (i.e, off-base, on-base, or enclaved on-base). Specifically, all facilities in the same location (i.e., off-base, on-base, or enclaved on-base) are considered equally vulnerable to attack modes of the same class (i.e., land, water-borne surface, water-borne subsurface, air, chemical, biological, or radiological). This reduces the data assessment task to 105 assessments (i.e., 3 locations × 7 attack classes × 5 DAWD parameters).

Table I.  DAWDR Protection Component Descriptions
Force Protection ComponentMeasureDescription
DetectionProbability of detectionProbability of detecting an adversary initiating an attack, given the adversary has launched an attack.
AssessProbability of threat identificationProbability of correctly identifying an activity as hostile or threatening, given the activity has been detected.
WarnProbability of warning (responders)Probability of warning responders of an imminent attack, given attack has been correctly identified as hostile.
DefendProbability of interdictingProbability of interdicting the path of the adversary toward the target, given the responders were warned.
Probability of neutralizingProbability of neutralizing the attacker before he or she can cause adverse effects, given the attacker has been intercepted.
RecoverPercent decrease in personnel loss, mission impact, and economic lossRecovery efforts reduce the negative effects of an attack that successfully penetrates the protective measures.

3.2.2. Determine the Consequences of Each Scenario

The DoD directives require consideration of three consequence components: mission impact, personnel loss, and economic loss. Utility theory is still the best tool for aggregating multiple attributes.(14) The consequence of a successful attack scenario as defined in the ARDA process is a function of: (1) the percentage of the estimated damage to the facility from the attack mode, (2) the mission value of the damaged facility, (3) the personnel in or around the facility who would be killed by the attack, (4) the plant replacement value (PRV) measured in dollars ($) to repair or replace the damaged facility, and (5) the monetary value of any war-fighting assets that would be damaged in the attack. We sum the two latter dollar attributes into one economic loss attribute.

An aggregate consequence score is then calculated as the weighted sum of the mission impact score, the personnel impact score, and the economic impact score, where each individual attribute is a utility score based on the estimated consequences for that attribute. Fig. 4 shows the three attributes (mission impact, personnel loss, and economic impact). The utility score for each attribute actually represents the “dis-utility” for the outcome because lower numbers are preferred and is scaled between 0 and 1,000, where 1,000 is the worst plausible outcome on that attribute.

Figure 4.

Multiattribute consequence assessment.

The utility score for mission impact is derived from two components: the mission value of a facility and the percentage of the facility damaged. These two components are necessary to differentiate between minor damage to a critical facility and complete destruction of a less critical facility. A two-dimensional matrix is used to assess the utility scoring function for mission impact. The matrix is assessed with eight categories of mission value ranging from a low of 16 (the minimum in any database) to a high of 1,000 and six levels of damage partitioning the consequences between no damage and 100% destruction. The mission value is a function of a Navy indicator of installation mission criticality (called Required Operational Capability (ROC)) and a Navy assessment of the criticality of a facility relative to the installation mission (called Mission Dependency Index (MDI)). MDI is a function of the time to resume facility functionality and dependency of other Navy activities on a fully functioning facility. We applied an installation criticality weighting associated with the facility's ROC level to a facility's MDI (scaled 0 to 100). With this scoring of facility criticality, we were able to assess a facility's overall value to the Navy as a function of its specific mission on an installation and the degree to which other Navy functions were dependent on this facility.

The percentage of facility damaged quantifies only the physical damage to the asset as defined, for example, from explosive blast models (i.e., a damage model might predict based on many factors, including the quantity of explosives, distance from building, construction materials of building, etc., that a certain percentage of the building is expected to be damaged in an attack of a particular type). One-hundred percent damage to a main installation (with mission value of 1,000) has the highest mission impact of 1,000 and, for example, 10% damage to a low criticality facility should have a mission impact score close to 0. Utility scores between the discrete column and row preferences in the mission impact matrix are derived using linear interpolation.

The number of casualties from a successful attack is estimated by multiplying the percentage of the facility damaged by the number of personnel in or around a facility. The results were examined for linear utility functions, concave functions that assigned higher impact scores for low personnel losses, and convex functions that assigned low scores for losses less than 100 personnel and increasing greatly after that point. The decisionmaker who provided our assessments considered the function shown in Fig. 4 as most appropriate, that is, any loss of life in a terrorist attack would be highly significant.

The economic impact is a function of the PRV of the facility, the monetary value of the war-fighting assets typically located at a particular facility (i.e., pier, airfield, or maintenance facility), and the percentage of facility damage from the attack mode. Similar to personnel impact, three utility functions were examined for economic losses: linear, concave, and convex. Our decisionmaker considered a linear utility function, as shown in Fig. 4, to be appropriate for economic losses.

Data to complete the consequence assessment were available from several sources. The Defense Threat Reduction Agency has models and databases that provide estimates of damage to different facility types from various attack modes, in particular, from explosives and hazardous agents. Additionally, average personnel loading data exist for most facility types or can be approximated based on the size or mission of the facility. For example, a reasonable assumption is 1 person per 150 square feet in a training facility or administration building and 2 persons per aircraft on the flight line. Data exist as part of the Base Re-Alignment and Closure (BRAC) studies for the total number of personnel assigned to each major military installation. Databases also exist, with the PRV in dollars, for major facilities and war-fighting assets. Finally, most Navy facilities have an MDI assigned in a database that identifies it as critical, significant, relevant, moderate, or low importance. Thus, gathering the consequence data is a straightforward task.

Finally, weights are required for aggregating among the three consequence attributes. Our decisionmaker was educated to consider the range of potential consequences as well as the importance of each attribute (i.e., the relative importance of changing each attribute from its worst to best level of performance) when determining weights. From our discussions, weights of one-third, one-third, and one-third were chosen, with other alternative weights considered as part of the sensitivity analysis.

3.2.3. Determine Threat Scenario Likelihoods

Of the three components, threat, vulnerability, and consequence, reliable threat data are the most difficult to assess because we are dealing with an intelligent adversary.(13,14) To partially overcome this lack of threat data, relative probabilities are often used when comparing scenarios in a terrorist context.(21) A base probability of an attack against a U.S. Navy installation during the time frame of the analysis, in this case, five years, (denoted p(Attack)) was assessed by Navy intelligence experts. When using relative probabilities for attack scenarios, p(Attack) must reflect only the scenarios considered within the scope of the analysis. If fewer scenarios are examined, p(Attack) must be adjusted accordingly or else the computed risk scores will be inflated. The likelihood that any particular scenario xX would comprise the attack is computed from the assessed technical ease of the attack mode (see Section 3.1), the vulnerability of the facility to the attack mode (see Section 3.2.1), and the potential personnel consequences from a successful attack (see Section 3.2.2). With an expert from Naval Intelligence, we examined alternative functions to determine the most appropriate representation of an adversary's preferences for attack scenarios. For example, consequences of the attack could consider economic impacts, mission impact, personnel losses, or any combination of the three factors. On the basis of input from this intelligence subject matter expert, we determined that the scenario attractiveness (SA) score should be the product of the probability of success, given an attack was initiated (i.e., the facility vulnerability defined in the previous section), the average casualties during a successful attack scenario, and the technical ease factor of the attack:7

image(2)

This calculation assumes: (1) that given an equal ease of execution among attack scenario choices, the adversary is more likely to choose an attack scenario that yields the greatest expected number of casualties, (2) given an equal expected number of casualties among attack scenario choices, the adversary is more likely to initiate an attack scenario that is easier to execute, and (3) an attacker will be limited to an attack type a that he or she has the technical means to execute. Equation (2) explicitly captures the tradeoffs among adversary goals, for example, that easy attacks generally have less consequences. We then use an attractiveness score to calculate the probability an attacker will initiate each attack scenario xX.

The scenario likelihood (x) values used in the calculation of risk scores are determined as a relative probability based on that scenario's attractiveness relative to all other SA scores:

image(3)

Thus, Equation (3) is allocating a relative probability of attack to each scenario xɛX proportional to each scenario's attractiveness, as defined by Equation (2).

3.2.4. Describe Decisionmaker's Risk Preferences for Likelihood Versus Consequence and Calculate Baseline Risk Scores

Eliciting the decisionmaker's risk preferences is an important step in risk-based decision making because defining a threshold for what is an acceptable level of risk, by definition, must include the decisionmaker's values. Risk preferences among consequences are previously captured in the weights for the three consequence attributes and the three single attribute utility functions. Assessed risk preferences are also needed that consider likelihoods and consequences of different scenarios. The challenges are that the expected value is not always an appropriate representation of the acceptability of catastrophic events and their consequences,(15) and risk research has shown that people consistently weight consequences more heavily than likelihood.(16)

The risk scoring function assigns a risk score to a scenario as a function of the likelihood and the consequences. To use a utility function, preferences should be linear in relationship to the probability. For a risk score to be based on a utility function, the risk matrix is the product of the utility for consequences (C) and the probability (p) of that scenario with a linear utility for probability, as shown in Fig. 5(a):

image(4)
Figure 5.

Figure 5.

(a) Risk utility function with risk scores for likelihoods and consequences. (b) Decisionmaker's risk matrix with assessed risk scores for likelihoods and consequences.

Figure 5.

Figure 5.

(a) Risk utility function with risk scores for likelihoods and consequences. (b) Decisionmaker's risk matrix with assessed risk scores for likelihoods and consequences.

Unfortunately, decisionmakers do not always exhibit a linear utility for probability, especially for scenarios involving catastrophic events. If this is the case, a two-dimensional risk scoring value function8 that captures the decisionmakers’ preferences for probability and consequence can be assessed as:

image(5)

Risk scores are scaled between 0 and 1,000, with a risk score of 1,000 corresponding to the scenario with the highest likelihood of impact and the highest levels of mission impact, personnel impact, and economic impact; thus, multiple scenarios can have equivalent risk scores even if they have different likelihoods of successful attack and different impacts. For example, the destruction of a communications station may have a higher likelihood of successful attack and a lower impact than destruction of a pier with berthed ships, but the likelihood-impact pair when considered together might be equal in risk from the decisionmaker's perspective. Risk scores assessed by our decisionmaker appear on the risk matrix in Fig. 5(b). Risk scores between the discrete column and row preferences in the matrix are derived using linear interpolation. The preferences assessed by our decisionmaker were not linear for probability and thus reflect a risk score value function and do not satisfy the axioms of utility theory. The challenges and issues associated with this value function will be discussed further in Section 5.

The baseline risk scores provide a quantitative measure of the relative degree of risk associated with each scenario x. The quantitative measure of risk is necessary to subsequently rank the effectiveness of risk mitigation alternatives. However, we found that decisionmakers may prefer a less quantitative presentation of risk, and that a red-yellow-green risk matrix is an effective qualitative way to communicate the degree of risk to decisionmakers.

3.3. Assess Risk Mitigation Alternatives

The risk score provides a standard to compare possible improvements from alternative anti-terrorism mitigation alternatives. We use some simplifying assumptions to avoid the challenge of assessing the portfolio combinations of all 22 alternatives (222 combinations, ∼4.2 million combinations). Initially, the study team worked with NAVFAC subject matter experts to identify the primary DAWDR functions performed by each mitigation alternative, that is, some improved detection, other assessment, etc. Not all alternatives improved all DAWDR functions, and not all alternatives are effective against all attack classes.

We worked with the same NAVFAC subject matter experts to assess DAWDR data for risk mitigation options for a representative installation. The experts were asked how each alternative yY would improve the baseline protection, assuming that the individual alternative is the only enhancement added. Additionally, the experts were also asked how much the maximum portfolio (MP) of risk mitigation options would improve the baseline protection, where the MP of risk mitigation alternatives was defined as the best DAWDR performance achievable between 2007 and 2015 if the Anti-terrorism/Force Protection (AT/FP) program and all mitigation areas in set Y are well-resourced.

Interpolation is used to determine the effects of portfolios of risk mitigation alternatives rather than trying to assess all combinations. The algorithm uses both the assessed DAWDR performance data for each alternative, assuming that the individual alternative is the only enhancement to the baseline protection and the MP if all the alternatives are applied.

Using the DAWDR data, we recalculate facility vulnerabilities in each scenario with each individual capability assigned and the MP assigned. These new facility vulnerabilities are determined by updating Equation (1) (see Section 3.2.1) using the improved DAWD estimates based on the portfolios of risk mitigation options. We recalculate the consequences in each scenario (see Section 3.2.2) based on improved recovery estimates and the threat scenario likelihood (see Section 3.2.3) based on the recalculated vulnerabilities and consequences. Finally, we recalculate the risk scores for each attack scenario (see Section 3.2.4).

For example, assume the estimated baseline probability of detection given attack p(Detect | Attack)baseline is 0.5 and the assessed probability of detection given attack with the MP of mitigation alternatives applied is p(Detect | Attack)MP = 0.8. The benefit of each individual mitigation alternative y contained in the MP is also assessed and, by definition, must fall in the 0.5–0.8 range for its performance. The analysis assumes that alternatives that degraded capabilities would be eliminated from consideration.

To determine the effects of portfolios of mitigation alternatives in ARDA, we interpolate. Interpolation apportions the difference between the MP and the best alternative in the portfolio in proportion to each individual alternative's performance. This algorithm is based on portfolio performance estimation techniques developed in Buckshaw et al.(22) The equation for the interpolation algorithm, for example, for p(Detect | Attack) for a portfolio n, is:

image(6)

A similar algorithm is used for each DAWD parameter. Fig. 6 shows an example of portfolio calculation for p(Detect | Attack).

Figure 6.

Interpolation algorithm example.

Additionally, some mitigation alternatives improve on baseline Recover capabilities and reduce the consequences of the attack scenario. The exact same procedure, described above for assessing the effects of individual and MPs of mitigation alternatives and interpolating the combinations, is used to assess consequence reduction. Note that using this to assess the value of alternative portfolios assumes independence among mitigation alternatives. This was true in our case where the 22 mitigation areas were defined at a broad, high level (i.e., improve access control). We did not evaluate all the possible methods to improve access control, but instead assumed that the set of alternatives proposed were already identified as the best within the area. However, our method could include multiple system alternatives within an area.

LaTourrette et al.(18) examine 39 potential security options for improving security at shopping centers. As part of their data assessment, they allow options that are not mutually exclusive, but they identify these overlaps. Then, their prioritization will not add a redundant capability once a similar capability is chosen in the portfolio. We can adopt a similar approach if alternatives are not mutually exclusive.

As AT/FP mitigation alternatives are added to improve the baseline protection of a facility, a reduction in vulnerability and/or consequence leads to a change in the attractiveness score of an attack scenario. The model recalculates the threat scenario likelihood for each AT/FP portfolio examined in the analysis by replacing the baseline attractiveness score for the scenario with the adjusted attractiveness score calculated with the AT/FP mitigation alternatives in place. The adjusted attractiveness score is calculated using the adjusted vulnerability and consequence data from the respective submodels.

Because the baseline threat scenario likelihoods are relative probabilities, and we have applied mitigation alternatives to reduce these relative probabilities, these values no longer necessarily sum to the baseline p(Attack). One could consider a sum of the updated threat scenario likelihoods as a measure of the degree of an adversary's preference for attacking a U.S. Navy facility in comparison to the baseline conditions that provided the context for the p(Attack) estimate. The reduced sum could be considered a measure of deterrence. This reduction in threat scenario likelihoods also assumes that, by applying the risk mitigation alternative, the risk does not simply shift to another target within this analysis. Instead, we assume the displaced probability of attack goes to targets or scenarios outside the scope of the analysis.

Predictions of adversary intentions are complex and difficult. ARDA provides a traceable, transparent submodel that reflects primary drivers of adversary motivations as they relate to attacks on U.S. Navy facilities. Analysts should work with AT/FP intelligence experts to reassess initial p(Attack) estimates and adversary attack scenario preference assumptions to ensure that the updated threat scenario likelihoods are realistic.

3.4. Ranking Risk Mitigation Alternatives

Using the assessed risk mitigation effectiveness data, we can rank mitigation alternatives based on the subsequent risk reduction impact of adding the next alternative to the portfolio. The goal is to determine an order of implementation of mitigation alternatives for facilities based on risk reduction effectiveness with or without acceptable risk thresholds. If costs are available, the prioritization can be cost-benefit rather than simply risk-benefit. But currently, we do not have the required cost investment for mitigation alternatives because the decisionmakers were primarily concerned with risk reduction benefit that could be achieved from the identified alternatives, all of which were receiving some level of investment.9

We assume that the risk reduction effectiveness can be measured as the sum of the reduction in the risk score for each relevant attack scenario for the facility. Then, the recommended implementation is based on the risk reduction across attack modes when a package of mitigation alternatives is assigned to that facility. To identify an order of implementation for each facility, we review the mitigation areas relevant to that facility as defined by set Y. We examine any relevant mitigation area not yet implemented to calculate the scenario risk reduction score if each were applied. The mitigation alternative that reduces the risk the most is added next. The threat scenario likelihood, the vulnerability, and the consequence for that facility are recalculated when the new mitigation alternative is added. If there are remaining relevant mitigation areas, the process is repeated until a complete order of implementation is specified.

Additionally, a decisionmaker could assign an acceptable level of risk for each facility to identify the maximum acceptable risk score for any plausible attack scenario. This will prevent the prioritization algorithm from adding a mitigation alternative to a facility's portfolio if the facility is currently below its risk acceptance level. If a facility is already below its acceptable risk threshold, the model defers implementation of additional alternatives to this facility until all other facilities are below their acceptable risk threshold. The red-yellow-green risk score matrix can serve as a useful tool for communicating acceptable risk. For example, the “green” zone could represent an acceptable threshold but, as will be discussed later, mitigation alternatives must exist that can lower scenarios to the acceptable level. Fig. 7 illustrates the portfolio prioritization concept. From baseline conditions for an attack on a facility, the model evaluates mitigation alternatives A and B to determine which to implement first. Alternative A delivers greater risk reduction than alternative B, so A would be added first. If the decisionmaker has established the acceptable risk threshold indicated by the dashed line, another alternative is required, so the mitigation portfolio is recalculated, with B also included.

Figure 7.

Example of order of implementation for two mitigation alternatives.

4. RESULTS

The results from our baseline analysis of two U.S. Navy installations are highly intuitive. The high-risk scenarios are improvised explosive devices (IED) attacks on population concentrations (i.e., large vehicle-borne IED on a hospital, small or large water-borne IED on a carrier). This result is consistent with LaTourrette et al.'s(18) that found that explosive attacks are most common in historical analysis of past terrorism events (nearly 50% in their sample). The medium-risk scenarios are of lower likelihood but have high impact such as chemical and biological attacks against the entire installation or a large or small aircraft attack on a carrier or maintenance facility. The lowest-risk scenarios are small arms attacks and attacks against lower mission critical targets such as utilities, navigational aids, morale, welfare, and recreation (MWR) facilities, and ammunition storage.

The results for the prioritization found that mitigation alternatives that addressed clear risk areas such as waterside barriers should be a high priority. Additionally, mitigation alternatives that could help reduce the likelihood of many scenarios (i.e., vehicle inspections) or the recovery aspect in many scenarios (i.e., an operations center) scored better than those that helped very specific scenarios (i.e., chemical, biological, radiological, and nuclear (CBRN) surveillance). The analysis team reviewed the initial risk assessments with a senior security officer in the Navy, and he felt the risk score generally conformed to his professional assessment. The prioritized countermeasures also conformed to subject matter expert assessments derived in other Navy analyses.

In a sensitivity analysis, we examined changing the weights on the three consequence attributes (mission loss, economic loss, and personnel loss). If a greater proportion of the weight is allocated to the mission loss, this results in minor deviations in relative risk ranking among scenarios. There is an increase in the risk of attacks on waterfront facilities due to increased emphasis on mission value of ships, and there are reductions in all the risk scores for attack scenarios against heavily populated facilities without mission assets (such as hospitals). If a greater proportion of the weight is allocated to personnel loss, there are very few deviations in the relative risk ranking of scenarios, but all the risk scores increase for attack scenarios against heavily populated facilities (such as waterfront facilities with large concentrations of personnel, maintenance facilities, etc.). We also examined changing the shape of the single-attribute utility function for personnel losses to a linear function. Similar to the above analysis, there were only minor adjustments in the relative risk ranking. There were, however, major decreases in all the risk scores due to lower scores for personnel losses. Finally, we examined the effect of using a risk scoring function that is linear for probability (Fig. 5a) for combining likelihoods and consequences rather than using the value-based risk scoring function (Fig. 5b). There were only minor adjustments in the relative risk ranking, but there were major decreases in the risk scores from the baseline. This is because the assessed risk scoring value function is “consequence averse.” In the assessed risk scoring function, there is significantly more weight in the upper left quadrant than in the lower right quadrant, and this is not captured in a scoring function that is linear for probability.

5. DISCUSSION AND OBSERVATIONS

Although not entirely solving all the challenges identified in Section 2, we make progress in each of them. We generate comparable risk scores across multiple facilities and attack scenarios. The risk score can be updated when risk mitigation alternatives are implemented, and improvements in risk can be measured against the original baseline scores. We can prioritize resources and clearly identify why certain areas should receive more resources. Using Equation (6), we approximate the contribution of each additional mitigation option in a portfolio, and using Equations (2) and (3), we provide an approach for consistently estimating the relative threat likelihood for scenarios in a defensible way. Finally, we explicitly document the preferences of the decisionmaker and do not obfuscate the tradeoff between low-probability, high-consequence events and high-probability, low-consequence events. We do, however, identify other challenges.

First, with lots of scenarios, no one scenario has a significant likelihood of occurring. If you assume a p(Attack) in the range of 0.1–0.2 for an attack against a U.S. Navy facility in the next five years, then that seems significant. But the probability that any one of the more than 100 facilities at any one of the more than 60 installations in the United States is the target of the attack is small. We do identify scenarios that were twice as likely as other scenarios, but all of the threat likelihoods are small, less than 0.004, for an individual scenario.

Second, although we capture the decisionmaker's preferences for risk and include a step to prioritize mitigation alternatives to reduce the risk below identified thresholds, in reality, such mitigation options do not always exist. Specifically, when assessing the decisionmaker's preferences for risk in terms of the risk matrix shown previously in Fig. 5(b), our subject matter expert considered all scenarios with the worst consequences (i.e., the top row in the matrix) to be a “red” risk.10 Most mitigation alternatives are effective at lowering the likelihood of the event; very few, especially with explosive attacks, can mitigate the consequences. If a scenario is red because of the potential consequences, and mitigation alternatives can only reduce the likelihood but not the consequence, no option exists to move this scenario out of the red zone. Additionally, by being consequence adverse (and not linear as a function of probability), our decisionmaker will violate the axioms of utility theory in some situations involving small-probability events. Descriptively, our model represents the concerns of our decisionmakers, but also shows that if we (as a society) also insist on overly weighting consequences, we are probably spending too much trying to mitigate small risks that we should just live with.

Third, although we had clear steps and willing participants, most of our subject matter experts reported low confidence in their DAWDR performance estimates. Even if the data are not precise, if they are relatively coherent, it can help make prioritization decisions. Additionally, much work is being done to develop simulation tools to improve the modeling of vulnerabilities and risk mitigation options,(24–26) so our understanding of these data will continue to improve in the future.

Finally, as the model evolved and provided more insight, the question regarding mitigation alternatives shifted and costs became more of a focus. The question became when is it appropriate to invest in certain mitigation alternatives (i.e., the expected costs of the damage from an attack outweigh the costs of mitigation)? The model and data gathered could support this answer. This additional analysis could be done using the risk score metric, but for the final analysis that we did, we focused on only the replacement cost of lost assets and the casualties, assuming, for example, $1 million per fatality. Fig. 8 shows the tradeoff space for two different example sites. Both sites have approximately the same costs to implement the mitigation alternatives, but Site 1 has significantly more expensive war-fighting assets that would be damaged in the attack. As shown in Fig. 8, investing in the example risk mitigation alternatives estimated for Site 2 is not justifiable, given the range of feasible attack probabilities; however, investments for Site 1 may be justifiable, even though they are more costly.

Figure 8.

Risk tradeoffs.

6. CONCLUSIONS

We helped the U.S. Navy identify effective mitigation alternatives that reduce risk to Navy facilities. The same model can include other target sets with minor modifications. We use a comprehensive examination of scenarios of attacks and portfolios of mitigation alternatives to reduce risk by reducing either vulnerability or impact. The benefits of the model are numerous. The model provides a repeatable, systematic approach to examine a multitude of attack scenarios and thousands of risk mitigation portfolios documenting both data and preference assessments. It provides a way to examine risk reduction effectiveness in a more deliberate, traceable manner. We also developed a submodel that includes a thinking adversary and the adversary's motivations in attacking. The model can be updated dynamically to consider new threats, vulnerabilities, and consequences as we implement risk mitigation, thus benchmarking our progress in achieving risk reduction from terrorist attacks.

Footnotes

  • 4

    For example, LaTourrette et al.(18) present a quantitative model for prioritizing security measures at shopping centers.

  • 5

    Technical ease factors are assessed for each attack mode, but the exact numbers are not reported here due to the sensitivity of the information.

  • 6

    The Navy Antiterrorism Technology Coordination Office identified 22 risk mitigation research areas (i.e., capabilities) in a “Functional Needs Analysis.”(19) These 22 areas were examined in this study.

  • 7

    As stated in the text, we recognize that interdependencies exist between the components in the SA formula. Reviewers have suggested that covariance terms should thus be included in the formula. Revisions to this formula will be examined in future work.

  • 8

    This value function will no longer be a utility function because if the decisionmaker's preference is not linear in probability the function may not satisfy the axioms of expected utility theory.

  • 9

    We recognize that this is imperfect, and that, in reality, a priority list can become totally reordered by a small change in budget.

  • 10

    Cox(23) explains normatively some of the problems with the decisionmaker's preferences described in the risk matrix in Fig. 5(b). His article was published after we completed our study.

ACKNOWLEDGMENTS

The U.S. Navy supported the work described in this article through a contract with Innovative Decisions, Inc. We thank Commander Michael Williamson, Mary MacDonald, David Brown, James Chinnis, Harry Newton, and Greg Parnell for their major contributions to this study. We also thank Metron, Inc., for their interesting work in developing vulnerability simulations for various attack scenarios against U.S. Navy facilities.

Ancillary