Modeling, simulation, and trade-off analysis for multirobot, multioperator surveillance

Asunmannedvehiclesbecomesmallerandmoreautonomous,itisbecomingfeasibleto use them in large groups with comparatively few human operators. Design and analysis of such distributed systems are complicated by the many interactions among agents and phenomena of human behavior. In particular, human susceptibility to fatigue and cognitive overload can introduce errors and uncertainty into the system. In this paper, we demonstrate how advanced computational tools can help to overcome these engineering difficulties by optimizing multirobot, multioperator surveillance systems for cost,speed,accuracy,andstealthaccordingtodiverseuserpreferencesinmultiplecase studies. The tool developed is a graphical user interface that returns the optimal number and mix of diverse agent types as a function of the user’s trade-off preferences. System performance prediction relies on a multiagent simulation with submodels for human operators, fixed-wing unmanned aerial vehicles (UAVs), quadrotor UAVs, and flapping wing UAVs combined in different numbers.


INTRODUCTION
The advent of affordable, autonomous unmanned aerial vehicles (UAVs) is creating opportunities to deploy heterogeneous UAV teams that work in concert with human operators in various missions. These systems are scalable and adaptable, which makes them attractive for deployment in uncertain and adversarial environments. They can perform several functions simultaneously, and mixing diverse asset types enables deployment of complementary capabilities. Use cases of multiagent, multioperator systems have been envisioned for the military, agriculture, and infrastructure domains among many others. But the complex interactions of the many agents and humans F I G U R E 1 Conceptual framework for this study. The individual agent models, combined with a mission scenario, are executed in a multiagent simulation to develop a database of system configurations and their performance. The user can interact with this database to score system configurations according to his preferences.
interactions among the many agents make analysis difficult. When human behavior also affects the system, the complex and nonmonotonic effects of human factors 2 and humans' inherent diversity 3 can make the system's behavior even more unpredictable. For example, it has been shown that adding semi-autonomous assets to a system only increases system-level performance up to a certain number of agents, and may have no effect or even a detrimental effect if more are added. [4][5][6] Humans' stress, emotional state, and physical condition may all add variability to human-agent team performance, and the workload must be carefully balanced among humans and automated systems. 7 Using human performance simulation can aid systems engineers in balancing workload among the operators, preventing boredom and mistakes, and maximizing the throughput of the overall system. 8 This is especially helpful when hardware prototypes do not exist, or human-system experimentation is otherwise infeasible. 9 To overcome the challenges of system analysis and take advantage of the potential for customization, we have created an integrated simulation-optimization framework in the design of multiagent systems. We pursued three objectives in developing this framework: • Create a multiagent simulation for predicting system-level metrics as a function of agent type and number. The simulation must have a submodel for human operators that accounts for known phenomena in human-automation interaction.
• Create a trade-off framework for ranking different system configurations. The framework must account for multiple metrics, and the decision-maker's weightings among attributes.
• Integrate the simulation, trade-off framework, and a user-friendly graphical user interface (GUI), to allow a decision-maker with little technical knowledge of the simulation to input preferences and be automatically presented system configurations that are optimized for those preferences.
Together, these represent an instantiation of the decision support system shown in Figure 1, where a decision maker is able to input preferences on objectives to query a database of simulated system configurations. In our use case, the metrics that the decision-maker must weight are cost, accuracy, speed, and stealth, where the number of operators and the total number of UAVs are used as proxies for system cost, and stealth is measured by the total noise level of autonomous agents throughout the environment in dB.

Agent-based modeling and simulation
In agent-based modeling, complex systems are represented as collections of agents. The agents have an individual state, and they can react to local environmental conditions while proactively pursuing their own goals. 10 With humans modeled as agents, their behavior can be simulated 9 as part of the system interacting with other humans or robots. The system-level behavior that emerges is very difficult or impossible to predict from analytical equations, but can be simulated using multiagent simulation. Large scale, distributed, and communications-limited systems are well suited to be modeled as collections of agents. For example, agent-based models have been used in the healthcare, transportation, warfare, and disaster response domains. 11 Multiagent simulations with imperfect information are especially suitable for analyzing military engagements. 12

Trade-offs in multiattribute decision-making
Decision-making is an essential element of the systems engineering process. 13 To make effective decisions in complex environments, decision-makers should clearly define the problem, involve stakeholders continuously, use appropriate models, properly develop objectives, and effectively communicate recommendations. 14 Simulation-based recommender systems may positively contribute to these principles by enabling precise definition of behaviors, outcomes, and metrics; providing accessibility to non-expert stakeholders; narrowing the modeling scope to tractable agent-level phenomena and interactions; and tracing decisions to the models and metrics.
Designers and leaders must often select among alternatives whose worth is a function of multiple competing metrics. The process involves identifying metrics, documenting preferences toward metrics, measuring system performance according to the metrics, and ranking system performance according to the preferences. 13 Cost is a ubiquitous metric. Speed, accuracy, and stealth can also be important in specific contexts. Acceptable trade-offs among these objectives depend on preferences derived from the mission. While trade-offs must be made at every stage of a system's lifecycle, 15 here we focus on the deployment and operation stages, where groups of assets must be sized appropriately for the mission.

Overview and contributions
This paper presents research on modeling, simulation, and goal tradeoffs of multirobot, multioperator surveillance systems, described in In Section 2, we present related work in the field of multioperator, multirobot systems. In Sections 3-5, we present the surveillance scenario, human and agent models, simulation, and decision-maker's GUI. In Section 6, we explain our methods for integrating and applying all these tools. Sections 7-8 present the trade-off framework and case studies. We conclude with a summary and discuss future work in Section 9.

RELATED WORK
The outputs of simulation are often used to inform decision-makers in interdisciplinary problems. 10

Models and simulation of human performance
To incorporate models of humans into complex systems, we are primarily interested in their throughput (to estimate their effects on system-level performance), and their workload (affecting both tolerability of the task and performance). Harriot and Adams provide an in-depth review of the factors, outputs, and engineering relevance of human performance modeling. 9 Here we focus on models of cognition and decision-making for humans interacting with autonomous systems.
Hooey et al. provide an in-depth taxonomy of the elements of workload that affect UAV operators, including environment, task, equipment, and operator characteristics. 19 Systems designers must account for myriad sources of workload, and compensate for both underload and overload conditions, using interventions such as advanced user interfaces, decision support tools, and automation. 19 The drift-diffusion 20 model (DDM) of human decision making considers a decision to be made once a certainty threshold is reached. The human's certainty drifts, partially stochastically, during the process as relevant information is digested. Once the threshold is reached, the human makes a decision and takes action accordingly. Huang et al. 21 used DDM as an analytical input to develop a human-robot control method with task completion guarantees in finite time. Peters et al. 22 used DDM to optimize task allocation in a human-UAV persistent surveillance mission.

Multiple resource theory (MRT) is commonly used in situations
where a human is expected to multitask. 23 In MRT, the mind is mod-eled as a composite of processors for different types of tasks, such as fine motor input and listening. Overloading a processor or using several simultaneously can lead to high workload and lower performance. One of the most prevalent human work performance modeling tools, IMPRINT Pro, treats humans as MRT-based operators and uses discrete event simulation to predict performance. 8 Using this tool, analysts can construct task sequences and simulate them to predict and prevent unacceptable spikes in workload. IMPRINT has been used to predict workload of UAV remote pilots, 23 military intelligence analysis, 8 and teams operating a semiautonomous UAV. 24 Harriott and Adams 9 provide a review of many prevalent cognitive models, such as GOMS 25 and ACT-R. 26

Simulated human behavior in human-agent teams
Several researchers have simulated human behavior in order to study the system-level performance of human-robot teams. Using multimodeling, Golightly et al. 27 give an example of an operator-UAV team, where the operator's task sequence and the UAV's flight in windy conditions are modeled. In this way, surprising interactions were uncovered, such as windy conditions' decreasing wait time in some instances (due to serendipitous delays leading to more consistent operator workload). The Fusion and IMPACT projects 1,28 are an attempt to create agile teams of humans and multiple UAVs through advanced user interfaces, sports-inspired command schemes, cognitive modeling plugins, and artificial intelligence. Similar studies on simulated human-UAV teams have investigated the performance of mixed asset types 29,30 and the impact on speed and accuracy of various human-automation workflows. 31

Placement of current work
While agent-based simulation for optimization and decision making has a long history, there is comparatively less literature on simulating human cognition within a multiagent system. This paper draws upon empirical and phenomenological findings in the human factors and cognition literature to develop operationalized, agent-based models of UAV operators. This work accounts for human workload, fatigue, and time allotted to make decisions. It is, to our knowledge, the first example of using human cognitive modeling and multiagent simulation for sizing of scalable multioperator, multirobot teams.

Mission
In our chosen scenario, a military commander is deciding on a route to safely move personnel through a populated area. It is known that there is a mixture of adversaries and neutral bystanders in the area, and that they may appear similar from distant observation (e.g., it may be difficult to determine if a large truck is a military or civilian vehicle).
The commander has at his disposal several UAVs and operators, and must use them to quickly map out all threats in the field. The objective is to gather this intelligence quickly and accurately, while minimizing the number of UAVs, the number of operators, and the noise impact of the flights.

Workflow
The workflow begins with a field with points of interest (POI) distributed throughout at unknown locations. A POI can be either a threat or an initially suspicious but ultimately non-threatening object in the environment. The workflow output is a location of every threat and confirmation of the non-threats. It follows four steps: Multiple POIs can be processed in parallel and will be at different stages of the workflow at any given time. The speed of processing the two queues must be balanced to prevent bottlenecks in the overall system.

MODELS
The models here are refined from our prior work, 16 with a new ornithopter model added for this paper. Note that for the following sections we use these abbreviations for the drones: fixed-wing UAV (FXW), flapping-wing UAV (FPW), and quadcopter (QCP).

Environment and mission
The field is 3 km × 3 km and contains 50 POIs. The POIs are static, and their location is initially unknown to the reconnaissance system. There are 25 each of true threats and false positives.

Humans
Operators' behavior accounts for human factors at various levels of task loading. The humans are processors of event queues. These events are generated by the FXWs, QCPs, and FPWs. In one queue, there is a list of POIs that must be assigned to QCPs and FPWs, and the other contains imagery that the drones have captured and must be classified as a threat or non-threat.
Operators self-assign to process a queue when they are free. For this simulation, they attend to assigning QCPs and FPWs first, and then process images once the assignment queue is exhausted. Operators perform only one task at a time. 30,32 Humans' performance is affected by their cognitive workload, their level of fatigue, and the time they are allotted to make decisions. 2,33 Accordingly, we use the following equation to model their accuracy in where I is the inherent skill of the operator, in the range [0-1]. The where time t is measured in hours. The factor k f is the accuracy drop per hour and must be determined empirically. The formula for K f must be defined over a range such that the factor does not drop below 0.
The factor for workload, K w , varies according to an operator's previous 5 min of work history. It is a function of utilization, the percentage of the previous 5 min that was spent in active work. When teaming with automated systems, human performance degrades if workload is too low or too high. 2 Boredom and disengagement cause performance degradation at low workload. Stress and mental overload cause it at high workload. [37][38][39] The exact decrements must be determined F I G U R E 3 Curve defining the performance factor due to utilization. Notionally, such curves will anticipate poorer performance at both the low and high ends of utilization. The curve for this study assumes peak performance in the range 40% -70%.
empirically, but the literature shows that performance typically peaks at utilization in the range of 40% -70%. 19 The particular function for K w that we use is shown in Figure 3. Other, empirically-derived, curves would be expected to have a similar "inverted U" shape. We use a piecewise function to capture this shape: The factor for speed reflects the speed-accuracy trade-off from psychology literature. 33,40 In empirical studies, decision accuracy as a function of time typically follows a sigmoidal function. This phenomenon reflects the phases of the decision making process: early in the process, the decision-maker can do no better than random guessing; after orienting himself with the problem, through analysis he can quickly become more accurate; at later stages, the curve asymptotes as he reaches diminishing returns on further time spent analyzing. An example sigmoidal curve 16 is given in Figure 4. In our model, K s , is a sigmoidal function of the time spent making a decision. To model rushed decisions, the operator is allotted a certain amount of time to make a decision, and the value for K s is taken as a function of this amount.
To model confident decisions, the operator is assumed to take enough time to reach a target K s value, and this time affects his throughput.
In the current application, we assume that the operator makes confident decisions, spending enough time to reach a K s value of 0.98, which requires approximately 30 s for images from a high quality camera, and 45 s for low quality cameras.
The operators also assign QCPs and FPWs to POIs. They first assign agents to any POIs near the road (see Figure 2), then to any remaining sites. Within each set of POIs, operators use a greedy heuristic for task assignment, forming agent-POI pairs in order of increasing distance.

F I G U R E 4
Example curve with sigmoid shape for accuracy as a function of time spent making a decision. At very early stages, the decision-maker can do no better than random guessing. At later stages, the accuracy asymptotically approaches a maximum.

TA B L E 1
Fixed-wing UAV flight performance variables.

Fixed-wing UAVs
The fast and high-endurance FXWs are assumed to fly completely autonomously, following preplanned routes, and to fully sweep the field once. The field is divided into sections that each FXW can surveil in parallel. Any POIs that are sensed are logged in a queue for the operators to eventually process. After completing their fixed routes, they land and are idle for the remainder of the mission. They have flight performance as defined in Table 1. The battery life of FXWs is high enough that we do not consider it here, but it would of course be an important variable in longer-duration missions.
FXWs fly at a constant velocity and altitude. They are unable to hover in place or make sharp turns; thus they turn following a Dubins Path in a constant-altitude plane. A Dubins path is the shortest planar, constant-speed path between two positions and headings under turning radius constraints. 41 FXWs have a sensing radius that delimits a circle on the ground directly below the FXW in which it can sense POIs.
They fly a "lawnmower" search pattern with the sensing circle of each leg slightly overlapping the previous leg until the entire field is swept.
It is challenging to perform image recognition from platforms that fly quickly at a high altitude. 42 The process of sensing a POI is abstracted in the simulation such that every threat is sensed as a POI once a FXW's sensory cone passes over it. An equal number of false

Quadcopter UAVs
The highly maneuverable but loud QCPs have performance as defined in Table 2. QCPs can move vertically, are not subject to turning radius constraints, and can hover in place. When commanded by an operator, they rise vertically to their transit altitude to minimize noise impact during transit. They then fly at a constant altitude to their target POI, descend vertically to their imaging altitude, and fly a circle around the POI to obtain surveillance. After imaging the POI, they return to their transit altitude and hover while awaiting further commands.

Flapping-wing UAVs
The FPWs, or ornithopters, are slower than conventional aircraft and less maneuverable. They are more difficult to fly stably in adverse weather conditions, as their large wing surface area and slow cruise velocity make them sensitive to gusts, which can also destabilize onboard photography. Their low lift capacity restricts the size of batteries and payloads that they can carry. 43 FPWs do have certain advantages for stealth missions, however.
With proper design, ornithopters can be very quiet 44,45 and inconspicuously blend in with their environment. 46 The performance parameters of the FPWs in our simulation are given in Table 3

Noise
Auditory noise in our work is calculated at simulated sensing points evenly spaced throughout the field. These points calculate the total noise level as a function of the position and noise profile of every UAV.
We model the UAVs as point sources of noise that propagates uniformly (i.e., spherically) in 3D space. Given a noise level in dB at a reference distance from a spherically propagating source, the level at any other distance from the source can be calculated from Equation (3), where L is the sensed noise level, r is the distance from the source to the sensed point, and L 0 is a known noise level, in dB, at a known distance r 0 .
Multiple sources of noise pressure can be combined using equation (4), where the noise levels L i are given in dB and i iterates over every source.

SIMULATION AND USER INTERFACE
The simulation is executed in GAMA, 50 an open-source agent-based simulation platform. We developed several custom extensions to GAMA for UAV behavior. It can be run in either graphical or headless mode, which runs with no visualization and is used for batch experiments in the trade-off analysis. An example of a graphical simulation is shown in Figure 2. User-friendly dashboards are an important tool to enable decisionmakers to interact with datasets and gain insights. This is especially true when the datasets are generated with simulations that require technical skill to configure and execute. With an intuitive front-end, diverse stakeholders, including non-technical stakeholders, can quickly extract and understand the information they need.
In this work, we created a Shiny 51 application in R. 52 Shiny is a package that allows data scientists to create interactive web applications with R as the backend. It provides many common user interface elements such as buttons, sliders, and monitors, and allows for extensibility.
The main requirement for our application is to recommend combinations of operators, FXWs, FPWs, and QCPs that best meet a decision-maker's preferences over the design goals of cost, speed, accuracy, and stealth, within constraints. It therefore must offer an interface for the user to easily input preferences and constraints.
In addition to the trade-off section, the application contains a landing page, tutorial, and other information. The Simple Additive Weighting (SAW) section is shown in Figure 5. On this page, the user inputs constraints on the number of UAVs and operators and weights on the metrics with sliders. The user then clicks to trigger a calculation, which scores all system configurations in the database according to the metrics. A list of the top scoring configurations is then presented to the user.

Main tools
In this work, we primarily use GAMA, Microsoft Excel, SALib, 53,54 and R. We used these for simulation execution, design of experiments, and data analysis and presentation, respectively. We also created small wrapper programs that integrate these pieces of software by creating individual simulation files, queuing simulations for execution, and organizing the outputs. The overall framework is illustrated in

Design of experiments generation
Various options for Design of Experiments (DoE) exist to generate the most informative data in the most efficient manner. Because the number of variables that we change is reasonably small and the simulations run quickly, we decided to do a full factorial DoE, which is a parameter sweep that looks at every possible combination of variables within our range. We used four factors: the number of each of the three different UAV platforms, and the number of operators. Our factor bounds were 1-10 for operators, 1-10 for FXWs, 0-10 for QCPs, and 0-10 for FPWs, with the requirement that there be at least one FPW or QCP.

Simulation job execution
A single run of the simulation only takes several seconds to complete.
However, a standard computer was not capable of processing the large number of runs in a reasonable time. Therefore, we used a Linux-based HPC at the Naval Postgraduate School to run the simulations, with many test cases run in parallel. In total, the simulation runs took 134 h of computing time.
The inputs and results of each simulation run were saved to a CSV file on a separate line. The outputs were speed, measured in seconds

Noise scoring
There were 100 noise sensing points spaced in a grid throughout the simulated field that tracked the amount of time that location was subject to noise above certain levels. We organized the noise data into bands of dB. The bands were 0-20 dB, 5 dB bands in the range 20-55 dB, and above 55 dB. At every timestep, the running subtotal in each band was incremented by the length of the timestep (in seconds, 0.5 for this simulation) for every sensor reading in that band. Each test point was then given an overall noise score, which penalized high levels more severely than low levels, as detailed in Equation (5), where t * is the subtotal in a dB range at the end of the simulation, and the subscripts indicate the lower dB value of the range. The total score, S, was appended to the outputs of each test run. Lower S indicates a stealthier system.

Sensitivity analysis
We performed a global sensitivity analysis considering both the afore-  Table 4.
We used the Python library SALib 53,54 to implement the method of Saltelli 55 for calculating the Sobol indices of our variables' contributions to the outcomes of accuracy, duration, and noise score.
We use N of 500, which combined with Saltelli's efficient sampling algorithm, 55,56 required 6000 simulation runs distributed over the parameter space. We calculated the first-order sensitivity index and total sensitivity index for each parameter and metric. Results are discussed in section 8.1.

TRADE-OFF FRAMEWORK
SAW is a weighted sum method for multiattribute decision-making (MADM). 57 It is one of the simplest MADM techniques to implement, It relies on weights that reflect a decision-maker's attitudes toward trade-offs. A fitness score is calculated for every system configuration, and every test point is ranked according to its score. The score is a function of weights and subscores for the following metrics: number of operators, total number of UAVs, simulation duration, operator accuracy, and noise score. Cost is assumed to scale with the number of operators and UAVs, and stealth is assumed to decrease with increasing noise. A metric with a higher weight will be prioritized when selecting the final system, at the possible expense of lower-weighted metrics.
The subscores for each metric are normalized to the range 0-1. Normalization is essential for further calculation because the attributes are measured on different scales. 57 The normalization equation is given in Equation (6), where the index i iterates over test cases, the index j iterates over the optimization attributes, x ij is a raw value of a metric for a test case, and L j, * is a maximum or minimum value of a metric over all test cases.
The subscripts "best" and "worst" depend on the sense of the metric, as some are minimized and others are maximized. Equation (6) is only valid for preferences that are monotonic over the measurement range.
All preferences in our study meet this criterion.
In SAW, only the ratios between weights matter, not their respective absolute magnitudes, so one weight can be chosen arbitrarily. To make the tool intuitive, we instruct the user to give the most important attribute a weight of 100 and restrict all others to be at or below 100.
Although it is not strictly necessary for ranking, before further calculations, we normalize the weights so that they sum to 1 while maintaining the ratios of their magnitudes, using Equation (7), where n is the number of metrics. Combined with the normalization of subscores, this ensures that the fitness score for every test point will be in the range 0-1, and will be consistent across any sets of weights with consistent ratios.
The overall fitness score for any test row is calculated according to The system configurations are then ranked according to their fitness, with higher fitness resulting in a higher rank. The optimal system is the highest ranking configuration that meets all constraints.

RESULTS AND CASE STUDIES
We present the results of model sensitivity analysis and short case studies here to encapsulate typical use cases for the modeling, simulation, and trade-off framework we have established. The case studies will show comparisons between two different metric weightings, representing different decision-makers with unique goals. We also show the outputs from optimizing only a single metric at a time. The target user in these studies is a mission commander with limited manpower and a limited supply of UAVs. He would consult the application to select an optimal mix of humans and assets according to priority weighting on size, speed, stealth, and accuracy as they vary between missions. The case studies were chosen to elicit diverse weightings and thus diverse system configuration recommendations.

Sensitivity analysis results
The first-order and total effect Sobol indices for our model parameters are given in Table 4. These are estimates of the fraction of the TA B L E 5 Output configurations and metrics in response to optimization of a single metric. variance in a model's output that can be attributed to variance in the input. The first-order effect, S 1 , is a parameter's direct effect, and the total effect, S T , includes the effects of a parameter when combined with other parameters. We divide the interpretation into "design variables," those we expect the end user to modify, and "fixed parameters,"

Optimized metric
which are normally fixed in the simulation, and were only varied for this sensitivity analysis.
The effects of the design variables Operator number, FXD number, QCP number, and FPW number tend to be the largest of all the variables, justifying our decision tool's focus on them. The total effect indices recall the results of our previous work, 5,16 showing that proper ratios of agents are essential for preventing bottlenecks in image processing and using all resources efficiently.
Among the fixed simulation parameters, the largest effects are from the FXD and QCP noise levels on the total noise score. This indicates that the simulation is dependent on the accuracy of the UAV acoustics models. While simple dB measurements are easy to acquire, more sophisticated noise propagation models may be required and are left for future work. Some interesting negative results also appear in Table 4. The FPW noise level has very little effect on the noise score, presumably because the typical value is already so low that any noise will be dominated by the other UAVs. The POI number had a surprisingly low effect on mission duration. This may be because the area of the mission field was fixed, and surveilling a fixed area is largely insensitive to the number of POIs within (e.g., the FXWs need to scan the entire field regardless). The effect of the operator's decision speed was also quite low, which is reassuring as the operator's model parameters present some of the greatest uncertainty within the simulation. Refining the operator model is a focus of future work, but at present it appears the simulation can tolerate a range of 0.5-2.0 of its nominal value.

Single-attribute optimization results
To illustrate the extremes of the design space, we present here a table of outputs from the system when a user is optimizing only a single attribute. In this case, the user sets the target attribute's weight to 100, and all other weights to 0. While most realistic decisions would require a trade-off, this type of exercise is important to learn about the extremes and verify that the tool gives reasonable results in simple cases. Table 5 shows the top recommended system configuration when optimizing for each of speed, accuracy, and stealth.
The results from Table 5 match with intuition. When optimizing only for speed, the maximum number of assets is deployed: 10 each of operators, FXWs, FPWs, and QCPs. The massed assets minimize any delays in processing POIs. When optimizing only for accuracy, a mixture of assets is recommended. In previous work, 5 we have shown that careful balance of the number of humans and automated systems is necessary to provide steady stimulation to the operators without causing task overload; the recommended configuration keeps them in an optimal utilization zone to make the most accurate decisions. When optimizing for stealth, the application recommends no QCPs, one FXW, and one FPW; this ensures that the airspace is sparsely populated with only the quietest UAVs.

Scenario 1: Speed priority, with accuracy secondary
In the high-speed scenario, a commander has to move through a potentially dangerous urban environment in order to deliver VIPs and supplies to an embassy therein. At his disposal are 4 operators, and up to 10 UAVs. Because the presence of his personnel is expected, he does not expect to move through undetected in any case, and thus minimizing noise is a low priority, but accuracy and speed are high priorities with the objective to move through the dangerous zone quickly and safely.
He sets the weight of speed to be the maximum, 100, with accuracy secondary at 55. Noise level is weighted 15, and the numbers of operators and UAVs are weighted 5 (although there is a hard constraint on the maximum number of UAVs).
The top 5 results, given in Table 6

Scenario 2: Stealth priority, with accuracy and cost secondary
In the high-stealth scenario, a commander needs to survey the safety of a field in an area with a heavy civilian presence. He wishes to minimize the impact of his operations on the civilians, so low noise levels and fewer overhead aircraft are a high priority. Accuracy is still important, but speed can be sacrificed. He has an upper limit of 4 operators and 10 total UAVs. Thus his weightings for number of operators, TA B L E 6 Top 5 platform combinations given primary user priority of speed and secondary objective of accuracy.

Summary and discussion
We have shown a simulation of humans interacting with multiple autonomous systems to perform surveillance. The simulation accounts for human factors in interaction with automation and allows for the deployment of up to 40 total humans and robots. We generated a large dataset of mission-level metrics using a parameter sweep of possible system configurations and simulating their results on an HPC.
Finally, we developed an application for decision-makers to interact with the dataset and perform trade-offs among cost, speed, accuracy, and stealth to select the optimal system according to their preferences.
The application behaves as intended and is presented in an intuitive format for decision-makers. It outputs a table of the top 5 system configurations according to their fitness scores, rather than recommending a single system. This allows the user to quickly consider diverse configurations before making a decision.
The case studies gave valid results with respect to their optimization goals. The single-metric optimization for speed gave a recommenda-tion to use the maximum number of operators and unmanned assets.
The single-metric optimization for stealth used only the minimum unmanned assets: one FXW and one FPW. The optimization for accuracy found a mixture of systems whose timing and throughput kept the operators in their optimal workload range. The multi-criteria optimizations, meant to capture realistic scenarios, also presented balanced systems to account for human workload, but their respective primary biases toward speed or stealth are readily apparent in the outputs. In general, the fast systems favored FXWs and QCPs over FPWs, and vice versa for the stealthy systems. The case studies presented here are just a small subset of the possible criteria weights, and the tool is adaptable to any decision-maker's preferences.
As a whole, the modeling, simulation, and trade-off frameworks developed here will aid designers of multioperator, multirobot systems.
By using the advanced computational tools, designers can make sense of the inherent complexities of such systems and take advantage of their reconfigurability.

Limitations and future work
In future work, more complex threat models should be incorporated More complex trade-off methods could be used. Many formal tradeoff methods exist beyond simple additive and multiplicative utility functions. 13 Also, more finely tuned scoring functions could be made available to the user, rather than using a linear 0-1 scale that is automatically applied by the software (which may unfairly penalize otherwise satisfactory performance that is low compared to other systems).
The scoring method for noise impact should be refined so that a system is penalized proportionally to its probability of detection. This may be a non-linear function of the respective noises made by the drones, and it could heavily rely on just a single loud noise signature over a sensor or person, whereas the current scoring function heavily penalizes long-duration but low-intensity noises.
Additional scenarios and robot models will also be incorporated into the simulation. New scenarios could include perimeter defense, persistent surveillance, and convoy protection. Robot models could include tethered drones, vertical takeoff fixed-wing aircraft, and ground vehicles. Currently, changing the individual models, mission, or static simulation parameters (e.g., number of POIs) would require completely redoing the process of Figure 1, including the lengthy HPC database generation, which limits the approach's scalability. Future work will consider how to use efficient sampling methods and user-interactive optimization to enable real-time decision support of a wider range of systems.