IDEAcology: An interface to streamline and facilitate efficient, rigorous expert elicitation in ecology

1. Here, we demonstrate how IDEAcology aids in preparing for and implementing a structured expert elicitation using the IDEA protocol, an iterative quantitative expert elicitation framework. 2. Expert judgement is used to inform decision-making on environmental assessment

This interface is designed to facilitate managing an IDEA elicitation, the process prior to statistical analysis.
The IDEA protocol (Investigate, Discuss, Estimate and Aggregate) has been described in length elsewhere Hanea et al., 2017), but we summarise it briefly here to aid in understanding the impetus and key advantages of IDEAcology. IDEA is a structured elicitation protocol modified from the well-established Delphi procedure  and was designed to derive judgements of quantitative and probabilistic estimates. Briefly, a group of experts with knowledge of previously defined questions are convened (Figure 1a). The IDEA protocol first asks experts to individually INVESTIGATE the problem before providing an initial private estimate and accompanying rationale ( Figure 1b). Next, the experts convene to DISCUSS the anonymised judgements (Figure 1b). Experts are then asked to provide a final revised ESTIMATE, in which they revisit and potentially update their initial judgements ( Figure 1b). Finally, expert judgements are AGGREGATED into a quantitative dataset (Figure 1c). The protocol has mostly been applied in ecological and biosecurity applications; however, it can be used outside of these domains in areas such as Defence procurement (Hemming, Armstrong, et al., 2020), metascience (Wintle et al., 2021), peer-review (Marcoci et al., 2022) and food security (Barons & Aspinall, 2020).

| Key challenges
While IDEA has shown promise in improving judgements of experts, several key challenges in implementing the IDEA protocol may arise when trying to design a survey to capture the expert's initial estimates and rationales. In this section, we elaborate on common hurdles.

| Expert users
Typically, experts are required to provide an upper, lower and best estimate (which form the basis of a credible interval, or a probability distribution). Experts are often not familiar with providing their judgements in these formats Low Choy et al., 2009;Revie et al., 2010) and, particularly when elicitation is undertaken remotely without the assistance of a facilitator. This challenge can be overcome by using more interactive methods to help experts visualise and communicate their judgements via graphical feedback on their estimates (Grigore et al., 2017;Wintle et al., 2013). Similarly, interactive methods can be used to check expert judgements prior to submission (e.g. ensuring best estimates fall within upper and lower limits, or estimates fall within a plausible range). Unfortunately, few programs are available to help collect and collate the judgements of multiple individuals for these formats (an exception being MATCH; Morris et al., 2014). Elicitation managers often, therefore, elicit judgements in a basic spreadsheet format, such as Excel, using standard questions that provide no immediate feedback to experts, or a survey platform that contains some interactive feedback such as slider bars, but typically does not provide the capability to show the experts full estimates in one graphic (e.g. a range and best estimate). A secondary challenge arises with how experts' initial and final estimates are elicited. Online interfaces may require a paid licence (e.g. Qualtrics software, Qualtrics, 2022), facilitating transparent discussion, improving the accuracy of estimates, enabling fast and efficient reporting by providing analysis-ready data outputs and lastly, flexibility in the types of elicitation questions that can be accommodated in the interface.

K E Y W O R D S
expert judgement, IDEA protocol, interface, platform, quantitative estimates, structured expert elicitation which can limit who can access these platforms, while free platforms may lack security required by companies, governments, or universities for the storage of personal data. While many online interfaces now provide quick access links for experts, they may not have the capability to protect expert anonymity. Furthermore, those that do provide anonymity may provide a new anonymous unique identifier to an expert for each stage of the elicitation, which can lead to a mismatch in expert initial and final responses if the administrator or elicitation manager has not developed strategies to circumvent this.

| Administrator users and workshop facilitation challenges
While detailed and descriptive instructions for implementing the IDEA protocol and steps exist , the elicitation requires attention to detail to help guide experts through the process and avoid misleading or influencing the collection of their judgements. The design and implementation of elicitation can take a considerable amount of time and remains a barrier for many seeking to improve the judgements derived from experts.
Before entering the important DISCUSSION phase, which serves as feedback for experts on their Round 1 judgements relative to the group, the administrator or elicitation manager must collate the individual estimates of experts, check for data anomalies, standardise estimates (e.g. when the four-step question format is used), plot the data, aggregate estimates (if desired) and tabulate rationales provided by the experts for each question into a feedback report. Even when online interfaces are used, these programs do not perform these functions. Instead, data are typically required to be downloaded from the platform and manually processed, which can be time consuming and prone to errors. An R F I G U R E 1 Conceptual overview and summary of the key phases for implementing the IDEA protocol (three-or four-step elicitation; briefly detailed in the grey text box) and how functionality within the IDEAcology interface assists in the process (highlighted in the dashed text box below the IDEA protocol information). Three distinct stages in the IDEAcology procedure: (a) Pre-elicitation handles the administration and planning tools in the expert elicitation; (b) the Elicitation stage handles the key steps of the IDEA protocol; (c) the Post-elicitation stage handles the Aggregation of expert estimates and prepares data for analysis.

| Post-elicitation
The final step of the IDEA protocol involves AGGREGATING the judgements after the experts submit their second private estimate.
It is important to note that the IDEA protocol specifies that there is no pressure for experts to reach consensus, there is no 'behavioural aggregation'. Rather, during the discussion phase, the facilitator encourages open conversation and exploration of the rationale behind different estimates so the experts can reach their own conclusions (Clemen & Winkler, 1999;Hanea et al., 2017;Hanea et al., 2018).
This creates a challenge in aggregating the variable estimates of experts. There are multiple ways that expert judgement can be aggregated, for example, quantile averaging or expert weighting (refer to Cooke, 1991;McAndrew et al.,  still is a requirement to download and store data from the elicitation in a format that can be readily used by other software packages.

| THE IDE ACOLOGY INTERFACE
The IDEAcology interface was designed as a template to plan and facilitate the implementation of the IDEA protocol by reducing logistical difficulties and providing a reliable and efficient way for experts to input, visualise and cross-examine the estimates of their peers, reducing ambiguity, while also facilitating data management.
In designing the IDEAcology interface, we sought to implement the following processes in a dedicated platform: (i) simplify the expert elicitation process, including streamlined pre-elicitation project management; (ii) meet ethics and confidentiality requirements; (iii) foster transparent discussion among experts; (iv) facilitate data management and (v) provide fast and efficient reporting back to experts and (vi) provide cost-effective quantitative assessments in both face-to-face and remote formats with comparable efficacy. All these processes aimed to improve accessibility of, and practitioner engagement with scientific information (Walsh et al., 2015). The IDEAcology interface, including training materials, can be freely ac-

| ID E ACO LO GY WO RK FLOW
Here, we outline the workflow of the IDEAcology interface and how it assists in the implementation of the IDEA protocol. The structure of the interface aligns with the four phases of the IDEA protocol: INVESTIGATION, DISCUSSION, ESTIMATION and AGGREGATION.

| Administrator users
IDEAcology has an administrator portal that guides the project team through the development of an expert elicitation process. Where possible, the pre-elicitation process is automated, including sending email invites for experts to register on the interface, email alerts to

| Expert users
Once an expert has electronically accepted an invitation and completed the user agreement, IDEAcology guides them through registration on the interface. The interface collects and collates customisable demographic information, which can be used to keep track of the group diversity and mitigate potential bias in expert selection. This may include, for example, experience and skills of experts provided in a description or quantitative format (e.g. years of experience) or type of experience (modelling, field surveys, management, laboratory etc.), gender, age and self-rating of experience Hemming, Walshe, et al., 2018).
This information is stored securely to comply with ethical considerations (see Interface Availability for more details on data storage and compliance). In experimental settings, it may also be used to explore

| Expert users
IDEAcology guides the experts through the INVESTIGATION, DISCUSSION and ESTIMATION phases of the IDEA protocol. At the beginning of the elicitation, experts can be provided with access to the practice questions and general knowledge resources to provide shared baseline information (e.g. species-specific information or distribution data).

INVESTIGATION:
The interface guides experts through the elicitation to provide their initial estimates in a prescribed order (upper and lower limits first, then best estimate), which (1) reduces the chance that users will anchor on their first or best estimate, which may be influenced strongly by availability bias (Figure 2) and (2)

4.
Legend to the IDEAcology interface: Discussion phase example

Focal unit drop down list
Choose what focal unit questions to view during the elicitatation 1.

Compiled expert initial estimates
Visualization of each user's initial estimate for a scenario, with comments justifying their decisions.

Access to uploaded resources
If any of the experts provided contextual material during their initial estimate it can be found here for reference, i.e. PDFs Optional supporting text providing further explanations on the scenarios experts are asked to assess.

Multiple scenario options
Optional supporting text providing further explanations on the scenarios experts are asked to assess.

User entered comments justifying their estimated values and/or uncertainty around their estimates.
estimates and unfounded assumptions, as well as identify sources of ambiguity or new information to help mitigate subjective biases in the expert elicitation process. Each question has an optional timer window (minutes in real time during the DISCUSSION phase, possible either online or face-to-face) that can be set and paused by the facilitator to keep discussions focused.

ESTIMATION:
With the IDEAcology interface, experts can privately review and reassess their initial estimates either during or following the discussion. While revision of the initial estimates is required, estimates do not have to be changed (i.e. consensus is not required). A growing body of evidence suggests that experts tend to anchor on their initial judgements, with those updating their judgements following insights from the discussion typically moving towards the realised truth Hemming, Armstrong, et al., 2020;Hemming, Walshe, et al., 2018;Wintle et al., 2021). To prompt the expert to actively revise estimates, the elicitation form displays their initial estimates and rationale and requires the estimates to be actively re-entered (following the same format as in the INVESTIGATION phase). The interface also requires the expert to enter additional reasoning to justify changes from their initial estimates, or lack thereof, based on insights shared during discussions within the DISCUSSION phase. Once experts have completed and submitted their revisions, the values are locked and are unable to be changed (Figure 2).

| Post-elicitation in IDEAcology
AGGREGATION: Aggregation and analysis of expert elicitation requires a contextualised approach and project teams may prefer to analyse the information gained from expert elicitation in a different and more nuanced way. While the IDEA protocol ends with a mathematical aggregation of the estimates, no specific type of aggregation is prescribed by IDEA. The IDEAcology platform aggregates expert estimates using the commonly used equal weighted quantile aggregation method (O'Hagan et al., 2006). This aggregation method calculates the mean lower, upper and best estimates across experts. These summarised statistics, in combination with expert individual estimates, are used during the Discussion stage.
Quantile averaging is also applied to the final round of estimates and can be exported as a separate summary statistic .csv file. Alternative aggregation methods such as linear pooling using equal weighting or performance-based weighting (when using calibration questions with answers known by the elicitation manager, but not by the experts, Cooke, 1991) are not implemented within IDEAcology.

| Analysis-ready data outputs
While IDEAcology provides strong support, there are still aspects of the elicitation process that cannot be proceduralised within the IDEAcology interface. For example, IDEAcology will only provide a simple aggregation at the conclusion of the elicitation process; however, the IDEAcology analysis-ready data output allows administrators to plug into a different tool and aggregate with the most appropriate aggregation method (Figure 1c). When calibration questions are used (and included in the pre-elicitation set up and preparation stage), IDEAcology data output includes expert estimates and calibration data compiled into the same data format, allowing the .csv data to be accessed and used within packages that calculate performance weighted aggregations.
The IDEA protocol and subsequently IDEAcology were designed to support elicitation of quantitative and probabilistic estimates; for example, estimating consequences of different decision alternatives or parameters in models. This assumes that many of the qualitative judgements required to structure the question or the model are made prior to embarking on an elicitation (Hanea, Wilkinson, et al., 2021;Mukherjee et al., 2018;Nyumba et al., 2018;. To help explain the estimates provided by experts and stimulate discussion, qualitative rationales are also collected for each question asked.
While there has been significant attention to qualitative methods for expert elicitation, the steps involved in collecting, cleaning and aggregating this quantitative data are often unique to quantitative expert elicitation and may not translate across to qualitative methods. Despite this, the platform does support the collection of qualitative rationales and evidence to help support and explain quantitative estimates and stimulate the discussion phase. This enables the project team to better understand the justification for judgements provided by experts, and if desired to use the collated set of anonymised notes of expert rationale in the raw data in a .

| IDE ACOLOGY US E IN CON S ERVATI ON AND NATUR AL RE SOURCE MANAG EMENT
We have engineered flexibility in the types of questions that can be accommodated in the interface based on feedback on beta prototypes and recognising the need to accommodate a wide range of ecological questions and management applications. By providing an interface, we aim to increase the uptake of structured expert elicitation to support decision making in conservation and management. Possible conservation and management questions F I G U R E 2 An example of an expert elicitation using the IDEAcology interface. Here experts can view a visualisation of the collated initial experts' estimates, with users anonymised by number. This visualisation is used during the DISCUSSION phase to aid discussion of similarities and differences among expert estimates. Below the visualised initial estimates, experts' view the panel to re-assess their initial estimates. Numbers in the legend to the IDEAcology interface highlight and detail the key features of the interface.

| CON CLUS IONS
Expert judgement has great potential to inform a range of conservation and natural resource management applications. Indeed, structured expert elicitations such as the IDEA structured protocol, are routine to augment and expedite empirical resources when imminent decisions are required, yet quantitative data are absent or uninformative. One of the key hurdles to implementation of the IDEA structured protocol is that there is no streamlined approach to the elicitation, analysis and feedback of estimates provided by experts, with elicitation managers often using a combination of Excel and survey platforms combined with manual data entry or R-coding to provide feedback. These manual processes typically pose a barrier for uptake of the protocol or can lead to mistakes being made. Hence our focus on developing the IDEAcology interface was to support and streamline this process.
The IDEAcology interface was designed to improve uptake and application of the IDEA protocol as a pragmatic basis to facilitate conservation and natural resource management decision making and policy .

ACK N O WLE D G E M ENTS
We acknowledge the Traditional Owners of Country, on whose traditional, ancestral and unceded land we work and live on. We recognise the continuing connection to lands, waters and communities across the landscapes and pay our respects to First Nations and Indigenous Peoples, and to Elders past, present and emerging.
Improvements to the IDEAcology interface relied on several rounds of expert elicitations and feedback. We thank all experts who par- detailed comments that markedly improved the manuscript.

CO N FLI C T O F I NTER E S T S TATEM ENT
We declare no conflict of interest.

PEER R E V I E W
The peer review history for this article is available at https://publo ns.com/publo n/10.1111/2041-210X.14017.

DATA AVA I L A B I L I T Y S TAT E M E N T
The IDEAcology interface and further information, including training materials can be found at http://www.ideac ology.com/. IDEAcology works across multiple platforms (Windows and Mac compatible) and browsers and is designed to be compatible with as many different devices as possible, including computers, laptops and handheld devices (tablets and smartphones). The interface is cloud-based, removing any infrastructure requirements and ensuring ready availability, aside from the need for internet connectivity. IDEAcology is a cloud-based solution and consists of a web-based frontend with a database backend (often referred to as a two-tier architecture).
All data within the system are stored in the backend database. The database itself is highly locked down following industry standard methodologies to ensure security and confidentiality of the data.
The database can only be accessed directly by Lighting Rock (the company developing the interface) technical personnel. The system is compliant with the General Data Protection Regulation (GDPR); however, it should be noted that the GDPR is mostly a framework of rules that apply to how organisations manage data and the technical implementation is only a small part of this. Users have the option to delete their personal data from the system in the profile section; however, they are instructed to contact the elicitation manager or a site administrator if they want their data removed from specific elicitations (personal data are not stored in elicitations).
All data collected can be securely stored in a cloud-based database to ensure data integrity and reliability. Importantly, the project team can specify the hosting arrangements depending on their security needs, for example, an internal provider or in-country cloud provider. The system enables trained users to create and