On the use of computer‐assistance to facilitate systematic mapping

Abstract The volume of published academic research is growing rapidly and this new era of “big literature” poses new challenges to evidence synthesis, pushing traditional, manual methods of evidence synthesis to their limits. New technology developments, including machine learning, are likely to provide solutions to the problem of information overload and allow scaling of systematic maps to large and even vast literatures. In this paper, we outline how systematic maps lend themselves well to automation and computer‐assistance. We believe that it is a major priority to consolidate efforts to develop and validate efficient, rigorous and robust applications of these novel technologies, ensuring the challenges of big literature do not prevent the future production of systematic maps.


| INTRODUCTION: THE NEED FOR TECHNOLOGY IN EVIDENCE SYNTHESIS
The volume of published academic research is growing rapidly, and in some topics exponentially (Bornmann & Mutz, 2015;Khabsa & Giles, 2014). This new era of "big literature" Minx et al., 2017;Nunez-Mir et al., 2016) poses new challenges to evidence synthesis, pushing traditional, manual methods of evidence synthesis (systematic reviews and systematic maps ;Gough et al., 2017) to their limits. At the same time, evidence mapping approaches (systematic maps (James et al., 2016), evidence and gap maps (Snilstveit et al., 2013), evidence maps (Saran & White, 2018); see Box 1) are becoming more popular than full syntheses in some disciplines, such as in environmental science (Figure 1). These approaches that aim to catalogue broad evidence bases typically deal with greater volumes of evidence than traditional systematic reviews and are thus even more sensitive to increasing publication rates. This increase in breadth and procedural reduction means that mapping is inherently more scalable, however. To illustrate, the average number of search results in recent environmental systematic maps is more than 34,000 (Haddaway & Westgate, 2019). A typical systematic map also requires in excess of 200 person-days to complete-a significant time and resource investment (Haddaway & Westgate, 2019).
We focus here on systematic mapping for two reasons (see Box 1 for a definition). First, mapping approaches are ubiquitous across evidence synthesis methodology, despite not formally being recognised: many systematic reviews also involve an initial mapping stage that identifies, collates and describes the nature of the evidence base. Second, systematic mapping methods focus on the location and description of corpora of literature: methods that lend themselves well to computer assistance and automation. The latter Box 1 What is systematic mapping?
Overview: Systematic mapping is an evidence synthesis method similar to systematic reviewing that aims to summarise an evidence base using robust and rigorous processes. However, systematic mapping differs from systematic reviewing in that it does not aim to summarise the findings of the studies in the evidence base, but rather summarise the evidence base as a whole. While systematic reviews typically focus on answering questions like "what works, when and for whom", systematic maps aim to answer "what do we know about a topic", or "what research exists on a particular area". Systematic maps are often conducted as a first step in the evidence synthesis pathway, and would be followed by a number of focused systematic reviews that could be conducted relatively swiftly following the mapping work already undertaken (searching for, screening and describing studies).
Methods: Systematic maps follow the same procedures as systematic reviews, but differ at the stage of extraction of information until the synthesis of studies. Systematic maps typically extract only meta-data (descriptive information about the studies and their systems) and apply predefined codes: they do not extract findings from the included studies. Systematic maps may include critical appraisal of included study validity, but this step is optional and would be conducted in full in subsequent systematic reviews. Synthesis in systematic maps involves a description of the evidence base and development of an interactive database of studies, and therefore does not include the quantitative or qualitative synthesis of study findings.

Outputs:
The key outputs of a systematic map are: (a) an interactive, searchable database of relevant studies and their descriptive information; (b) a list of knowledge gaps (topics where primary research is limited or absent) and knowledge clusters (topics where sufficient research exists to warrant full synthesis via systematic reviews); (c) visualisations of the evidence base to facilitate understanding (e.g., evidence atlases, heat maps, descriptive plots); (d) a summary report describing the background, methods and results.
Terminology: The term "systematic map" is widely used in environmental science and social science. Some fields refer to this method as "evidence maps", although this is perhaps a broader definition, procedurally. In the field of social welfare, the Campbell Collaboration has developed the term "evidence and gap map" to describe a form of evidence map that focuses on identifying systematic reviews (and often impact evaluations) on a topic, and developing a matrix of interventions and outcomes to visually represent the evidence base. In the field of health, an analogous approach would be described as a "scoping review", but this term includes a broader set of methods that include reviews conducted to a lower standard than systematic maps (Colquhoun et al., 2014). Furthermore, the term "scoping review" is used to describe a preliminary, exploratory step prior to the conduct of a full evidence synthesis in many fields (CEE, 2013). appraisal and synthesis) require in-depth analysis of individual manuscripts that is so far proving highly challenging for automation.
The approaches described herein could thus be relevant to a range of different evidence synthesis and mapping methods.
Despite this growing problem, there are signs that new technology developments, including machine learning, are likely to provide solutions to the problem of information overload (Zarocostas, 2020) and allow scaling of evidence syntheses to large and even vast literatures Lamb et al., 2019;Westgate et al., 2018). First, such technologies may enable reviewers to deal with vast evidence bases far more efficiently and swiftly than humans alone, by bearing the burden of repetitive and routine tasks and allowing manual efforts to be diverted towards more complex tasks. Second, they can identify patterns and content across large sets of documents that would otherwise be inaccessible or incomprehensible to humans, breaking new ground in evidence synthesis and knowledge aggregation. Thirdly, they offer new opportunities for identifying additional relevant literature within given resource constraints, while also potentially enhancing the consistency and accuracy of screening, study selection and coding. Finally, they may help to accelerate the process of conducting systematic maps. Computer-assisted reviews thus offer the promise of increased efficiency, scientific insight, comprehensiveness, rigour and speed.
In this paper, we outline how systematic mapping methods lend themselves well to automation and computer-assistance, and highlight evidence synthesis technology as a vital means of dealing with the rapid expansion of evidence across disciplines (so-called infodemics (Zarocostas, 2020). We argue that a major priority is to consolidate efforts to develop and validate efficient, rigorous and robust applications of these novel technologies. In doing so, we will ensure the challenges of big literature do not prevent the future production of systematic maps.

| THE BENEFITS OF SYSTEMATIC MAPPING
Systematic mapping aims to catalogue and describe the evidence base, unlike systematic reviews, which aim to extract and synthesise study findings. Systematic maps are typically used to provide an Haddaway, N. R., Bernes, C., Jonsson, B. et al. The benefits of systematic mapping to evidence-based environmental management. overview of a broader landscape of research, and as such are useful as a first step on the "evidence synthesis pathway": the process of moving from broad stakeholder concerns ("what do we know about…") to a series of focused answers to targeted questions about impacts and "what works" (Gough et al., 2017). Often, the scope of systematic maps more closely aligns to the broad concerns fielded by commissioners and other stakeholders: certain aspects of the key elements of systematic maps (populations, interventions/exposures, comparators, outcomes, methods/study types) are typically included iteratively and are not restricted by prespecified inclusion criteria.
Because of this and by design, systematic mapping can be used to catalogue multiple types of evidence in the same map: modelling studies, laboratory studies, field studies, quantitative research, qualitative research, commentaries, theories and methodologies.
Although they do not synthesise study findings, systematic maps are an important means of highlighting knowledge gaps (subtopics where primary research is lacking), knowledge clusters (subtopics where sufficient studies exist to allow full synthesis via systematic review) and best-practices and/or deficiencies in research methods.
They can thus help to direct impactful and needs-driven research and funding towards primary and secondary gaps in the evidence.
Systematic maps can be upgraded into full systematic reviews relatively quickly for given knowledge clusters and, in these cases, can be particularly resource efficient. They can thus be used to determine for which subtopics systematic reviews are most appropriate and feasible, reducing the risks of undesirable empty reviews.
Representativeness is arguably more important than comprehensiveness in systematic maps, since the aim is to identify patterns in gaps and clusters rather than generate precise estimates: as an analogy, a photograph of the Mona Lisa is recognisable at low resolution, but it could not be considered an accurate representation of the underlying information in the painting (Figure 2). This reduces the risks of non-exhaustive searching in systematic maps: in systematic reviews, there can be severe consequences if even one relevant study is missed, but this is not the case for maps, which summarise broad patterns. Furthermore, there are minimal risks of including a small number of irrelevant studies in a map, since it is broad patterns that are of interest, and further screening and critical appraisal is necessary before proceeding to full synthesis in any subsequent systematic review. This relaxation of the exacting standards traditionally required of systematic reviews makes computer-assisted (and perhaps even automated) maps more viable.

| HOW EVIDENCE SYNTHESIS TECHNOLOGY CAN SUPPORT SYSTEMATIC MAPPING
There are many key points in the traditional view of systematic mapping processes where evidence synthesis technology and computational methods could be applied, broadly speaking, project management, retrieval, screening, data extraction and narrative synthesis. We discuss these processes in the following sections.

| The benefits of computer-assisted mapping to research, policy and practice
The computer-assisted (or automated) systematic mapping that we describe herein has a suite of benefits to researchers, policymakers and practitioners. More rapid mapping of the evidence will result in faster synthesis in general (including the transition from systematic maps to systematic reviews), by facilitating the identification and analysis of the nature of an evidence base. In turn, this will allow evidence synthesis to be more responsive to the knowledge needs of decision-makers. Where decision-makers require urgent evidence on a subject, computer-assisted mapping can make the provision of a timely and systematic map, where previously evidence synthesis would have had to be either timely or systematic. In addition, because computer-assisted systematic maps can be continually updated to include relevant research as and when it is made available, maps can be kept up to date with the latest evidence.

| Current technologies
We highlight below some key examples of evidence synthesis technologies (defined here as novel tools and frameworks used to support evidence syntheses) that currently exist and can already be used to increase the efficiency, rigour and accessibility of evidence synthesis.
We explain these across the main processes of systematic mapping.
1. Project management-In its broadest form, review management tools are evidence synthesis technologies that suit themselves well to systematic maps, since they support an entire flow of information through the review process, can deal with big data review projects, and facilitate good record keeping, which is vital with such large evidence bases.
F I G U R E 2 The Mona Lisa and a pixelated version, still easily recognisable. Image Source: https://www.newmediacampaigns.com/ page/how-to-optimize-an-image-for-your-website 2. Retrieval-Text analysis processes can suggest search terms to support search string development based on computer-assisted scoping of the literature (Grames et al., 2019;Stansfield et al., 2017). Text analysis processes can be used to retrospectively improve on Boolean search strings to improve search strategies for updates to bibliographic searching. Similarly, text analysis and citation network processes (e.g., Weller et al., 2018) can be used to suggest additional relevant articles that have been published since searches were conducted to update systematic review searches, ensuring the longevity of the map, which would otherwise become out-of-date as soon as searches are concluded.

| How technology could redefine evidence synthesis methods
Evidence synthesis methods at present separate the processes of searching and screening to maximise first sensitivity and then specificity, somewhat independently: they gather a large body of potentially relevant evidence by searching for specific terms that must be mentioned, and then narrow this corpus down by manually assessing and extracting explicit and implicit meanings (Figure 3a). This approach is necessary, but time consuming and inefficient. It is necessary because, at present, sensitive searches are the only way to ensure all available records are retrieved. It is inefficient because a large volume of information must be read and screened for possible relevant, with the vast majority being discarded (only 8% [n = 11] of relevant, deduplicated search results were included after title and abstract screening, and only 3% [n = 17] after full-text screening in a recent assessment of systematic reviews and maps (Haddaway & Westgate, 2019).
Evidence synthesis technology could allow us to blend the processes of searching, screening and coding into a single, organic research discovery step by developing algorithms to predict document relevance based on trialled and validated human assessments of relevance, blended with other methods like citation/collaboration network analyses (Figure 3b).
Developing a blended research discovery process in this way, could help to develop curated bodies of evidence where the information generated through the act of reviewing is shared across researchers and project, instead of wasting effort across reviews and having to repeat the work of screening vast numbers of potentially relevant records. The use of automation in cataloguing this evidence makes this more feasible by drastically reducing the efforts needed to screen and classify studies. This iterative process of blended searching and screening, could drastically increase comprehensiveness and efficiency, and reduce research waste.
In healthcare, technology is being harnessed to automate several steps in the evidence synthesis process, collating, summarising and appraising all randomised control trials (TrialStreamer; https:// trialstreamer.robotreviewer.net/).

| Facilitators to automated systematic mapping
We describe here what we believe to be some of the key areas that require development to support further computer-assistance and full automation of systematic mapping. 4. Data extraction/coding-Automated extraction of meta-data from full texts is a key challenge but would speed up a timeconsuming process and should focus on all aspects of the map's key elements, including study location, population type, intervention type and strength, comparator type, measured outcome, context, study design and methods. Some of this information is easily identified through text analysis (e.g., using dictionary methods; Welbers et al., 2017), but some may be more challenging to identify and extract, for example, the context, study design and methods.
5. Scientometric mapping-The integration of scholarly networks (who works with whom, who cites whom) and systematic mapping to facilitate computer-assisted "research weaving" (analysis of the who as well as the what; Nakagawa et al., 2018) will allow for more efficient and comprehensive discovery of relevant research.
6. Off-the-shelf classifiers-The establishment of public machine learning libraries (classifiers) of broad concepts (e.g., "developing nations") for use in screening and coding that can be validated will F I G U R E 3 Current and future approaches to document discovery, eligibility screening and cataloguing. (a) The status quo of systematic mapping. (b) A potential future of computer-assisted systematic mapping facilitate rapid and comprehensive discovery and analysis of relevant evidence (e.g., Marshall et al., 2018).

| BARRIERS AND FACILITATORS TO COMPUTER-ASSISTED SYSTEMATIC MAPPING
Such transformations to the way that we conduct evidence syntheses are not without their challenges. We highlight some of the major barriers below and suggest factors that may facilitate the transition to more computer-assisted systematic mapping.

| Technical issues
1. Lack of interoperability-At present, tools are rarely interoperable, although there is some standardisation of citation formats.
Automated systematic mapping requires that existing and future technologies are interoperable (i.e., use compatible file formats and data standards) to facilitate multiteam tool development and collaboration (and avoid redundancy and development waste).
2. Paywalls-Many effective tools with long-term support must charge subscription or access fees to ensure their longevity.
These paywalls may be restrictive for reviewers working in resource-constrained contexts. Waivers for low-and middleincome country users and institutional subscriptions can help to support wider use, however.

3.
Lack of open discovery-Automation of systematic mapping requires comprehensive access to large bodies of abstracts and full texts. However, this big data is typically held by publishers and in bibliographic databases that are hidden behind paywalls, or have built-in use restrictions that limit the volume and type of data that can be accessed. Some novel platforms and databases, such as Microsoft Academic (https://academic.microsoft.com/home), Crossref (https://www.crossref.org/), and Lens.org (https://www. lens.org/), do allow large-scale access to bibliographic information (so-called Open Discovery). Open Access (freely accessible full texts) is increasing but is not yet sufficient for use in automated mapping. Even when researchers do have access to papers, obtaining full texts at scale and in a machine-readable format is rarely possible.
4. Lack of standardised validation-Many tools require robust and independent validation before they are regarded as being safe to use. However, validation has not yet been standardised and is typically self-assessed by those producing evidence synthesis technologies. In order to validate tools, there is a need for "gold standard" datasets from example reviews, which have been scarce to date (O'Connor et al., 2019).

Challenge in extracting implied meaning-Coding and extraction
of meta-data from full texts require a detailed understanding of nuance (i.e., interpretation of implied meaning rather than explicit terminology) that is not yet fully feasible with automation technologies because of the wealth of information in research articles and the nuance and implicit meanings hidden within narrative style documents. For example, the presence of a phrase in an article's methods text does not necessarily mean that method was used; "We chose to use a blocked design because of the challenges associated with randomised control trials".

| Cultural barriers
1. Coding skill barrier-Currently, tool use sometimes requires knowledge of coding or programming ability, for example, many of the packages in the R environment (e.g., litsearchr, Grames et al., 2019). Graphical user interfaces (GUIs) allow users to interact with tools without requiring programming skills, but typically require substantial resources to create. Tools like Shiny (https://www.shinyapps.io/) allow standardised GUIs to be produced for R packages with relative ease, reducing necessary resource investment while increasing usability. what procedures should be in place for verifying that they are still functioning appropriately? These questions must be answered before broad-scale automated systematic mapping can be rolled out.

| CONCLUSIONS
A wide variety of tools already exist to support computer-driven systematic mapping. These tools sit on a spectrum from those intended to facilitate human-driven mapping, to those intended to replace humans entirely. However, current technology is not yet sufficient to automate all of the processes involved in rigorous evidence synthesis. Furthermore, rigorous validation is necessary before many of these tools can be trusted for wide-scale use by the broad research community.
We have highlighted some of the main opportunities that already exist, where the key intervention points are in systematic mapping, and what developments in technologies and frameworks are needed to facilitate the increased use of evidence synthesis technology in systematic mapping. We believe that systematic mapping lends itself well to at least partial automation and call for the methodology and tool development community to collaborate to reduce redundancy and increase the efficiency of evidence synthesis technology development and integration.