Advancing the methodology of mapping reviews: A scoping review

This scoping review aims to identify and systematically review published mapping reviews to assess their commonality and heterogeneity and determine whether additional efforts should be made to standardise methodology and reporting. The following databases were searched; Ovid MEDLINE, Embase, CINAHL, PsycINFO, Campbell collaboration database, Social Science Abstracts, Library and Information Science Abstracts (LISA). Following a pilot‐test on a random sample of 20 citations included within title and abstracts, two team members independently completed all screening. Ten articles were piloted at full‐text screening, and then each citation was reviewed independently by two team members. Discrepancies at both stages were resolved through discussion. Following a pilot‐test on a random sample of five relevant full‐text articles, one team member abstracted all the relevant data. Uncertainties in the data abstraction were resolved by another team member. A total of 335 articles were eligible for this scoping review and subsequently included. There was an increasing growth in the number of published mapping reviews over the years from 5 in 2010 to 73 in 2021. Moreover, there was a significant variability in reporting the included mapping reviews including their research question, priori protocol, methodology, data synthesis and reporting. This work has further highlighted the gaps in evidence synthesis methodologies. Further guidance developed by evidence synthesis organisations, such as JBI and Campbell, has the potential to clarify challenges experienced by researchers, given the magnitude of mapping reviews published every year.


Highlights
What is already known • A mapping review aims to identify what is known about a topic, what research exists on a particular research question, or where this evidence comes from.• The growth of these types of approaches reflects the needs of decision makers to know about the breadth of a topic, and the increasing number of reviews of this type being published is an indication of their value in the evidence ecosystem.
What is new • To our knowledge, this is the first scoping review on comparing mapping review methodologies and limitations.• The results of this scoping review highlighted the misclassification of evidence synthesis methodologies.• This work also highlighted the need for the development of mapping reviews guidance to clarify the scope, objective and the methodology for researchers in order to support them to generate trustworthy evidence.
Potential impact for Research Synthesis Methods readers • Further work in this area has the potential to guide researchers, clinicians and policymakers on the appropriate use of the right methodology to address their research questions.

| BACKGROUND
To date, there is no established or agreed definition on mapping reviews.However, Campbell et al. 1 propose that the purpose of mapping reviews is to 'collate, describe, and catalogue the available evidence relating to the question of interest'.Booth 2 proposes that 'A mapping review aims at categorising, classifying, and characterising patterns, trends or themes in evidence production or publication'.A mapping review aims to identify what is known about a topic, what research exists on a particular research question, or where this evidence comes from. 1 Mapping reviews are related to scoping reviews and evidence gap maps but are sufficiently different. 3They are all part of the 'Big Picture' review family. 1 Mapping reviews, scoping reviews and evidence gap maps have not been used consistently in the literature, both mapping and scoping reviews have been used reciprocally describing the same type of review methodology.Moreover, the PRISMA-ScR has been used to guide the reporting of both methods.However, there are differences between these two methods especially in the way data is extracted.Scoping reviews tend to have an indepth data extraction of study findings, whereas mapping reviews seem to have a broader more superficial data extraction which includes mostly categorisation or a higher level of data classification or assortment. 1 Evidence gap maps, on the other hand, mostly refer to the tool used to visualise the data produced by mapping or scoping reviews. 3here are variants of mapping reviews, and several organisations that have developed, or adapted existing methods, for creating their own product.][6] The growth of these types of approaches reflects the need of decision-makers to know about the breadth of a topic, and the increasing number of reviews of this type being published is an indication of their value in the evidence ecosystem.While there has been much written about the methods used for scoping reviews 7,8 mapping reviews have been less well described.In this review, we sought to explore reviews that described themselves as being a mapping review.
A paper by Saran and White 9 mapped several comparable evidence mapping methods with important differences. 10However, to our knowledge, this is the first scoping review on comparing mapping review methodologies and limitations.This scoping review aims to identify and systematically review published mapping reviews.

| AIM
The purpose of review was to provide an overview of existing mapping reviews in the literature.
The five specific objectives of this scoping review were to: 1. Conduct a systematic search of the published and grey (i.e., difficult to locate and unpublished) literature to identify mapping review papers.2. Describe the characteristics and range of methodologies used in the identified mapping reviews, including research questions (e.g., effectiveness, feasibility, meaningfulness and appropriateness, association, prevalence, economic), type of evidence included (e.g., study designs included), appraisal of evidence, geographical coverage, types of outcomes captured.3. Describe the potential users of mapping reviews.4. Examine reported challenges and limitations of the mapping review approach, and. 5. Propose recommendations for advancing the approach and enhancing consistency within which they are undertaken and reported.

| METHODS
'Scoping reviews can clarify key concepts/definitions in the literature and identify key characteristics or factors related to a concept, including those related to methodological research'. 5Since we are interested in charting the available evidence on the methodological research topic of mapping reviews, the scoping review is an appropriate evidence synthesis method to conduct this research.The methodology of this scoping review is based on the JBI methodology by Peters et al., 7 Khalil et al. 11 and Munn et al., 8 4,5,11,12 A protocol was compiled using guidance for scoping reviews and registered in the Open Science Framework (https://osf.io/9xwpv/).The scoping review was conducted by members of the JBI Scoping Review Methodology Group and the Campbell Collaboration 'Big Family' Taskforce. 13This review is reported using the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA-ScR) extension to scoping reviews. 14

| INCLUSION CRITERIA
The inclusion criteria were developed using the PCC-Population, Concept, and Context.The Population was omitted as we intended to capture any health and nonhealth reviews.The Concept was any mapping review published using any methodology and referred by the authors as mapping reviews. 15The Context of the review is any discipline in which the mapping review is published.The types of reviews for this review included any mapping review with or without an evidence map published.We included all reviews mentioned by the authors as mapping reviews if they were in the title, abstract or full text.Only articles published in English were included in this review due to resource constraints.

| SEARCH STRATEGY
A three-step search strategy was utilised in this review. 16n initial limited search of Ovid MEDLINE, followed by analysis of the text words contained in the title and abstract, and of the index terms used to describe the article was collated to create the initial search strategy, which was then reviewed by the JBI/Cochrane Mapping Reviews working group.Key terms used were mapping review, mapped review, mapping overview, mapped overview, literature mapping, evidence map reviews, and evidence mapping.A full search strategy can be reviewed in Appendix 1.
A second search of the following databases then occurred: Ovid MEDLINE, Embase, CINAHL, PsycINFO, Campbell collaboration database, Social Science Abstracts, Library and Information Science Abstracts (LISA).Third, the reference lists of all identified reports and articles were searched for additional studies.The search was limited to English.Reviews published from 2007 onwards were included, to ensure capturing reviews published by the Global Evidence Mapping (GEM) initiative on topics such as trauma brain injury and spinal cord injury. 17We excluded grey literature as we were interested in peer review studies, in which the methodology was reviewed by other researchers.

| SCREENING PROCESS
Following a pilot-test on a random sample of 20 citations included within title and abstracts, two team members independently completed all screening.Ten articles were piloted at full-text screening, and then each citation was reviewed independently by two team members.Discrepancies at both stages were resolved through discussion.This stage was done in Covidence.

| Data items
Using guidance for scoping reviews, 11,12 relevant data were extracted from the included studies.The data were classified into main conceptual categories and summarised to align with the scoping review objective, aims and research question. 18The data extracted included the following: author, country, type of research question (e.g., effectiveness, diagnostic, prevalence, economic), use of an a priori protocol, review size, funding, duration of review, methodology used, types of evidence included (e.g., study designs), presence of an evidence map, objective of the mapping review, challenges and limitations, and discipline.A list of the data extracted is shown in Appendix II.

| DATA EXTRACTION PROCESS
Following a pilot-test on a random sample of five relevant full-text articles, one team member abstracted all the relevant data.

| SYNTHESIS
The results are presented descriptively through tables and figures and simple analysis, such as percentages and frequencies were calculated in Excel. 19

| RESULTS
There were 1658 citations retrieved from the search, of which 111 reviews were removed as duplicates (Figure 1). 20fter title and abstract screening, 1038 reviews were excluded leaving 438 reviews for full-text retrieval.At fulltext screening, 103 reviews were excluded.A total of 335 articles were eligible for this scoping review and subsequently included.

| Country of origin and publication rate
Overall, the study country of origin according to the corresponding author's affiliations extended across 39 countries.The top five countries were 67.9% of the total study sample (n = 228).The most common countries were: the United Kingdom (n = 75; 22.3%), followed by the United States of America (n = 67; 20.0%), Australia (n = 33; 9.8%), Canada (n = 30; 8.9%) and China (n = 23; 6.9%) as shown in Figure 2. Figure 3 also shows the significant rise in the publications of mapping reviews since 2015.

| Aims of the mapping reviews
Figure 4 presents the aims stated in the mapping review studies.Only 58 reviews (17.3%) provided a single review aim.The majority of the reviews (n = 301; 89.8%) aimed to describe the nature and characteristics of the research in a particular area or field.One-third of the reviews (n = 213; 63.6%) sought to identify and analyse knowledge gaps.Half the reviews aimed to propose future research opportunities (n = 176; 52.5%) and examine the volume of research that was available in a specific field (n = 168; 50.1%).
A minority of reviews (n = 33) stated a combination of other aims that included: to develop a typology, conceptual framework, or classification system related to a specified research topic; to map defined policies; to examine methodological quality and process related to identified research areas.An even smaller number of reviews (n = 9) described an explicit singular research aim, for example: to assess the effectiveness of neural mobilisation techniques in the management of musculoskeletal neck disorders with nerverelated symptoms; to develop a machine learning algorithm to automatically map the literature assessing the effect of medication; and to identify and describe which methods have been used to adjust for confounding bias in longitudinal observational data and identify the potential inappropriate use of baseline adjustment method.Mapping Review PublicaƟons per year opinion/policy reviews (n = 3; 0.9%) or evaluate the data from a cost/economic perspective (n = 2; 0.6%).

| Types of research questions
A minority of reviews (n = 5; 1.5%) presented research question types that were not be categorised into the standard research question types outlined above.For example, these included a review that aimed to analyse a human research ethics website.

| The type of reference provided for the mapping review methodology
Half the reviews (n = 130; 55%) did not state which mapping review methodology was used to undertake the review.Of the total reviews that mentioned methodologies used, a third (n = 72; 31%) stated the Grant and Booth, 2009 typology, followed by Miyake-Lye et al., 2016 (n = 26; 11%) and Bragge et al. 20 (n = 19; 8%).The Campbell collaboration methodology was stated in four reviews, 10 as shown below in Figure 6.

| Inclusion/exclusion criteria
The majority of reviews (n = 328; 97.9%) had included a defined inclusion and exclusion criteria in their methods of the review.

| Use of an a priori protocol
Almost three-quarters of the mapping reviews (n = 248; 74.0%) did not mention use of an a priori protocol or declare that a protocol was published or registered.Of the 87 reviews (26.0%) that did mention a protocol, over half (n = 49; 56.3%) did not state where the protocol was published or registered, 28 (32.2%) were registered with PROSPERO, and a further six (6.9%) with Open Science Framework.The remaining four reviews (4.6%) reported that the study protocol was: (i) mentioned in the original study; (ii) published in Campbell Systematic Reviews; (iii) registered with the International Platform of Registered Systematic Review and Meta-analysis Protocols website; (https://inplasy.com/)and (iv) registered with an international register (name not stated in the review).

| Framework for data extraction
For this section, a framework is defined as a pre-existing theory or a model that is used to classify the data obtained from the review. 18Half the reviews (50.7%) described a pre-existing framework used to extract and map the data (n = 170).A total of 145 studies (43.3%) did not use of a framework, with the remaining 20 reviews (6.0%) categorised as 'unable to determine' whether a framework was used, however, most of the reviews referred to a data extraction sheet.The highest number of evidence sources were experimental studies (n = 177; 52.8%), followed by observational studies (n = 152; 45.3%) .Evidence from qualitative studies (n = 119), other-multiple sources (n = 118) and systematic reviews (n = 116) were almost equally reported in the mapping reviews.Reports (n = 29) and guidelines (n = 13) were the least included evidence types.

| Types of evidence
A total of 118 mapping reviews (35.2%) included multiple evidence types with findings from sources categorised as 'other' and 50 reviews (14.9%) revealed a single 'other' evidence source.This included a range of sources, for example, case reports/case studies, conference abstracts, editorial materials, letters, books, mixed methods studies, commentaries, grey literature not described further, websites, applications, toolkits, government and policy publications, dissertations, quantitative studies, and studies that included meta-analysis.

| Number of databases
A total of 322 reviews reported the number of databases they used to conduct the mapping review (96.1%).The number of databases used ranged from a minimum of one to a maximum of 249.The average number of databases used was 5.9.Of the remaining 13 reviews (3.9%), four did not report on the number of databases used, six described the sources as number of journals or publications, and the final three mapping reviews used another description.The other descriptions were official government websites searched to identify national policy documents, the databases of 35 health technology assessments; and an approximation of 20+ databases.

| Number of studies
A total of 322 reviews stated the number of studies included in the mapping review (96.1%), ranging from a minimum of zero (n = 1) to a maximum of 119,546.The median was 85 studies.The remaining 13 reviews did not report the number of studies used to conduct the mapping review.Figure 8 details the number of studies included in the mapping reviews.For example, 56 mapping reviews included between 51 to 100 studies and 11 mapping reviews included more than 10,000 studies.

| Critical appraisal
A total of 87 mapping reviews performed a critical appraisal of the evidence and reported the findings in the study (26.0%).The majority of mapping reviews (n = 248) did not present the findings of a critical appraisal (74.0%).

| Geographical coverage of studies included in the mapping reviews
Almost three-quarters of the mapping reviews (n = 246) reported on the geographical coverage of the included reviews (73.4%).A total of 89 reviews did not state a

| Data presentation
Table 2 lists the types of visuals used to present data in the mapping reviews.
The majority of the mapping reviews (n = 322; 96.1%) presented accompanying graphics or visuals to communicate findings.Table 2 lists the breakdown of the visuals presented in the mapping reviews (n = 335).Tables (n = 302; 90.1%) and graphs (n = 159; 47.5%) were the most commonly employed visuals used by the mapping reviews to present findings.
A diverse range of visuals were noted in the mapping reviews (n = 51; 15.2%).Geographical maps were the most common type of map included in just over one-third of the reviews that presented a map graphic (n = 19).This was followed by evidence and gap maps that featured in five reviews as stated by the authors.Within the 'other' category (n = 45), a diverse range of figures were most commonly presented (n = 17), followed by forest T A B L E 1 Geographical coverage reported in the mapping review studies-Top 10.

| Use of software
A total of 80 mapping reviews stated that software was used in the conduct of the study (23.9%).A slightly higher number of mapping reviews did not used software (n = 101; 30.1%), whilst it was unable to be established whether software was employed in the remaining 154 reviews (46.0%).
Whilst the use of software was mentioned in 80 of the studies, almost half the mapping reviews did not describe the type of software used in any further detail (n = 36).Microsoft Excel ® was the most frequently employed software amongst the reviews that did describe what software was used (n = 14).Other software mentioned in the mapping reviews included: VOSviewer ® (and VOS mapping) in eight studies; EPPI-Reviewer ® (and EPPI-Mapper ® ) and Rstudio ® software in three studies respectively, Geonames ® in two studies and Microsoft PowerPoint ® in two reviews.

| Types of data synthesis
Figure 9 outlines the types of data synthesis included in the mapping review studies.The majority of mapping reviews provided a synthesis of findings (n = 297, 88.7%) whilst 38 reviews did not include a synthesis of the data in the review (11.3%).The most frequent type of data synthesis employed by 264 mapping reviews was descriptive analysis (78.8%), followed by thematic analysis in 26 reviews (7.7%), and statistical analysis in 15 reviews (4.5%).The least frequent data synthesis types revealed in the mapping reviews included bibliometric, framework and meta-ethnographic analysis used in one study respectively (0.3%).

| Presence of evidence and gap maps
Approaches vary in terms of presentation of maps as identified by Saran 2020.Some authors use visualisation, like 3ie and Campbell, whereas others only present findings as descriptive representations of the results such as the Global Evidence Mapping Initiative and the Collaboration of Environmental Evidence.Evidence map or maps were present in 159 reviews (47.5%).Our reporting of evidence maps was based on the authors mentioning them in their reviews and if there was any visual representation of results in a user-friendly format, often a visual figure or graph, or a searchable database. 21Different approaches were used to present evidence maps, including geographical maps, descriptive report with/ without visualisation or in the form of a matrix.It should be noted that 52.5% (n = 176) reviews included do not provide visualisation but rather present the results of the mapping in a descriptive report.Usually, a descriptive report describes the overall amount of evidence and its characteristics, such as its geographical distribution and the design of the studies.

| Equity
The majority of the mapping reviews did not address equity (n = 258; 77.0%).Equity was addressed in 60.9% 28.3% Types of data synthesis Types of data synthesis included in the mapping review studies.
66 mapping reviews (19.7%) whilst in the remaining 11 studies, it was not able to be categorically established (3.3%).Equity was defined in our review when authors described the population by either low income or racialized populations or if they have included outcome data on a particular type of population such as disadvantaged groups, or groups from a specific region that are known to be at a disadvantage, for example, sub-Saharan Africa, and other underdeveloped countries. 22

| Stakeholder engagement
Stakeholder consultation is an important part of the map, and its role appears at several places in the conduct of mapping reviews.This consultation includes defining scope, engagement in piloting, identifying sources for the search, such as organisational websites, and discussing and promoting the use of the map.However, 63.8% (n = 210) did not report stakeholder involvement in the conduct of the mapping review.

| Funding
Half of the reviews received funding (n = 192; 57.3%) compared with those that did not receive research funding (n = 143; 42.7%).

| Methodological challenges
This section was based on what authors of mapping reviews reported as challenges when undertaking their reviews as shown in Figure 10.A total of 108 mapping reviews reported methodological challenges (32.2%).The most common concern was centred around heterogeneous data reporting, as noted in 40 reviews (11.9%), followed by the challenges associated with reporting (e.g., lack of data synthesis, lack of in-depth analysis or lack of reporting on quality appraisal) reported in 30 mapping reviews (9.0%).Low methodological quality or methodological limitations of the retrieved reviews (n = 17; 5.1%) and the limited amount of data available (n = 12; 3.6%) were also reported.The least frequently reported difficulty declared in four reviews was that the large volume of initial data extracted was challenging to screen (1.2%).

| Study limitations
This section presents the limitations stated by the authors of the included reviews as shown in Figure 11 5/1%).A few mapping reviews declared that they were unable to perform a meta-analysis (n = 5; 1.5%) or a risk of bias assessment as stated by the authors (n = 3; 0.9%).Also included in the least frequently mentioned study limitations was the difficulties in managing large study volumes as reported in three mapping reviews (0.9%).

| DISCUSSION
This scoping review presents a systematic evaluation of the methodological characteristics of mapping reviews and indicates a vast diversity in methodological approaches.The findings of this review present a number of discussion points and implications for those developing mapping review methodology and those planning to undertake a mapping review.Currently, there is no existing tool to assess the risk of bias or updated methodological guidance of mapping reviews which could reflect the diversity seen within mapping reviews in this scoping review.Best practice evidence synthesis methods require an a priori protocol, and a rigorous and documented search strategy.This current review identified that protocols were rarely used or published, and a limitation identified from mapping review authors was a lack of fully developed search strategy.Transparent and rigorous methodological approaches in the conduct and reporting of mapping that align with current evidence synthesis practices should be championed.Implementing elements from PRISMA, PRISMA-ScR and PRISMA-Search may be beneficial to improve the conduct and reporting of mapping reviews.
Extracted data varied across the included reviews with some performing a formal statistical analysis such as meta-analysis and meta-ethnographic analysis or thematic analysis.Ideally, the coding form for mapping reviews should include basic study characteristics, that is, bibliographic details, participants, types of interventions, comparators and study design and data required for critical appraisal (if included). 1,3,10The coding form should be piloted with a small number 18,21-30 of included studies.The piloting exercise may include the researchers and stakeholders in some cases. 24,25nterestingly, although all of the reviews included in this scoping review described themselves as some variant of a mapping review, it is questionable whether they all actually 'mapped' the evidence and presented this in what could be reasonably described as a 'map'.Approximately half of the mapping reviews did not present what we would consider a map-which begs the question; what constitutes a 'map' in a mapping review, and is a visual 'map' a requirement for mapping reviews?The majority of the included studies presented data in tables, which were oftentimes supplemented with visual methods to represent the data.In the definition for an evidence map, no mention is made of providing a visual presentation of the data or other 'map' type presentation.However, if it is deemed that a visual overlay should accompany evidence maps, then it may be useful for proponents of evidence maps to agree to shared formats and to introduce an element of standardisation in their displays to enable end users to become accustomed to a common format for representing data in these reviews.
The results of this scoping review highlighted the misclassification of evidence synthesis methodologies and the need for the development of mapping reviews guidance to clarify the scope, objective and the methodology for researchers in order to support them to generate trustworthy evidence.Further work in this area has the potential to guide researchers, clinicians and policymakers on the appropriate use of the right methodology to address their research questions.
This scoping review has several limitations including the broad definition we included for mapping reviews, while some of the included reviews did not clearly follow a robust mapping review methodology but had mapping in their titles and abstracts, we felt that it was important to include them to showcase the challenges experienced by the researchers.Moreover, some of the included reviews did not include the required information that was necessary for this scoping review and were left blank.Finally, heterogeneous methods and data reported made it difficult at times to extract relevant information.
In conclusion, this work has further highlighted the gaps in evidence synthesis methodologies despite the significant advances made in this field.Further guidance developed by evidence synthesis organisations such as JBI, Campbell and Cochrane researchers has the potential to clarify challenges experienced by researchers, given the amount of research generated and the need to develop robust evidence synthesis methods to address the increasing number of articles published every year.

Figure 5
Figure5presents the research question types as stated in the mapping review studies.The highest number of mapping reviews (n = 92; 27.5%) specified a research question consistent with measuring the effectiveness of a particular intervention, strategy or programme.The second most frequent research question type (n = 76; 22.7%) focused on conducting a bibliometric analysis of the retrieved studies.The least number of reviews sought to either assess expert

Figure 7
Figure 7 outlines the types of evidence retrieved and reported in the mapping review studies.The majority of the mapping reviews included different types of research evidence (n = 271; 80.9%).The remaining 64 reviews (19.1%) included one type of research evidence.These were: systematic reviews (n = 23), experimental (RCT) Types of visuals used to present data in the mapping reviews.
F I G U R E 8 Number of studies reported in mapping reviews.
Study limitations reported in the mapping review studies.