Estimating pairwise overlap in umbrella reviews: Considerations for using the corrected covered area (CCA) index methodology

Umbrella reviews (reviews of systematic reviews) are increasingly used to synthesize findings from systematic reviews. One important challenge when pooling data from several systematic reviews is publication overlap, that is, the same primary publications being included in multiple reviews. Pieper et al. have proposed using the corrected covered area (CCA) index to quantify the degree of overlap between systematic reviews to be pooled in an umbrella review. Recently, this methodology has been integrated in Excel‐ or R‐based tools for easier use. In this short letter, we highlight an important consideration for using the CCA methodology for pairwise overlap assessment, especially when reviews include varying numbers of primary publications, and we urge researchers to fine‐tune this method and exercise caution when review exclusion decisions are based on its output.


Highlights
What is already known • Umbrella reviews are becoming more common and there is a need to further develop methodological resources. • Overlapping systematic reviews (reviews including the same primary publications) represent a challenge when conducting umbrella reviews. • The corrected covered area (CCA) index methodology measures this overlap and facilitates inclusion decisions.
What is new • In this short letter, we highlight a specific consideration for using the CCA index methodology when reviews include varying numbers of primary publications.
Lucas Morin and Amaia Calder on-Larrañaga share the last authorship.
• We urge researchers to fine-tune this method and exercise caution when inclusion decisions concern reviews with varying number of included studies.

Potential impact for RSM readers
• Being aware of the described potential limitation will help researchers working on umbrella reviews to ensure quality and rigor of their work.

| INTRODUCTION
Reviews of systematic reviews are known as "umbrella reviews". They are increasingly used for mapping out the available evidence on a given topic. An umbrella review systematically identifies relevant systematic reviews (with or without meta-analysis) and synthesizes findings to summarize the current knowledge base, to describe discrepancies across reviews, and to form a judgment about the strengths and weaknesses of the research field. One important challenge when pooling data from several systematic reviews is to avoid double-counting, that is,Ágiving too much weight to primary publications included in several reviews. 1-3 Determining overlaps in publications included in several reviews is challenging when the umbrella review includes a considerable number of reviews. Therefore, to quantify the degree of publication overlap in umbrella reviews, Pieper et al. 4 have proposed using the corrected covered area (CCA) index. In this methodology, researchers start by building a citation matrix in which primary publications are listed in rows and the different systematic reviews included in the umbrella review are represented in columns. It is then relatively easy to calculate how many times a given study is cited in systematic reviews. The mathematical formula at the basis of the CCA index is as follows: where N is the total number of times primary publications appeared in reviews (inclusive of double-counting), r is the number of unique primary publications, and c is the number of systematic reviews included in the umbrella review. 4 The formula is accompanied by a classification of the degree of overlap, whereby 0%-5% is considered as "slight overlap", 6%-10% is considered as "moderate overlap", 11%-15% is considered as "high overlap", and >15% is considered as "very high overlap". When the degree of overlap is substantial, the selection of reviews should be critically assessed by authors.
The CCA methodology has since been adapted in several automated tools for overlap analysis. In 2022, two such examples were published: Pérez-Bracchiglione et al. 5 developed an Excel-based tool named GROOVE, and Bougioukas et al. 6,7 released an R-package (ccaR) for overlap calculations using the CCA methodology. There are several publications and discussions available regarding different aspects and potential limitations of the CCA index. 8-10 However, we deem it necessary to highlight one more issue that might be relevant for researchers using the methodology to guide inclusion decisions in their reviews.

| DESCRIPTION OF THE PROBLEM BASED ON HYPOTHETICAL EXAMPLES
We believe that the current CCA formula should be used with caution when assessing the overlap between a pair of reviews wherein one review includes a substantially larger number of primary publications than the other.
To illustrate this limitation, one can think of a hypothetical (and voluntarily simplistic) umbrella review of two systematic reviews: the first one includes 19 primary publications, and the second one only 3. The citation matrix (Table 1) shows that 2 articles included in the first review are also listed in the second review. According to the CCA formula, the degree of overlap is moderate (10%), and does not "flag" the smaller review to be potentially excluded for being redundant. Yet, the second review only adds evidence from a single publication that wasn't already included in the first review. Although the level of overlap indicated by the CCA index seems trivial at first glance, inclusion of the second review would lead the authors to double-count 2 out of 3 primary publications and thus to potentially give too much weight to the evidence presented in these 2 articles. This can be contrasted with another hypothetical example, wherein two reviews are similar in size and contain 20 and 24 primary publications. If these two reviews have 4 publications in common, the degree of overlap will be 10%, which indicates a relatively low overlap and could therefore be considered negligible by reviewers. In this latter case, the CCA formula performs as intended.

| IMPLICATIONS AND DISCUSSION
We believe that the above-described scenario demonstrates how the CCA index can underestimate the degree of overlap between pairs of reviews that contain dramatically different numbers of studies. This could lead to incomplete assessment of overlap, followed by flawed inclusion decisions.
Following the rise in umbrella reviews as an evidence synthesis tool, together with the recently published Preferred Reporting Items for Overviews of Reviews (PRIOR) guideline 11 highlighting the importance of assessing and reporting of publication overlap in umbrella reviews, we urge researchers to further finetune the CCA methodology. In the meantime, the aforementioned consideration will need to be accounted for when performing pairwise review overlap assessment based on the CCA formula. While the CCA allows identification of reviews with a large overlap in primary publications, it should not be relied upon blindly, especially in the presence of reviews of widely varying sizes. It is important to acknowledge that reviews may vary in size for a number of reasons, such as differences in inclusion criteria, searched databases, and limitations on date, language, or type of publications. It is necessary to carefully analyze these characteristics to maintain consistency in evidence synthesis within umbrella reviews. In addition, we recommend researchers to carefully consider, independently of the CCA, the inclusion of smaller reviews with regard to how much new evidence they add to the pool of primary publications within larger reviews.

CONFLICT OF INTEREST STATEMENT
All authors of the manuscript declare that they have no competing interests to disclose.

DATA AVAILABILITY STATEMENT
Data sharing is not applicable to this article as no new data were created or analyzed in this study.