Risk‐of‐bias VISualization (robvis): An R package and Shiny web app for visualizing risk‐of‐bias assessments

Despite a major increase in the range and number of software offerings now available to help researchers produce evidence syntheses, there is currently no generic tool for producing figures to display and explore the risk‐of‐bias assessments that routinely take place as part of systematic review. However, tools such as the R programming environment and Shiny (an R package for building interactive web apps) have made it straightforward to produce new tools to help in producing evidence syntheses. We present a new tool, robvis (Risk‐Of‐Bias VISualization), available as an R package and web app, which facilitates rapid production of publication‐quality risk‐of‐bias assessment figures. We present a timeline of the tool's development and its key functionality.


| INTRODUCTION
Synthesis of evidence from the totality of relevant research is becoming more important than ever in informing policy across an increasingly wide range of fields. 1 Risk of bias assessment-evaluation of the internal validity of studies included in a systematic review-often forms a key part of the evidence synthesis process, particularly in the health sciences. 2 A well-developed family of tools is widely used, which have in common the characteristic that they evaluate specific domains of bias rather being constructed as a checklist or a quantitative score. 2 These tools include the RoB (Risk of Bias) 2 tool for randomized trials, 3 the ROBINS-I (Risk Of Bias In Nonrandomized Studies-of Interventions) tool, 4 the QUADAS 2 (Quality and Applicability of Diagnostic Accuracy Studies) tool, 5 and the ROBIS (Risk Of Bias in Systematic Reviews) tool. 6 Within each bias domain a judgement is reached about the strength of the study in that regard: for example, the first domain in the Cochrane RoB 2 tool deals with bias arising from the randomization process. 3 Accessible graphics summarizing the results of these domain-based risk-of-bias assessments are included in reports of systematic reviews. A convenient plot in many reviews is a "traffic light" plot, which tabulates the judgement for each study in each domain. For larger numbers of studies, when such a table becomes unmanageable, a popular alternative is a (possibly weighted) bar plot, which shows the proportion of information with each judgement for each domain. 7 Researchers can face a number of barriers in creating these plots. While some evidence synthesis platforms, such as Cochrane's Review Manager, 8 can produce these visualizations, not all researchers use these systems to conduct their systematic reviews, and copying the risk-ofbias data into these systems solely to produce the plots is inefficient and error prone. On the other hand, creating the figures "by hand," through software such as MS PowerPoint or Adobe Illustrator, may lead to unintentional errors and require the plots to be redrawn when a review is updated. Additionally, while the field of evidence synthesis software has grown rapidly in recent years, 9 this growth has not been equally distributed across the different aspects of the systematic review process. For example, a recent review found several software offerings aimed specifically at the abstract screening stage of the review process, 10 but no similar time-and errorreducing tool has been proposed for visualizing the results of risk-of-bias assessments.
It is now straightforward to produce such a tool, thanks to the availability of powerful computing offerings including R, RStudio, and Shiny (an R package for building interactive web apps). [11][12][13] Here, we present robvis (Risk Of Bias VISualization), 14 an R package and Shiny web-app that allows users to create publication-ready risk-of-bias plots quickly and easily. While primarily designed for use with the major risk-ofbias assessment tools used in health research (ROB2, ROBINS-I, and QUADAS-2), the tool allows users to visualize the results from any domain-based risk-of-bias assessment or quality appraisal tool.
The tool is open-source and available to use free of charge. Users can download a stable version of the R package from CRAN (https://cran.r-project.org/ package=robvis); or access and contribute to the opensource code that powers the package via GitHub (https:// github.com/mcguinlu/robvis). Extended guidance for the tool, including a step-by-step walk-through for those new to the R programming environment, is also available via the "Doing Meta-Analysis in R" online guide. 15 Below, we discuss the tool's development and key functionality.

| DEVELOPMENT
Development of robvis began in April 2019 at the Evidence Synthesis Hackathon (ESH), an event which brings together interested researchers, practitioners, and coders to discuss and develop new open-source evidence synthesis technologies. Test versions of both the R package and the web app were made available in early June 2019, with attendees of the ESH and members of the Bristol Appraisal and Review of Research (BARR) group at the University of Bristol being invited to test the tool and provide feedback. This feedback, along with other feature suggestions from the wider evidence synthesis community captured via GitHub issues, was incorporated and the first release version of the package was uploaded to CRAN in November 2019. The tool has been well received and is beginning to be cited in the evidence synthesis literature. 16

.1 | Tool templates and example data sets
At the time of writing, the tool includes templates for three major tools: the Cochrane RoB 2 tool for assessing randomized trials, 3 the ROBINS-I tool for assessing non-randomized studies of interventions, 4 and the QUADAS-2 tool for assessing diagnostic accuracy studies. 5 These templates automatically apply the correct risk-of-bias domain names to the figures, and label the judgement levels appropriately (eg, "Low," "Some Concerns," and "High" in the case of the RoB 2 tool). In addition, robvis contains a general template that can be used to visualize the result of any domain-based assessment tool. The generic template has greater flexibility than the tool-specific templates, allowing a user-specified number of domains and custom domain titles to be used. It is suitable for use with the original version of the Cochrane risk-of-bias tool for randomized trials, 21 in which flexibility in the specification of domains was permitted. Since users of more recent tools such as RoB 2 and ROBINS-I are not permitted to modify the domains, we strongly encourage use of the in-built templates for these.
In order to help users familiarize themselves with the package and its functionality, robvis contains built-in example data-sets for each template. We illustrate the example data for the RoB 2 tool for assessing risk of bias in randomized trials in Table 1.

| Data import
robvis expects the risk-of-bias data file to be arranged in a specific way (see Table 1 for an example). The first

Highlights
• Risk-of-bias assessment is a key element of the systematic review workflow. • No other tool exists explicitly for the purpose of visualizing risk-of-bias results. • Here, we present robvis, an open-source R package and Shiny web app for creating publication-ready risk-of-bias assessment figures. • robvis forms part of the metaverse, a collection of R packages designed to provide an evidence synthesis workflow in R.
column should contain the unique study/result identifier. The second-to-last column should contain the overall risk-of-bias judgments. The final column contains some measure of the result's precision (eg, the weight assigned to that result in a meta-analysis, or the sample size of the analysis that produced the result). This weight column is used to create the summary bar plot, as current guidance recommends dividing the bars to show the proportion of information at each level of risk of bias, determined by the cumulative weight at that level, rather than simply showing the number of studies/results in each category. 2 If a measure of precision is not available, or to reproduce

T A B L E 2
Description of the arguments available in the two main functions of the robvis R package. "X" indicates that the option is available for the respective function Argument rob_traffic_light() rob_summary() Description data X X Defines the data frame containing the summary (domain) level risk-of-bias assessments. See the text and Table 1 for the format expected by robvis.
color X X Defines the color scheme for the plot. The default is colour = "cochrane" which uses the "Cochrane" (red, yellow, green) colors, while a preset option for a color-blind friendly palette is also available (colour = "colourblind"). Alternatively, users can specify their own color scheme for example, colour = c ("#f442c8," "#bef441," "#000000"). Note: This option (tool = "Generic") reflects the general template name used in the current development version of robvis, which will become the standard for all future iterations of the package. However, in the current CRAN version, the generic template is accessed using tool = "ROB1".
"equally" weighted bar charts as have traditionally been presented in Cochrane Reviews to date, these weights may all be specified to be 1.

| Functions
robvis contains two main functions. The first, rob_-traffic_light(), creates a traffic light plot. This displays every risk-of-bias judgement in a matrix, with domains along the horizontal and results/studies down the vertical, similar to the data set. The second function, rob_summary(), creates a weighted bar plot.
This shows the proportion of information with each risk-of-bias judgement separately for each domain in the assessment tool specified. A worked example using these functions is outlined below, illustrating the simple steps involved in creating riskof-bias plots using robvis. A detailed description of the additional options that can be used with each function is presented in Table 2. All examples produced in this article are created using the stable version available from CRAN.
To install and load the package in R, enter the following into the console: install.packages("robvis") library(robvis) F I G U R E 1 Example risk of bias traffic light plot of ROB2 assessments created using robvis and the colourblind palette [Colour figure can be viewed at wileyonlinelibrary.com] Using the example data set (data_rob2) that is built into the package and is presented in Table 1 for reference, the traffic light plot shown in Figure 1 is created using: Similarly, using the same data set, the summary barplot shown in Figure 2 is created using: rob_summary(data = data_rob2, tool = "ROB2", overall = TRUE)

| Further customization
The ggplot2 package in R, based on the "The Grammar of Graphics," allows users to create detailed graphics and was used to create the templates found in robvis. 22,23 As a result, both robvis functions return a ggplot object, meaning they can be easily customized further using the ggplot2 framework. For example, to add a title to the summary bar plot: plot <− rob_summary(data_rob2, tool = "ROB2") plot + ggtitle("Summary Bar Plot") For a full discussion of the range of post-production modifications that can be made to the plots via the ggplot2 package, we refer the reader to the extensive guidance available. 23

| Shiny web app
Although robvis was originally designed for use in the command-line-based R programming environment, we developed a web app to make the tool accessible to those without knowledge of R. This is available via www. riskofbias.info. 24 The app was built using Shiny, 13 an R package which makes it easy to produce interactive webapps, and provides a graphical user interface (GUI) for the robvis package, allowing users to interact with the functions presented above without the need to download R or type any commands. Users can upload their data as either an Excel spreadsheet (recommended) or a comma-separate values (CSV) file, or can manually enter it directly into the app. Uploaded data are passed through a number of quality control checks that ensure the app will work correctly, and users are prevented from producing the plots until any issues identified have been addressed. Finally, users can customize their plots by defining a color scheme and other parameters, all through the online app.

| DISCUSSION
robvis facilitates the rapid production of two common risk-of-bias assessment figures at publication quality. By implementing robvis both as an R package and a Shiny web app, its functionality is available to evidence synthesists with varying levels of ability in R. robvis serves as an example of the advantages of "packaging" the R scripts that evidence synthesists often create for personal use. 25 It is likely that several other evidence synthesists have written scripts to produce similar risk-ofbias plots to those presented here-in fact, we personally know of at least one other research group that has done so. This duplication of time and effort is inefficient, and creating and sharing well-documented R packages represents one way to reduce this inefficiency. Taking this approach one step further, Shiny apps represent a straightforward way to provide a user-friendly GUI for a F I G U R E 2 Example risk of bias summary plot of ROB2 assessments created using robvis and the standard Cochrane palette [Colour figure can be viewed at wileyonlinelibrary.com] newly created R package within a very short time-frame, expanding the potential pool of users of the package to anyone with an internet connection.
Creating a package using R has a number of particular advantages. R provides access to a range of powerful tools including the ggplot infrastructure as demonstrated above, and RMarkdown, which enables creation of documents that can be rendered in a range of formats such as PDF, HTML, or Word. 26 Furthermore, and focusing specifically on evidence synthesis, building new tools as packages in R allows for easy integration with the range of existing evidence synthesis packages. Recently, the metaverse project, 27 of which robvis is a part, has begun to curate a collection of R packages that cover different aspects of the systematic review and meta-analysis process which, when taken together, form a coherent endto-end open-source alternative to commercial offerings such as Covidence or Review Manager. Key offerings in this suite of packages include litsearcher, which facilitates systematic search strategy development, revtools, a package for managing the review process and performing title and abstract screening, metaDigitise, a package for automatic extraction of data from figures in research papers, and metafor, a package for conducting metaanalyses in R. [28][29][30][31] While robvis is a stable package, a range of additional functionality could be added. At present, the number of tools with a specific template included in robvis is limited-adding additional templates is a priority. For example, a template for ROBIS, the risk-of-bias assessment tool for systematic reviews, 6 is in development. Additionally, the robvis tool does not yet allow for the production of paired forest plots, where the risk-of-bias judgement is presented alongside each specific result included in the meta-analysis. 2 This was initially considered to be beyond the scope of the tool, as it involves the visualization of something other than risk-of-bias assessments. However, following user-driven demand, this functionality is in development and will be available in the near future. Finally, we would like to add similar functionality to that provided by the metafor:: reporter() function, which generates a brief paragraph of text describing the results of a meta-analysis. The future robvis::reporter() function would provide a boilerplate description of the assessment tool used and the key domains at risk of bias.