How do Cochrane authors conduct web searching to identify studies? Findings from a cross‐sectional sample of Cochrane Reviews

Background: Searching the World Wide Web using search engines and websites can be conducted to identify studies for systematic reviews. When searching to support systematic reviews, the searcher faces challenges in using the basic search interfaces of most search engines and websites. Objectives: To describe and evaluate current practice of web searching in a cross-sectional sample of Cochrane Reviews. The study also describes the stated aims of web searching, i.e. the identification of published or unpublished studies or both. Methods: A six-month cross-sectional sample of Cochrane Reviews was identified via the Cochrane Library. Reviews were inspected for detail about web searching. Findings were described and evaluated using a framework of key principles for web searching. Results: 423 Cochrane Reviews published August 2016–January 2017 were identified of which 61 (14%) reported web searching. Web searches were typically simplified versions of the bibliographic database search. Advanced and iterative approaches were not widely used. Google Search and Google Scholar were the most popular search engines. Most reports stated identification of grey literature as their aim. Conclusion: Basic web search interfaces necessitate simple searches. However, there is scope to use more diverse search features and techniques and a greater variety of search engines.

• Searches used to identify studies via web searching in a cross-sectional sample of Cochrane Reviews typically involved simplified versions of bibliographic database searches.
• There is scope for more advanced searching than observed in the sample, albeit the optimal use of advanced search features and techniques requires further research.
• Google Scholar and Google Search were the most popular search engines in the sample.
• Most reviews reported that their aim in web searching was to identify grey literature study reports.

Background
Searching the World Wide Web (hereafter, web searching) via search engines and websites is one of several supplementary search methods that can be used to identify studies for inclusion in a systematic review (Cooper, Booth, Britten, & Garside, 2017). The primary search method for a systematic review usually consists of searching bibliographic databases, which provide access to a large number of journal articles. Supplementary search methods, such as citation searching, contacting authors, searching trials registries and web searching, aim to identify studies that are not retrieved by searching bibliographic databases. This is important when carrying out a Cochrane Review in order to identify a comprehensive set of relevant studies for the purpose of ascertaining the best evidence based estimate of the effectiveness of an intervention (Lefebvre et al., 2019a). Reasons for missing studies in bibliographic databases include the omission of the necessary search terms and instances where relevant studies are not indexed in the searched bibliographic databases. Web searching is not, however, mandatory for Cochrane Reviews, and thus the decision to conduct web searching is made on a case-by-case basis depending on the likelihood of identifying relevant studies using this approach (Higgins, Lasserson, Chandler, Tovey, & Churchill, 2016).
Web searching usually involves using resources that are not purpose built for hosting and searching for studies. Commonly searched websites for systematic reviews that are not dedicated resources for identifying studies include those of charities, government health care departments and manufacturersall of which have multiple purposes, such as dissemination of information and marketing, in addition to providing access to studies (Briscoe, 2015(Briscoe, , 2018Godin, Stapleton, Kirkpatrick, Hanning, & Leatherdale, 2015;Stansfield, Brunton, & Rees, 2014). Web search engines can be used to identify studies or hints to studies (i.e. a promising lead) on websites which are investigated (Eysenbach, Tuische, & Diepgen, 2001). Commonly used search engines for systematic reviews include Google Search (www.google.com) and the scholarly search engine Google Scholar (https://sc holar.google.com; Briscoe, 2015Briscoe, , 2018. Google Scholar and other scholarly search engines, such as Microsoft Academic (https://academic.microsof t.com/), are exceptions to the general rule that web searching involves using resources that are not purpose built for identifying studies.
The non-specialist content and functionality of search engines and websites (i.e. from the point of view of searching for studies for systematic reviews) can present technical and logistical challenges (Lefebvre et al., 2019b;Stansfield, Dickson, & Bangpan, 2016). For example, the diverse content can make it difficult to focus a search sufficiently or decide how much time and resources to invest in searching and screening the results. Although some search engines and websites support the use of advanced search functions such as Boolean operators, truncation and date limits, they do not support the development of complex multi-line searches. Furthermore, websites are often searched by following links between webpages, which is potentially less systematic than searching using a pre-specified set of search terms due to its exploratory nature. With respect to reporting and updating searches, despite best efforts to report searches transparently, the reproducibility of searching is typically compromised because content on the web frequently changes and search engines use algorithms that change over time and personalise the results to the user's search history and location (Briscoe, 2015(Briscoe, , 2018. By contrast, content on bibliographic databases is stable and the search results do not vary depending on the location or search history of the searcher. Technical and logistical approaches to the challenges posed by web searching in the context of a systematic review have been presented with respect to conducting (Eysenbach et al., 2001;Giustini & Boulos, 2013;Godin et al., 2015;Haddaway, Collins, Coughlin, & Kirk, 2015;Haddaway, Collins, Coughlin, & Kirk, 2017;Harzing, 2007;Stansfield et al., 2016) and reporting web searching (Briscoe, 2015(Briscoe, , 2018Eysenbach & Trudel, 2005). This research on the challenges of web searching is summarised in systematic review guidance (Centre for Reviews & Dissemination, 2008;Collaboration for Environmental Evidence, 2013;Lefebvre et al., 2019b;Rethlefsen et al., 2019). A recent and comprehensive summary on conducting web searching for systematic reviews is presented in the online Technical Supplement (Lefebvre et al., 2019b) to the Searching for and selecting studies chapter of the Cochrane Handbook for Systematic Reviews of Interventions (hereafter, Cochrane Handbook; Lefebvre et al., 2019a).
To what extent web searching conduct in systematic reviews reflects research and guidance on web searching has not previously been studied.
The rationale for carrying out such a study is twofold: both to glean insights on web searching from actual practice and to make suggestions for improving practice. To this end, the aim of this study is to review current practice of web searching in a sample of Cochrane Reviews with reference to the web searching guidance in the aforementioned Technical Supplement (Lefebvre et al., 2019b). This includes two specific objectives: 1. To describe and evaluate the conduct of web searching in a cross-sectional sample of Cochrane Reviews published in the sixmonth period August 2016 to January 2017 with reference to a framework of key principles for conducting web searching derived from the Technical Supplement (Lefebvre et al., 2019b). 2. To document and report the stated aim of web searching in each of the Cochrane Reviews that conducted web searching, that is whether web searching aimed to identify published studies in journal article format or grey literature study reports, or both. Given that the framework of key principles is derived from the Technical Supplement, which post-dates publication of the reviews in the crosssectional sample, it was not used to critically appraise and score the conduct of web searching. Rather the framework provides a structure for describing and evaluating the findings (Lefebvre et al., 2019b).

Identification of key principles on the conduct of web searching
Key principles for web searching were identified by reading and re-reading the web searching section of the Technical Supplement and extracting key items of guidance (Lefebvre et al., 2019b). The primary sources on web searching cited in the Technical Supplement were also inspected for any additional useful detail. Update searches for primary studies on web searching for inclusion in the Technical Supplement were last reviewed in April 2019 (Lefebvre et al., 2019b).
In total, eight key principles on the conduct of web searching were identified in the Technical Supplement (see Table 1; Lefebvre et al., 2019b). They are divided into general principles (1-3), search engine specific principles (4-5) and website specific principles (6)(7)(8). Research in the peer reviewed literature relating to the key principles is cited in Table 1 where available.

Eligibility criteria
The cross-sectional sample of Cochrane Reviews used in this study was the same as in a sibling study on the reporting of web searching, i.e. the six-month period August 2016 to January 2017 (Briscoe, 2018).
Cochrane Reviews were eligible for inclusion if they reported using web searching to identify studies for inclusion in the review. Web searching was defined 'as the use of a search engine or website that has not been specifically designed to host and facilitate searching for studies' (Briscoe, 2018). This included general web search engines, such as Google Search, and the websites of topically relevant organisations, such as charities and manufacturers. The exceptions to these inclusion criteria were scholarly search engines, such as Google Scholar (https://scholar.google.c om/) and Microsoft Academic (https://academic. microsoft.com/), which are specifically designed to host and facilitate searching for studies. These were included in the study as they have similar design features and functionality as general search engines. Web based trials registries were excluded as dedicated resources for identifying studies, for example ClinicalTrials.gov and the World Health Organisation (WHO) International Clinical Trials Registry Platform Search Portal (ICTRP).

Search, screening and data extraction
The searching and screening processes were undertaken as part of an earlier review on the reporting of web searching in Cochrane Reviews by SB (Briscoe, 2018). Cochrane Reviews were identified by searching the Cochrane Database of Systematic Reviews using the wildcard symbol (i.e. asterisk) and date limited using the Online Publication Date feature. The search was carried out in February 2017. The screening process to identify eligible reviews involved inspecting the methods section and appendices of each Cochrane Review thus identified for detail about web searching. In addition, to capture detail about web  (Mahood, Van Eerd, & Irvin, 2014). Instead, a limited pre-specified number of results may be screened or the screening process may stop after several pages of results are screened without identifying relevant information. A limited approach to screening is justified on the basis that search engines rank results according to relevance, so the probability of identifying relevant information is higher towards the beginning of the retrieved results (Stansfield et al., 2016). An exception is Google Scholar, where research suggests that it can be useful to screen the full set of available results, in particular, when seeking to identify grey literature (Haddaway, Collins, Coughlin, & Kirk, 2015). Publish or Perish software can be used to assist the search, download and screening process when using Google Scholar (Harzing, 2007)  searching that was not reported in the methods section or appendices, the Find (Control-F) search feature was used to search each review for the terms 'web', 'internet', 'online' and 'Google'.
Detail on the conduct of web searching was exported from the data extraction forms created for the sibling study on the reporting of web searching (Briscoe, 2018). The data extraction forms from Briscoe (2018) included detail on: 1. the names of any search engines or websites that were searched; 2. the URL(s); 3. the date(s) searched; 4. the search terms; 5. the number of search results. The data extraction form also included a freetext box for 'Any other detail reported about web searching'. These five items and the free-text box provided sufficient detail for describing and evaluating the conduct of web searching. To facilitate this process, a new data extraction form was developed that mapped onto the key principles in Table 1. Detail on web searching in the data extraction forms from Briscoe (2018) was then imported into the appropriate section of the new data extraction form. The conduct of web searching was then described and evaluated according to each principle.
The MEDLINE search strategy from each Cochrane Review was used to describe and evaluate those key principles that made reference to bibliographic database search strategies (e.g. key principle 1, 'Search terms used for web searching should be based on the search terms used for searching bibliographic databases'). The comparative complexity of the web search and MEDLINE search in each review was described and evaluated for key principle 2 with respect to: the number of search terms in the web search strategy compared to the MEDLINE search strategy; the components and Boolean structure of the search as described by the PICOS question formulation format (Population, Intervention, Comparator, Outcome and Study type); the use of phrase, proximity, truncation or wildcard searching; and the number of iterations of the search. Spider plots were used to show the distribution of types of websites searched per category of review for key principle 8. Included Cochrane Reviews were classified into categories of intervention with reference to the classification scheme for types of intervention described in Smith et al. (2015).
Findings relating to other key principles were summarised narratively and median or mean figures used where appropriate.
Data were also collected on the stated aim of web searching in the reviews in the sample. In particular, we sought to distinguish between searches that aimed to identify studies published in journal article format and searches that aimed to identify grey literature, that is 'that which is produced on all levels of government, academics, business and industry in print and electronic formats, but which is not controlled by commercial publishers' (Farace & Frantzen, 1997). In the context of a systematic review, potentially relevant grey literature typically includes ongoing

Websites Web searching involves following links between webpages and websites
Searching via websites is often less structured than using pre-specified terminology but a systematic approach should still be pursued (Stansfield et al., 2016) 8 Websites The selection of websites to search will be determined by the review topic The number of generic types and specific websites searched for different reviews will vary (Stansfield et al., 2016). Commonalities might be detectable between similar reviews studies, recently completed studies not yet in journal article format, and studies not intended for journal article publication, such as study reports produced by organisations without using a commercial publisher. For some reviews, these data were reported in the data extraction form for Briscoe (2018). However, to ensure no data were missed the Cochrane Reviews were re-visited and inspected for this detail.

Search results
The search identified 423 Cochrane Reviews published in the six-month period August 2016 to January 2017. Of these, 61 reviews (14%) reported using a search engine or website to identify studies or for an unspecified purpose (see Appendix). They included 25 reviews (6% of the total) that reported searching one or more search engine and 39 reviews (9% of the total) that reported searching one or more website. Three reviews (<1% of the total) reported web searching using both search engines and websites.
The 61 Cochrane Reviews that reported conducting web searching were all classified as Intervention reviews in the Cochrane library. The interventions in each review were further classified using the framework developed by Smith et al. (2015) as: complex interventions (n = 4); control of chronic disease (n = 19); diagnostic (n = 1); drugs for prevention of disease (n = 2); education and behaviour change (n = 6); health systems (n = 2); implementation programmes (n = 1); injury prevention (n = 1); maternal and neonatal (n = 1); nutrition (n = 1); pain management (n = 5); surgery and radiation (n = 10); treatment of infectious disease (n = 7); and vaccines (n = 1). See Appendix for a full list of included reviews classified by intervention type. All of the reviews included randomised controlled trial (RCT) study designs except two McLaren et al., 2016). A minority of reviews included other study types in addition to RCTs, including controlled before-and-after studies (n = 10), controlled clinical trials (n = 3), interrupted time series (n = 8), non-randomised controlled trials (n = 5) and uncontrolled before-and-after studies (n = 1). See Appendix for a breakdown of included study designs for each review. Twenty-five Cochrane Groups were represented in the sample (see Appendix).

Conduct of web searching in the cross-sectional sample of Cochrane Reviews
Findings on the conduct of web searching in the cross-sectional sample of Cochrane Reviews are described below with reference to the eight key principles in Table 1.
Key principles 1 to 3 relate to web searching in general.
1. Search terms used for web searching should be based on the search terms used for searching bibliographic databases.
Nine Cochrane Reviews reported sufficient detail about the search strategies used for web searching to be compared to the bibliographic database search strategies. Of these, five reviews reported the search terms used for one or more search engine Smith et al., 2016) and four reviews reported the search terms used for one or more website (Flodgren et al., 2016;McLaren et al., 2016;. Of the five reviews that reported the search terms used in search engines, three reported using search terms that were all also used in the MEDLINE search strategy . The remaining two reviews used search terms that were not used in the MEDLINE search strategy, albeit this was only one search term per search strategy Smith et al., 2016). In Smith et al. (2016), the additional term was combined using the OR Boolean operator, making this component of the search strategy more sensitive than the MEDLINE search strategy. In , the additional term was combined with the AND Boolean operator making this component of the search strategy more precise than the MEDLINE search strategy.
Of the four reviews that reported search terms used for searching websites, all reviews used search terms that were all also used for searching MEDLINE (Flodgren et al., 2016;McLaren et al., 2016;. However, one review configured the Boolean relationship of two search terms differently for the website and MEDLINE search; in the former AND was used to combine two search terms and in the latter OR was used, thus making the website search more precise than the MEDLINE search . 2. A simplified search strategy (compared to the bibliographic database search) or multiple searches of the same resource might be required.
The same nine Cochrane Reviews (see key principle 1) reported sufficient detail to compare the complexity of the web search strategies with the database search strategies Flodgren et al., 2016;McLaren et al., 2016;Smith et al., 2016;. The comparative complexity of the web search and MEDLINE search in each review is summarised in Table 2.
All the reported web search strategies were simplified versions of the MEDLINE search strategy. The median and range of search terms used for web search engines were 4(3-13), and the median and range of search terms in the corresponding MEDLINE search strategies were 21(15-63). The median and range of search terms used for websites was 5(1-17), and the number of search terms used in the corresponding MEDLINE search strategies was more than 100 in all reviews. Overall, the simplification process followed a trend of reducing the number of search terms to less than 10, regardless of how many search terms were used in the bibliographic database search strategy. Only two reviews used more than 10 search terms for web searching (13 and 17 search terms, respectively; . This general approach meant that the extent of the simplification was much greater in some reviews than others. For example, two reviews that included more than 300 search terms in the MEDLINE searches simplified the web search strategy to six search terms, i.e. less than 2% of the terminology in the original database search (McLaren et al., 2016;. By comparison, a review that included 16 search terms in the MEDLINE search simplified this to eight search terms for the web search strategy, i.e. half the number of terms in the original database search (see Table 2) .
In almost all web search strategies, the PICOS structure was also simplified. The most common simplification of the PICOS structure was the removal of study type terms from the search strategy used for web searching (see Table 2). All search strategies used in search engines (n = 5) included search terms for the population and intervention of interest. By comparison, the search strategies used in websites included multiple examples in three reviews where only one PICOS component was used (see Table 2; Flodgren et al., 2016;McLaren et al., 2016).
Combinations of PICOS components are achieved using Boolean logic (AND, OR and NOT). Boolean operators were not always explicitly stated in the search strategies; instead, the Boolean logic was determined by the logic of the selected search interface, for example if a search was reported as 'With all the words' this implied the AND Boolean operator, whereas if a search was reported as 'With at least one of the words' this implied the OR Boolean operator . 'None of the words' was also used which is equivalent to NOT . Furthermore, search engines often combine search terms using AND by default, including both Google Scholar and Google Search (Lefebvre et al., 2019b). Two reported searches of Google Scholar and Google Search respectively did, however, use AND, in conjunction with parentheses and OR, to build search strings: abscess AND (packing OR dressing) (Smith et al., 2016) (In Vitro Maturation OR IVM) AND (Human chorionic gonadotrophin OR HCG)  No search strategies reported for search engines made use of phrase, proximity, truncation or wildcard searching. One search restricted results to   where search terms appeared in the title, which could be construed as a simplification of the database search (which searched in title and abstracts) or could be construed as a way of limiting the number retrieved for screening (see key principle 4). Phrase and truncation searching was reported in four website searches in two reviews McLaren et al., 2016).
Two reviews conducted multiple searches via a search engine using a different set of search terms for each iteration . This meant that relatively complex database searches could be broken down into simplified parts to allow for a comparable search to be carried out via a search engine. In the remaining seven reviews, all web searches were conducted as a single search within each resource.
3. Wherever possible, a similar approach should be used for different web resources.
Four Cochrane Reviews reported searching more than one web based resource in sufficient detail to compare the conduct of searching between resources (Flodgren et al., 2016;McLaren et al., 2016;. In all four reviews, the searches were of websites. McLaren et al. (2016) reported searches of three different websites using mainly but not wholly the same set of keywords in each, differing by one or two search terms per search.  reported single keyword searches of two websites and multiple keyword searches (using nine and 17 keywords, respectively) of a further two websites. The single keyword searches both use the same word ('corruption') and the multiple keyword searches include a list of synonyms, for example. corruption OR corrupt OR "corruptive payment" OR "corruptive payments" OR bribe OR bribes OR bribery OR forgery OR fraud OR fraudulence OR fraudulent OR swindle OR swindling OR kickback OR kickbacks OR "informal payment" OR "informal payments"  .
Inspection of the four websites searched using a single keyword revealed that they do not support multiple keyword searches.
Key principles 4 and 5 relate to search engines.

A search engine might retrieve an
unmanageable number of results in which case the searcher will need a strategy for limiting how many are screened.
Five Cochrane Reviews that reported using a search engine also reported the total number of results that were screened. Of these, one review reported screening the total number of results  and one review reported using a date limit and then screening the total number of results . Three reviews reported screening a subset of the total retrieved results .  conducted one search of Google Scholar, retrieving 963 results that were screened in full. The search terms used were restricted to title only, either as a pragmatic limit for the purpose of screening or to increase the precision, reflecting the limitation that Google Scholar permits title or full-text searching but not abstract searching.  conducted 12 searches of Google Scholar which were each date limited to one calendar year, namely 2016. The searches retrieved a total of 550 results which were de-duplicated to reveal a total of 146 unique results . The reviews that reported screening a subset of the total results screened: • the first 500 results  • the first 200 results  • the first 100 results . All searches were conducted on Google Scholar. One review included a rationale for the number screened, stating that 'in our experience the yield [in Google Scholar] after 200 hits is poor' .
The Technical Supplement states that searches can be limited to specific file types (e.g. PDFs) as a strategy for limiting the number of results retrieved (Lefebvre et al., 2019b). This approach was not observed in the cross-sectional sample.

Experimenting with or combining the results of different search engines might be beneficial for retrieving relevant studies.
Almost all reviews that reported using a search engine used either Google Scholar (n = 19) or Google Search (n = 11). One review reported using the Chinese scholarly search engine, Baidu Scholar (https://xueshu.baidu.com/) and one review reported using an unnamed search engine (see Appendix for full details).
Combinations of search engines were reported in seven reviews, in all cases combining Google Scholar and Google Search. Of these, six reviews reported that Google Search was used to identify topically similar systematic reviews, but did not report the expected complementary aim of searching Google Scholar . One review reported searching both Google Scholar and Google Search to identify unpublished studies .
Key principles 6, 7 and 8 relate to websites.
6. Strategies to limit the number of results for screening are less likely to be needed for websites than search engines.
Two reviews  reported the number of results identified via websites and one review  reported searching for a known study of interest via a website.  screened the results of a relevant webpage on three separate dates during the period that the review was undertaken, identifying three, zero and nine records, respectively. The search report suggests these are the total numbers of results on the page. ('Browsed the alphabetical list from the Interventions tab for "hyperbaric" and downloaded the webpage') .  conducted searches of 22 websites. The searches retrieved a median of 31 results (range 0 to 892), totalling 2143 results.
Neither of these two reviews reported that search results had been limited. In particular, neither indicated that only a subset of the retrieved results had been screened, as observed for the results of search engines. The total number of results retrieved by  was much higher than , mainly because one website retrieved 892 results, almost 30 times more than the median number of results in the full list of 22 websites searched. Table 3 compares the median and range of reported results that were either retrieved in total or screened for search engines and websites per resource in the cross-sectional sample (column 1); and the median and range of results that were either retrieved in total or screened for search engines and websites per review in the crosssectional sample (column 2). Table 3 shows that although websites typically return a lower number of results per resource than search engines, the actual number of results screened from website searches per review can be higher than the number retrieved (or screened) by search engines where review authors search multiple websites. However, this finding was influenced by an outlier result in one website (n = 892 hits). The results are the same per resource and per review for search engines because only one set of results from a search engine was reported per review. 7. Web searching involves following links between webpages and websites.
Three Cochrane Reviews reported following links between webpages McLaren et al., 2016;.  reported browsing the alphabetically listed Interventions tab of the Research Autism website (ResearchAutism.net) to identify and download relevant content, specifically, information on hyperbaric interventions. McLaren et al. (2016) reported following the menu headings of three websites to guide the search, in particular reporting the specific menu headings that were sequentially followed.  reported browsing two websites but provided no specific detail on how the searches were conducted, that is stated 'browsed' without providing further details.
8. The selection of websites to search will be determined by the review topic.
Thirty-nine Cochrane Reviews reported searching a website. The frequency of types of website searched in these reviews is presented in Table 4.
Charities and NGOs included a diverse assortment of not-for-profit organisations; commercial organisations mainly included manufacturers of medical interventions and private health care providers; government included government departments and associated bodies (e.g. the US Centers for Disease Control and Prevention and the UK National Institute for Health and Care Excellence); professional societies included colleges of medicine and other health care professions; research organisations included universities and other organisations with a research focus (e.g. the international Alliance for Health Policy and Systems Research and the Canada-based Program in Evidence-based Care); other included clearing house websites (e.g. the US Clearing House on Abuse and Neglect of the Elderly).
Fifteen of the 39 reviews that reported searching a website did not report a full list of specific websites, for example reported examples of websites searched, or only reported searching types of websites, for example reported that charity websites were searched without reporting specific websites. A full list of websites searched was reported in 24 reviews (see Appendix). The median number of websites searched per review was two (range 1-30), and the most frequently reported number of websites searched was one (n = 9). The categories of review that searched the most websites (calculated as the mean number of websites searched per reviews in each category) were (where n = number of websites searched): • complex intervention reviews (n = 6) • education and behaviour change reviews (n = 22) • implementation reviews (n = 29) For the 24 reviews that reported a full list of websites searched, the distribution of types of websites searched per category of review is shown using spider plots in Figure 1. Figure 1 shows that the distribution of types of website searched varied for different review topics (i.e. categories of review), demonstrating the principle that the type of website searched will depend on the review topic. The reasons why certain types of websites are searched for some review topics but not others are not obviously apparent in Figure 1. An exception to this is implementation reviews which are the only category of review to search websites of professional societies, reflecting an interest in translating research into a professional context. Four categories of review only searched one type of website (drugs for prevention of disease, injury prevention, nutritional and pain management).

Aims of web searching
Of the 61 included Cochrane Reviews, 50 reviews reported the type of literature that web searching aimed to identify. These included 41 reviews that reported aiming to identify grey literature and nine reviews that reported aiming to identify published studies in journal article format. Websites were used exclusively to identify grey literature, mainly trial data from ongoing or recently completed studies, in 32 reviews. Search engines were used to identify journal articles in nine reviews and a further nine reviews reported using search engines to identify grey literature.

Conduct of web searching
This study has reviewed the conduct of web searching in a six-month cross-sectional sample of Cochrane Reviews using a framework of key principles for web searching derived from the Technical Supplement (Lefebvre et al., 2019b) to the Searching for and selecting studies chapter of the Cochrane Handbook (Lefebvre et al., 2019a).
The results clearly showed that web searches are simplified versions of bibliographic database searches. Overall, this is in accordance with the recommendations in the Technical Supplement (Lefebvre et al., 2019b).
The observed trend for using less than 10 search terms per search string might indicate that longer search strings are not well-supported by web search interfaces. No web searches used proximity searching and very few reported using phrase, truncation or wildcard searchingnone via search engines. Although this reflects advice in the Technical Supplement that comparable advanced search features to bibliographic databases might not be supported in web search interfaces, such Figure 1 Distribution of types of websites searched per category of review in the cross-sectional sample of Cochrane Reviews (total number of reviews = 24). n in parentheses denotes number of reviews represented per category, for example 'Complex interventions (n = 3)' denotes 3 complex intervention reviews. The number of websites searched per type of website has been normalised so that the distribution of types of websites searched can be shown on the same scale, i.e. the most frequently searched type of website per category of review was set to 1, and the frequency of searching other types of websites was calculated relative to 1 features are sometimes supported and the Technical Supplement suggests that they might be useful (Lefebvre et al., 2019b).
To the advice presented in the Technical Supplement, we add two cautionary notes on the use of advanced search features in web search interfaces. First, in general search engines have moved away from supporting search features that enhance the user's ability to precisely map a search query to the search results (Manning, Raghavan, & Sch€ utze, 2008). This so-called classical approach to information search and retrieval has been replaced by the use of algorithms to rank results according to their authoritativeness and relevance (Manning et al., 2008). In this context, the unqualified advocacy of complex search strings to improve the precision or sensitivity of search results in web search engines could be seen as a retrograde attempt to return to an earlier stage in the development of search engines.
Secondly, uncertainty exists about which advanced search features are supported. For example, two reported searches of Google Scholar and Google Search used the AND Boolean operator, in conjunction with parentheses and the OR operator, to build search strings. Some sources report that AND and parentheses are unsupported search operators in Google search engines, for example (Shameava, 2015;Tay, 2015), whilst other sources report that both are supported, for example (Hardwick, 2018;Van Hoosear, 2013). Furthermore, although neither AND nor parentheses are listed as supported operators on the Google Search syntax help page (Google Search Help, 2019), there are well-documented examples of supported search operators that are not listed by Google, for example the AROUND proximity operator (Chitu, 2010). This lack of clarity about supported features from search engine providers further complicates the use of advanced features (Bates, Best, McQuilkin, & Taylor, 2017). Our advice is that, when searching for studies for systematic reviews, searchers should try their hardest to use advanced search features appropriately, including checking whether the results of searches map onto what they are expecting to see, for example whether when using the AND Boolean operator, all the expected search terms are appearing in the results.
Perhaps surprisingly, the use of limits on the number of results screened was reported for Google Scholar but not its larger sibling, Google Search. Two reviews in our sample reported screening less than 300 results from Google Scholar, with one review presenting anecdotal evidence that 'the yield [in Google Scholar] after 200 hits is poor' to justify this decision . However, research published around the same time as the reviews in the sample (Haddaway et al., 2015) indicates that at least 300 results should be screened when searching Google Scholar for published literature, and that the results should be comprehensively screened when searching for grey literature. No limits were observed for searches of websites, reflecting the advice in the Technical Supplement that the practice of limiting results from websites is less likely to be required due to their relative size and scope (Lefebvre et al., 2019b). The use of only one PICOS component in some website searches also reflects the more bounded content accessed via websites compared to search engines, where at least two PICOS components were always included in the search.
The almost exclusive use of Google Scholar and Google Search reflects their dominance amongst search engine users (Sullivan, 2013). The Technical Supplement suggests alternatives to Google Scholar and Google Search that might be advantageous for the identification of studies, including DogPile (www.DogPile.com) and DuckDuckGo (https://duc kduckgo.com/; Lefebvre et al., 2019b). At the time of writing the most recent comparative study of search engines for the purpose of systematic searching for studies is almost 20 years old (Eysenbach et al., 2001). Eysenbach et al. (2001) evaluated 11 search engines with respect to their ability to handle complex search queries using Boolean, truncation and proximity search operators. Only one search engine, now obsolete, was found to be adequate to the task. In view of developments in search engines outlined above, any such comparative study today should also consider differences between search engine results arising due to algorithms, for example the identification of unique content and the reproducibility of search results.
When following links between webpages on websites to identify information, between one and three clicks to find relevant content is considered to be optimal (MacFarlane, 2007). When searching for studies for systematic reviews, searchers might be expected to search more extensively. However, none of the reports of searching using this approach (e.g. using menu headings to move between webpages) described needing more than two clicks to satisfy an information need. The observed variation in the description about the search process when following links on websites might not simply reflect different reporting standards, but rather the difference between browsing and navigating websites. Browsing is exploratory and relatively haphazard, for example, it might involve speculatively following links between webpages and websites rather than a clearly labelled pathway, and there is no clear endpoint to the search process. Navigating is structured by following a clearly identifiable path using menu headings to access the required information. Clearly, browsing is more challenging to document and report in detail. By contrast, directed or navigational searching is helpful and relatively easy to document and report (Stansfield et al., 2016).

Aims of web searching
Viewed collectively, the stated aims of web searching via websites and search engines in the cross-sectional sample revealed a dominant expectancy that web searching would identify grey literature. Although search engines were used more broadly than websites to identify studies in journal article format and grey literature, the focus of searches for the former was the identification of systematic reviews (which were subsequently checked for relevant primary studies) rather than direct identification of relevant primary studies. This approach to web searching might reflect confidence in bibliographic databases and other supplementary search methods for identifying studies in journal article formatparticularly in reviews that only include RCTs, which are well-indexed and largely identifiable via bibliographic databases and CENTRAL (Lefebvre et al., 2019b). Nonetheless, aiming to identify grey literature does not necessarily preclude the searcher from identifying journal articles, provided that the searcher does not attempt to exclude such studies. For example, through the use of publication type search terms. There was no evidence of this in the search strategies reported in the crosssectional sample. See Box 1 for practical tips on web searching arising from this discussion.
Box 1: Practical tips for conducting systematic web searching • Experiment with different search terms to refine the best approach and be prepared to carry out multiple searches when using simple search interfaces.
• Experiment with using different PICO components one PICO component might be sufficient.
• Take time to become familiar with the advanced search features of search engines and websites. Try to find up-to-date information as search features frequently change.
• To ascertain whether a search operator is working correctly, check whether the search results reflect what you expect to see, for example if using AND are all the relevant search terms appearing in the results?
• Take time to identify relevant sources to searchtopic experts may be useful in this regard.
• Take time to become familiar with the layout of a website before deciding how to conduct a search.
• Document and report all web searching in sufficient detail for searches to be transparent and reproducible.

Strengths and limitations
This study uses a large cross-sectional sample of systematic reviews to derive data on the conduct of web searching, which has not been done before. The findings can be used to inform future web searching guidance and conduct in a unique way. The findings were, however, limited by the overall low standard of reporting of web searching in the sample. Although 61 reviews reported conducting web searching, only a minority of reviews reported sufficient detail for observations to be made regarding several of the key principles in the framework. The low standard of reporting of web searching in the sample is reported and discussed in detail in the sibling study (Briscoe, 2018).
A potential limitation is that the study relies on the Technical Supplement to develop key principles rather than a wider selection of guidance (Lefebvre et al., 2019b). However, the Technical Supplement was issued for consultation to all Cochrane information specialists and members of the Information Retrieval Methods Group (the official group established to advise on Cochrane information retrieval activities) in January 2018, and we are confident that it contains reasonably comprehensive guidance on web searching for systematic reviews, with particular emphasis on Cochrane Reviews (Lefebvre et al., 2019b).
Finally, pharmaceutical manufacturer websites were included in the analysis which might have inadvertently captured data on searching company trials registries (which, if explicitly reported, would not meet the inclusion criteria for this study due to being specialised study identification tools). Furthermore, some of the data relates to searching repositories hosted on websites, which potentially have similar features to specialised study identification resources. Overall, a more in-depth exploration of the content, size and search features of web resources would be informative in terms of how the characteristics of web resources shape the development of search strategies and would facilitate a more detailed evaluation of web searching than has been possible in this study.

Conclusion
The systematic web searcher faces challenges when using non-specialist tools for systematic searching. This study has shown that web searching in the context of a systematic review is typically conducted using simplified versions of bibliographic database searches. This approach is necessitated by the limitations of web search interfaces. However, available search features extend beyond those identified within our crosssectional sample, and potentially advantageous approaches such as iterative searching were not widely reported. There is also scope for using a wider selection of search engines. Future studies on the conduct of web searching should test how different approaches to web searching affect the results that are retrieved and the overall contribution to the results and conclusions of systematic reviews.

Conflict of interest
We have no conflicts of interest.

Author contributions
Simon Briscoe conceived and designed the study and was involved in all stages. Michael Nunns contributed to the presentation of results and read and commented on the manuscript. Liz Shaw contributed to the development and presentation of the framework of key principles and read and commented on the manuscript.