Economic Evaluation in Global Perspective: A Bibliometric Analysis of the Recent Literature

Abstract We present a bibliometric analysis of recently published full economic evaluations of health interventions and reflect critically on the implications of our findings for this growing field. We created a database drawing on 14 health, economic, and/or general literature databases for articles published between 1 January 2012 and 3 May 2014 and identified 2844 economic evaluations meeting our criteria. We present findings regarding the sensitivity, specificity, and added value of searches in the different databases. We examine the distribution of publications between countries, regions, and health areas studied and compare the relative volume of research with disease burden. We analyse authors' country and institutional affiliations, journals and journal type, language, and type of economic evaluation conducted. More than 1200 economic evaluations were published annually, of which 4% addressed low‐income countries, 4% lower‐middle‐income countries, 14% upper‐middle‐income countries, and 83% high‐income countries. Across country income levels, 53, 54, 86, and 100% of articles, respectively, included an author based in a country within the income level studied. Biomedical journals published 74% of economic evaluations. The volume of research across health areas correlates more closely with disease burden in high‐income than in low‐income and middle‐income countries. Our findings provide an empirical basis for further study on methods, research prioritization, and capacity development in health economic evaluation.

Flow diagram of the data development process SUPPLEMENTARY TEXT Text S1 A note on database indexing terms Text S2 Supplementary information on article classification SUPPLEMENTARY TABLES Table S1 Searches in bibliographic databases Table S2 Mapping of 25 disease areas onto the Global Burden of Disease (GBD), International Classification of Disease (ICD-10), and search terms used

Figure S1 Flow diagram of the data development process
The figure is adapted from the flow diagram recommended in the PRISMA statement on systematic reviews (Liberati et al., 2009). The "eligibility" stage recommended by PRISMA is not used here as articles were not reviewed for quality; decisions to include records were based primarily on the record's source, title, and abstract; the full text was only screened where the title was unclear and the abstract was not available in any of the downloaded data.

Text S1 A note on database indexing terms
In developing our search strategy, we explored the use of controlled vocabulary indexing terms, if available, in each of the databases; unlike author-defined keywords, these terms are generally applied to publications by professional indexers from a pre-defined set. While this standardization should offer advantages, one drawback is the delays in their application; while many of the databases offer basic citation data as supplied by the journal first, indexing takes more time and so searches based exclusively on indexing terms will exclude the most recent literature, to which index terms have not yet been applied.
In Medline and Embase, indexing terms are known as medical subject headings (MeSH) and Emtree (which is not an acronym), respectively; both are organized hierarchically. While the only MeSH term relevant to our search is "cost-benefit analysis", Emtree appears much more detailed and appropriate, as it distinguishes "cost effectiveness analysis", "cost utility analysis", and "cost benefit analysis" from "cost control", "cost minimization analysis", and "cost of illness" within the broader indexing term "economic evaluation." When we compared the results of our searches in the title, abstract, and author-defined keywords for the key terms we identified above with the results of searches using MeSH terms (in Medline) and Emtree terms (in Embase), we found that the controlled vocabulary terms were both less specific and less sensitive. Our search terms identified many relevant articles missed by the MeSH and Emtree indexers. By contrast, the controlled vocabulary terms greatly increased the number of search results, but a review of the first hundred records identified by the MeSH term and, separately, by each of the three Emtree terms (i.e. 400 records in total) after excluding records identified by our search terms identified only one additional article meeting our inclusion criteria (identified by the Emtree term "cost-effectiveness analysis" ). We used this article to develop an additional set of search terms (based on "cost per x") and concluded that the MeSH and Emtree BM indexing terms were not useful for our final searches, as they identified a vast number of articles, many of which contained no cost or other economic data or analysis , while omitting many relevant publications .
Another database applying its own indexing is HEED . On the "compound search" page, HEED offers "type of econ eval" as a search category, as well as a "type of article". While the associated picklist does not make this obvious, HEED in fact categorizes economic evaluations as "cost effectiveness analysis", "cost utility analysis", "cost benefit analysis", "cost analysis", "cost of illness", "cost benefit analysis", and "cost consequences analysis"; it allows a single record to be classified as multiple types of economic evaluation, allows the user to specify only "applied study" as the "type of article", and reports that its indexers are professional health economists. After examining this classification, we found that the terms for CEA, CUA, and CBA were highly specific and useful when combined with "applied study" as type of study , however, many publications in the HEED database were not classified at all, making the search relatively insensitive even within the HEED database. In HEED, we therefore implemented two separate searches: 1) using the HEED classification of the type of economic evaluation, and 2) using our search terms in the title, abstract, and author-defined keywords, and excluding records containing the specified categories, such that any records identified by this search would be additional to records identified by the use of HEED's indexing.
The EconLit database uses the Journal of Economic Literature (JEL) classification system, however, unlike the indexing systems previously described, JEL codes are applied by the authors themselves. They break down the wider health economics field into 6 specified sub-fields, none of which mention in their descriptions or examples either applied or methodological work in economic evaluation; "general" and "other" health economics categories are also provided. On reviewing a selection of health economic evaluations in the EconLit database identified by title and abstract searches, we found that while some authors combine the codes "D61: Allocative Efficiency; Costbenefit analysis" (within the microeconomics heading) and "I12: Health Production" (within the health economics heading), other authors did not use these codes at all, choosing instead a wide variety of other codes within the health, microeconomics, and "miscellaneous" headings in particular, as well as others. Rather than using the JEL codes, we therefore decided to take a more sensitive approach in EconLit, and instead searched for "health" in all fields, which would capture the word "health" in JEL codes, but also in journal title, keywords, article title, or abstract; we combined this with keyword searches for our definition of economic evaluation.

Health areas
We developed a classification of 25 health areas so as to allow comparability with the Global Burden of Disease (GBD) estimates (World Health Organization., 2014), to be implementable with an electronic key term search, and to permit meaningful analysis. The GBD uses four hierarchical levels to classify disease. At its highest level, it classifies diseases as "Communicable, maternal, perinatal and nutritional conditions", "Non-communicable diseases" or "Injuries", while at its lowest levels, it breaks these down into 154 more specific conditions. We did not maintain the GBD's highest level classification because in some cases, it was not implementable (e.g. key term searches could not distinguish between communicable and non-communicable causes of respiratory diseases) and in other cases, we felt the distinction did not map coherently onto preventive and curative interventions (e.g. we separated "intentional injuries: self harm" from other injury categories and placed it in a single category with mental health issues).
A set of up to 49 search terms was developed for each of our health areas through an iterative process. We began by reviewing the names of sub-categories in the GBD and the categories and descriptions provided in the ICD-10 (World Health Organization., 2011) to develop an initial set of search terms. We then reviewed the titles and keywords of unclassified records in our database, and continued adding search terms until all records in our database which could be classified were classified according to at least one health area. Throughout the process, we reviewed samples of records within each health area, and reviewed in-depth the records identified by search terms we considered potentially ambiguous, before finalizing our search terms and disease classification.

Institutional and geographic affiliations of authors
We analyzed data on the institutional affiliation of all authors to develop a comprehensive picture of the institutions and countries contributing to health economic evaluations. We began by transferring the institutional affiliation data from wide to long form and implementing the country keyword searches previously developed. As affiliation data frequently did not name a country, unclassified affiliations were then iteratively reviewed and search terms for city names and non-geographic institution names (e.g. Harvard, Yale) were identified and linked to countries, taking care to avoid misclassifying search terms such as "York", which could refer to the city (York) or county (Yorkshire) in the United Kingdom, to York University in Canada, or to the city or state of New York in the United States. In this way, nearly all articles for which affiliation data were available were classified as being produced by researchers in one or more specified countries. This data was further cross-checked against the data on countries studied and inconsistencies reviewed. The original articles were sought to resolve inconsistencies and to obtain institutional affiliation data for any articles remaining without data. Articles were then classified by the income group of the country or countries of the author affiliations and the countries producing the greatest volume of economic evaluations were ranked within each income group.
We further identified the top ten institutions within each income group by volume of economic evaluations produced. The affiliation data for top-ranked countries within each income group were carefully reviewed to develop sets of specific key terms for institutions. As in previous work (Wagstaff andCulyer, 2012, Rubin andChang, 2003), schools, colleges and institutes were aggregated with the university to which they belonged, with the exception of the highly federal Universities of London, California, Texas, and other similar university systems, whose constituent members were analyzed separately. To the extent possible, hospitals and institutes were associated with their parent institution, even when that institution was not explicitly named. Even though they are independently owned and managed, Harvard's 16 affiliated hospitals were aggregated with Harvard. Once an initial set of ten institutions were identified for each income group, only affiliations from countries which had produced more than the tenth-ranked institution for that income group were reviewed to identify institutions which could have produced more economic evaluations than the currently tenth-ranked institution. For example, the tenth-ranked UMIC institution, the Instituto Mexicano del Seguro Social, produced 7 economic evaluations, and so only affiliations from UMICs which had produced at least 7 economic evaluations were reviewed to identify individual institutions which could have produced at least this number. The searches for city names were then used to facilitate the identification of institutions.
In addition, search terms were developed for international and inter-governmental organizations, such as United Nations agencies and the World Bank, and for multi-national pharmaceutical companies, regardless of the country, if any, with which they were associated in their affiliation data. These were then aggregated into two groups, "international organizations" and "pharmaceutical industry", to permit consideration of their relative influence.
This process allowed a comprehensive assessment of the total volume of articles produced by each country and by income group, as well as a comprehensive assessment of top institutions, taking into account the many and unpredictable variations in their listing. Less thorough approaches would be likely to bias rankings towards institutions such as Yale, with its unique name which also appears in the name of all its constituent schools and hospital, and away from institutions with a wider variety of permutations, abbreviations and possibly ambiguous versions of its name, such as the University of York (Univ York, U York, but not York University), with Hull-York Hospital (Hull-York Hosp), which were not always listed with the university name in the affiliation data.
We considered a number of possible approaches for analysing articles with more than one institutional affiliation. Both Wagstaff and Culyer (2012) and Rubin and Chang (2003) were constrained by the EconLit database, which only provides data on the first three or four authors, whereas we obtained institutional affiliation data for all authors. We considered assigning a fractional value (and even weighted fractional values reflecting author order) to each institution based on the number of different authors or institutions represented on a given article (Aksnes et al., 2012, Hagen, 2013, Retzer and Jurasinski, 2009). However, we rejected such approaches for two reasons: first, we believe that the use of zero-sum metrics establishes a perverse incentive against collaboration between institutions and against the crediting of collaborators. We therefore assigned one point per institution per article, regardless of the number of institutions or authors on a given article. This has the disadvantage of weighting the analysis towards articles from multiple institutions, as these articles are counted multiple times. cost-effective or "cost effective" or cost-effectiveness or "cost effectiveness" or cost-utility or "cost utility" or cost-benefit or "cost benefit" or "economic evaluation" or "cost per death" or "cost per case" or "cost per infection" or "cost per life" or "cost per disability adjusted" or "cost per quality adjusted" or "cost per disability-adjusted" or "cost per quality-adjusted" or "cost per qaly" or "cost per cost-effective or "cost effective" or cost-effectiveness or "cost effectiveness" or cost-utility or "cost utility" or cost-benefit or "cost benefit" or "economic evaluation" or "cost per death" or "cost per case" or "cost per infection" or "cost per life" or "cost per disability adjusted" or "cost per quality adjusted" or "cost per disability-adjusted" or "cost per quality-adjusted" or "cost per qaly" or "cost per daly" editorial or erratum or letter or note or report) 8. 6 not 7 9. limit 8 to (animals or animal studies) 10. limit 9 to humans 11. 9 not 10 12. 8 not 11 13. study protocol.ti. 14. 12 not 13 EconLit (Ovid SP) 1. ("2012" or "2013" or "2014").yr. 2. health.af. 3. (cost-effective* or cost-utility or cost-benefit or "economic evaluation").af. 4. ("cost-per-death-av*" or "cost-per-case-av*" or "cost-per-infection" or "cost-per-life" or "cost-per-disability-adjusted-life-year" or "cost-per-quality-adjusted-life-year" or "cost-per-qaly" or "cost-per-daly").af. 5. 3 or 4 6. 1 and 2 and 5 7. study protocol.ti. 8. limit 6 to (books or book reviews or collective volume articles or dissertations) 9. 6 not 8 10. limit 9 to working papers 11. 9 not 10 PsycInfo(Ovid SP) 1. ("2012" or "2013" or "2014" ("cost-per-death-av*" or "cost-per-case-av*" or "cost-per-infection" or "cost-per-life" or "cost-per-disability-adjusted-life-year" or "cost-per-quality-adjusted-life-year" or "cost-per-qaly" or "cost-per-daly").ti,ab,kw. 5. 2 or 3 or 4 6. 1 and 5 7. limit 6 to ("0200 book" or "0240 authored book" or "0280 edited book" or "0300 encyclopedia" or "0400 dissertation abstract" or "column/opinion" or "comment/reply" or dissertation or editorial or encyclopedia entry or "erratum/correction" or letter or obituary) 8. 6 not 7 9. limit 8 to animal 10. limit 9 to human 11. 9 not 10 12. 8 not 11 13. study protocol.ti. 14. 12 not 13 Global Health (Ovid SP) 3. cost-effective.ti. 4. ("cost-per-death-av*" or "cost-per-case-av*" or "cost-per-infection" or "cost-per-life" or "cost-per-disability-adjusted-life-year" or "cost-per-quality-adjusted-life-year" or "cost-per-qaly" or "cost-per-daly").af. 5. 2 or 3 or 4 6. 1 and 5 7. limit 6 to (annual report or annual report section or book or book chapter or bulletin or conference or conference proceedings or conference paper or correspondence or editorial or patent or thesis) 8. 6 not 7 9. study protocol.ti. 10. 8 not 9 Scopus (Scopus) MAIN SEARCH: ((((TITLE("cost-effective*" OR "cost-utility" OR "cost-benefit" OR "economic evaluation") AND SUBJAREA(mult OR agri OR bioc OR immu OR neur OR phar OR mult OR medi OR nurs OR vete OR dent OR heal OR mult OR arts OR busi OR deci OR econ OR psyc OR soci) AND PUBYEAR > 2011) OR (TITLE("cost per death" OR "cost per case" OR "cost per infection" OR "cost per life" OR "cost per disability-adjusted" OR "cost per quality-adjusted" OR "cost per qaly" OR "cost per daly") AND SUBJAREA(mult OR agri OR bioc OR immu OR neur OR phar OR mult OR medi OR nurs OR vete OR dent OR heal OR mult OR arts OR busi OR deci OR econ OR psyc OR soci) AND PUBYEAR > 2011) OR (ABS("cost-effectiveness" OR "cost-utility" OR "cost-benefit" OR "economic evaluation") AND SUBJAREA(mult OR agri OR bioc OR immu OR neur OR phar OR mult OR medi OR nurs OR vete OR dent OR heal OR mult OR arts OR busi OR deci OR econ OR psyc OR soci) AND PUBYEAR > 2011) OR (ABS("cost per death" OR "cost per case" OR "cost per infection" OR "cost per life" OR "cost per disability-adjusted" OR "cost per quality-adjusted" OR "cost per qaly" OR "cost per daly") AND SUBJAREA(mult OR agri OR bioc OR immu OR neur OR phar OR mult OR medi OR nurs OR vete OR dent OR heal OR mult OR arts OR busi OR deci OR econ OR psyc OR soci) AND PUBYEAR > 2011) OR (AUTHKEY("cost-effectiveness" OR "costutility" OR "cost-benefit" OR "economic evaluation") AND SUBJAREA(mult OR agri OR bioc OR immu OR neur OR phar OR mult OR medi OR nurs OR vete OR dent OR heal OR mult OR arts OR busi OR deci OR econ OR psyc OR soci) AND PUBYEAR > 2011)) AND (SUBJAREA(mult OR immu OR neur OR phar OR mult OR medi OR nurs OR dent OR heal OR deci OR econ OR psyc))) AND NOT (TITLE("study protocol"))) AND NOT ( 5,706 TI=("cost-effective*" or "cost-benefit" or "cost-utility" or "economic evaluation" or "cost per death" or "cost per case" or "cost per infection" or "cost per life" or "cost per disability-adjusted" or "cost per quality-adjusted" or "cost per qaly" or "cost per daly") # 3 13,274 TS=("cost-effectiveness" or "cost-benefit" or "cost-utility" or "economic evaluation" or "cost per death" or "cost per case" or "cost per infection" or "cost per life" or "cost per disability-adjusted" or "cost per quality-adjusted" or "cost per qaly" or "cost TI= ("cost-effective*" or "cost-benefit" or "cost-utility" or "economic evaluation" or "cost per death" or "cost per case" or "cost per infection" or "cost per life" or "cost per disability-adjusted" or "cost per quality-adjusted" or "cost per qaly" or "cost per daly") # 3 221 TS=("cost-effectiveness" or "cost-benefit" or "cost-utility" or "economic evaluation" or "cost per death" or "cost per case" or "cost per infection" or "cost per life" or "cost per disability-adjusted" or "cost per quality-adjusted" or "cost per qaly" or "cost per daly") # 4 227 #2 or #3 # 5 2 TI=("study protocol") # 6 227 #4 not #5 1,741 TI= ("cost-effective*" or "cost-benefit" or "cost-utility" or "economic evaluation" or "cost per death" or "cost per case" or "cost per infection" or "cost per life" or "cost per disability-adjusted" or "cost per quality-adjusted" or "cost per qaly" or "cost per daly") # 3 3,846 TS=("cost-effectiveness" or "cost-benefit" or "cost-utility" or "economic evaluation" or "cost per death" or "cost per case" or "cost per infection" or "cost per life" or "cost per disability-adjusted" or "cost per quality-adjusted" or "cost per qaly" or "cost per daly") # S1. TI ("cost-effective*" or "cost-benefit" or "cost-utility" or "economic evaluation" or "cost per death" or "cost per case" or "cost per infection" or "cost per life" or "cost per disability-adjusted" or "cost per quality-adjusted" or "cost per qaly" or "cost per daly") S2. AB ("cost-effectiveness" or "cost-benefit" or "cost-utility" or "economic evaluation" or "cost per death" or "cost per case" or "cost per infection" or "cost per life" or "cost per disability-adjusted" or "cost per quality-adjusted" or "cost per qaly" or "cost per daly") S3. S1 OR S2 S4.

Table S2 Mapping of 25 disease areas onto the Global Burden of Disease (GBD), International Classification of Disease (ICD-10), and search terms used
Health areas developed for this analysis are listed in alphabetical order in the lefthand column. We mapped each component of the Global Burden of Disease (World Health Organization., 2014) onto one health area. The mapping of the ICD-10 codes (World Health Organization., 2011) onto GBD codes is taken from the GBD appendices. Both GBD and ICD-10 definitions were used to inform the development of search terms for each health areas, which were applied as necessary to the titles, abstracts, and/or keywords in the final database of economic evaluations. Underscores ("_") have been used here to show single spaces and question marks ("?") reflect a single wildcard character. GBD: Global Burden of Disease. ICD-10: International Classification of Disease, version 10.

Table S4 Search terms to classify cost-utility and cost-benefit analyses
The following search terms were used to classify articles within our final database of full health economic evaluations according to study type. Searches were conducted in titles and abstracts. Search terms could classify an article as a cost-utility analysis, cost-benefit analysis, both, or neither. Articles in our database which did not contain search terms for cost-utility analyses or cost-benefit analyses were categorized as costeffectiveness analyses. Question marks ("?") represent a single wildcard character or space.

Table S9 Number and proportion of economic evaluations by type and income group
In this table, "cost-effectiveness analyses" refers to articles meeting our definition of a full economic evaluation but not containing any keywords to define it more specifically as a cost-utility or cost-benefit analysis. Articles can be classified as both cost-utility and cost-benefit analyses if they contain keywords for both. DALY: disability-adjusted life year, QALY: quality-adjusted life-year.