The IHME in the Shifting Landscape of Global Health Metrics
Abstract
The rise of the Institute for Health Metrics and Evaluation (IHME) has augured profound changes in the landscape of global health metrics. Primarily funded by the Bill and Melinda Gates Foundation, the IHME has offered donors a platform for assessing many health‐related Sustainable Development Goal (SDG) indicators and a toolkit to measure the progress of different countries. The IHME's increasing influence reveals the relative sidelining of international agencies and especially the World Health Organization which has long been central to global health metrics production. This shift reflects a growing conflict between the expertise and norms of national and intergovernmental statistical production on the one hand, and the distinct epistemologies and logics of new non‐state data actors. These transitions – from an international world of statistics to a more plural, global realm of data – have acute implications for the politics and accountability of knowledge production related to the SDGs and development writ large. Even as the SDGs embrace the rubric of ‘no one left behind’, the emerging data politics might be eroding the ability of poorer states to know and act upon their development problems on their own terms.
In the world of development, as indeed in other realms, measurement is never an innocent matter where as it were, the facts speak for themselves. What is measured, who finances and does the measuring, how data are collated, interpreted, and disbursed, how they are harnessed to decision‐making and program implementation, and how other measures and ways of collecting information are displaced – all these are contested matters because they are linked with the specific orientation of institutions and policies, the outcomes that they aspire to, and the forms of knowledge that they privilege. In many ways, this was the simple and profound insight that led Amartya Sen and Mahbub ul Haq to transform the understanding of development and poverty by the introduction of new measures as alternatives to GDP per capita and which led in the 1980s to the transformative HDI approach. This article details a significant shift in metrics that has been taking place in the world of international public health, and which has decisive implications for the Sustainable Development Goals (SDGs).
Global goals processes such as the SDGs and the Millennium Development Goals (MDGs) are, almost by design, heavily reliant upon quantitative metrics. They are anchored to measurable goals, targets, and indicators which are understood to be central to evaluating progress and ensuring accountability. Much social science commentary, in this special issue and beyond, has highlighted how the emphasis on quantitative metrics has narrowly conceptualized development and erased complex social and political processes.11
There is a growing literature on the limitations of quantitative global goals. For instance, on the MDGs, see Fukuda‐Parr (2014) and Fukuda‐Parr et al. (2014); on the politics of indicators as a form of governance, see Merry (2011) and Fukuda‐Parr and Hulme (2011); on global health metrics, see Adams (2016), Hickel (2016).
Other work, including under the label of ‘statactivism’, has countered that statistics have been used more variably and often have been effectively mobilized by social movements to address contentious inequities around health, gender, and poverty.22
See for instance, Bruno et al. ( 2014).
The emphasis of this article, however, is somewhat different. It focuses less on the perils or merits of quantification, and more on how the world of metrics for development is itself changing from within. There are new actors, new techniques, and new platforms of measurement that animate the contemporary ecology of global health. This shifting landscape of global health metrics has implications for the politics of knowledge production, which in turn undergird inequalities in development that the SDGs aim to address. In this, the article joins other work that takes global health and development metrics not as self‐evident facts but rather as the object of inquiry, examining the institutions and processes through which metrics are produced and gain credibility.33
Adams (2016), Desrosieres (2002); Lovell (2018).
The changes in what is measured and which metrics are deployed, while underway in many sectors, are the most starkly evident in the arena of global health. These shifts are vividly illustrated in the rise of the Institute for Health Metrics and Evaluation (IHME) at the University of Washington, Seattle, USA. Primarily funded by the Bill and Melinda Gates Foundation, the IHME and its signature product, the Global Burden of Disease (GBD) study, have become enormously influential in global health in a short period of time. The IHME is becoming the default source for a range of different estimates for global health, edging out older established actors that were central to measurement and modeling health and disease. This article specifically examines the rise of the IHME and its increasing dominance in the arena of health metrics. While the institute was not initially involved in the SDGs, the importance of the institute has only been heightened by the global goals’ exercise because it potentially satisfies the SDGs’ insatiable demand for measurement, audits, and forecasts.
The rise of the IHME challenges the long‐standing centrality of international agencies such as the World Health Organization (WHO) which has long been key to the production of global health estimations and comparisons. These global health estimations are germane to the work of donors and international policy makers that now increasingly use the IHME as their default source. The rise of the IHME is also accompanied by epistemological changes, that is, it is leading to new ways of knowing the world. Specifically, an older norm of government‐produced statistics is being substituted by newer, more plural, logics of data. International development has traditionally been grounded on government‐produced statistics about a country's population, be it in the realm of epidemiology, demography, income, or trade. While still heavily relying on state‐produced statistics, new actors like the IHME are challenging its traditional monopoly, calling for it to be complemented by more multifarious data that are produced by companies, universities, NGOs, and individual citizens. Moreover, the IHME's global estimation work is accompanied by a dazzling discourse of ‘big data’ and ‘complex computing’ that portrays older systems as inadequate.
These emerging transformations in the landscape of metrics production bring to the forefront questions of accountability and inequality. The data, the models, and the methodologies used by the IHME is often opaque to traditional development experts and publics. Indeed the IHME's global health estimations provide a remarkable juxtaposition of accessibility and opacity. The lack of transparency raises questions of accountability of powerful new actors. Moreover, the IHME showcases the infusion of resources in institutes in the North which are producing ‘global knowledge’. However, this is simultaneous to the capacity of older international institutions and state governments, especially of the poorest countries in the South, being depleted. The resultant asymmetry in the capacity for knowledge production has implications for development and policy making, with national actors increasingly having to rely on global estimates produced from afar.
To be sure, the IHME and its enormous resources are singular in many respects. It is arguably the most dramatic example of transformation in the world of development metrics. And yet, the impact of the IHME reflects overall trends – of sidelining of state governments and intergovernmental agencies, more plural epistemic practices, new logics of evaluation and accountability – that are manifest in many other sectors, albeit in heterogeneous ways. Thus, despite its uniqueness, the IHME and its transformation of the field of global health metrics serves as a lens through which to understand the evolving politics of measurement in the SDGs and the development arena writ large.
This article draws on interviews conducted over the last 3 years with officials at UNICEF, WHO and the World Bank. In addition, it draws upon conversations with officials in health‐oriented NGOs, National Statistical Offices (NSOs), philanthropic foundations, the IHME and global health journals. The article proceeds by first discussing the shifting institutional context within international health, with a decline of the WHO's autonomy, financing, and credibility juxtaposed against the rise of the IHME. It then traces the distinctive traits of the IHME's data work and especially its Global Burden of Disease (GBD) study. Initially commissioned by and produced under the auspices of the World Bank and the WHO, the GBD has now become the IHME's signature study. While enormously influential with international donors and increasingly responding to the demands of the SDGs, the IHME's GBD raises challenges of transparency and accountability. The concluding section discusses the implications for the SDG era due to the shifts in global health metrics brought on by the IHME.
Metrics have an increasingly outsized role in generating knowledge and analysis on questions of poverty, sickness, and sustainability. Global goals such as the MDGs and now the SDGs have only enhanced the epistemic and political authority of metrics.
And yet even as the MDGs and SDGs aim to ameliorate poverty, the differential capacities and resources of stakeholders might be exacerbating old inequalities. Ironically, even as the SDGs embrace the rubric of ‘no one left behind’, the emerging data politics might be eroding the ability of poorer states to know and act upon their development problems on their own terms.
Emerging Data Ecologies: Sidelining of WHO
In the post‐Second World War era, the WHO has been the designated United Nations agency for health. It has a paramount role as the norm and standard‐setting body in international health, and as a provider of technical and operational support to member states. Crucially for the purposes of this article, the WHO has also been the designated agency for collecting data from states and monitoring worldwide health trends. It compiles statistics from 194 individual member states in its Global Health Observatory data repositor, and provides analysis on a range of health‐related indicators.44
See www.who.int/about/en.
To be sure, other UN agencies, such as UNICEF, UNFPA, and UNAIDS, are also important players in monitoring global health. For instance, UNICEF has a crucial role in inputs into and evaluation of vital statistics. UNFPA has a lead role in monitoring sexual and reproductive health and rights. Both are lead agencies for SDG monitoring for specific health‐related targets. This article, however, focuses on the WHO because of its unique mandate within international institutions as the official repository of international health data and analysis.
The WHO, while remaining a crucial actor, has however seen its influence considerably eroded over the last two decades. Since the 1980s, member dues have been frozen. The United States and some other large donor countries have periodically withheld their assessed contributions to the UN, leading to a financial crisis in the organization. Indeed, the United States’ Trump administration's antipathy to pay its mandatory dues is consistent with a longer‐term trend of successive United States governments arguing to reduce the overall UN budget and their contribution to it. Caught in the crosshairs of Cold War politics, the WHO has been hit particularly hard by these financial cuts. Its central budget has shrunk over the last three decades, accompanied by staff layoffs and shutdown of crucial programs.55
For an excellent account of the history of the WHO and especially its challenges during the Cold War years, see Chorev (2012);. See also Clinton and Sridhar (2017).
While the WHO's central budget has shrunk, there has been a ballooning of voluntary contributions, or extra budgetary funds (EBFs). These voluntary contributions from donor countries and philanthropic foundations are earmarked for special initiatives deemed important by the donor. In the early 1970s, EBFs formed about 20 per cent of the total WHO expenditure; by the late 1980s, EBFs accounted for about 50 per cent of total WHO expenditure; by 2004, the number was up to 70 per cent; the share of EBFs approached 80 per cent in 2016 (Clinton and Sridhar 2017). The voluntary contributions from donor countries and private philanthropies, on one level, have been a lifeline for the WHO given the precipitous drop in mandatory contributions. But these voluntary contributions have come at a steep cost as the WHO's focus has been increasingly shaped by large donor countries’ and philanthropists’ funding preferences. The World Health Assembly, ‘the supreme decision‐making body for the WHO’, which provides equal votes to all countries, has been bypassed and its procedural autonomy considerably eroded (WHO, 2018). The WHO Secretariat, which comprises the administrative and technical personnel of the organization, is increasingly beset with conditions set forth by donors (Chorev, 2012).
The WHO's position has also been weakened by the entry of a host of new actors in international health. The World Bank emerged as a major actor in the health sector in the 1980s in a broader context of prevailing neoliberal economic ideologies. By the 1990s, World Bank loans for health‐related programs exceeded the entire WHO budget. Moreover, the Bank ushered in a style of economic reasoning that was oriented toward cost effectiveness and was often at odds with the WHO's emphases on universal primary health, equity and human rights. More recently, the entry of new actors including private organizations such as the Gates Foundation and public‐private partnerships including the Global Fund for Fighting Malaria, AIDS and Tuberculosis, and GAVI, the vaccine alliance, have made global health a crowded field. As a result, the WHO, previously the central pivot, has become just one among many organizations in global health.66
On institutional shifts in the field of international health, see Adams (2016), Brown and Cueto (2006), Lakoff (2017), Mahajan (2017).
Over the years, with donors earmarking most of the funds for special projects, many of WHO's fundamental functions have been neglected (Clinton and Sridhar, 2017). The decline in its pandemic preparedness functions was widely reported during the Ebola epidemic (Fink, 2014). Less visible has been the erosion in the WHO's capacities for data collection, monitoring, and modeling. As noted earlier, the WHO provides a consolidated repository of health indicators through its Global Health Observatory. The observatory operates a health‐related statistical database which publishes a compendium, World Health Statistics, that covers 1,000 indicators for 194 countries. However, the repository has been criticized as suffering from systemic data gaps; the information regarding its different topical areas of work varies widely; critics claim that many of the numbers are determined by processes vulnerable to interstate politics and are not peer‐reviewed; its reporting, its models, and norms of building its database are not transparent to those outside the institution.77
See for example, Bianchi and Peters (2013), Sridhar and Clinton (2017), Sridhar and Gostin (2011).
The WHO still retains a crucial position in the global health world. It still has moral authority as a norm setter. Its health indicators and estimates are still used as the touchstone in many state‐based and international reports. Nonetheless, the last few decades have left the WHO significantly embattled. It has been subjected to frequent onslaughts of criticism about its bureaucratic functioning, its slow response to emergencies, and the lack of reliability of its metrics. In many ways, the criticisms have served as a self‐fulfilling prophecy – they have led to decline in financial support for the organization's central budget, which has hampered its effective functioning, which then has increased denunciations and become further grounds for withholding support. The rise of the IHME and accompanying shifts in data politics have to be understood in this broader context of the diminished epistemic and political authority of the WHO over the last few decades.
The Rise of the IHME and its GBD
The Institute of Health Metrics and Evaluation was started in 2007 under the leadership of the health economist Christopher Murray who was invited by Bill Gates to build a metrics‐focused initiative at the University of Washington, Seattle, USA. In their recent book on global health governance, Sridhar and Clinton have suggested that IHME's inception can be understood as emerging from the Gates Foundation's ‘lack of confidence’ in the WHO's statistics (Clinton and Sridhar, 2017, p. 36). The IHME defines its mandate as work on ‘quantitative analysis health metrics science’ which provide ‘roadmaps to policy makers and donors’ (Murray, 2017). The Gates Foundation provided an initial grant of $105 million dollars for the institute while the University of Washington added another $20 million dollars (Bill and Melinda Gates Foundation, 2007). In January 2017, the Gates Foundation provided an additional grant of $279 million for expanding IHME's work over the next decade (Butler, 2017). While the Gates Foundation has been its biggest financier, the IHME has also received significant support from the University of Washington, the United States federal government, and the Washington state government, and private sources. The IHME is a strikingly well‐resourced organization in an international development landscape where large funds are seldom targeted exclusively for the production of metrics.
The IHME's signature product is the Global Burden of Disease study (GBD) which is an enormous endeavor undertaken in epidemiological modeling. Though now strongly associated with the IHME, the GBD predates the IHME by almost two decades. The first GBD study was initiated in 1991 by Christopher Murray and Alan Lopez under the auspices of the WHO and the World Bank. Its results were prominently featured in the World Bank's iconic 1993 World Development Report and then in the WHO Bulletin in 1994. Christopher Murray moved for a few years to the WHO, which became the home of the GBD, but he soon left the WHO to return to Harvard University. After Murray's exit, the WHO continued to release the GBD index every few years. However, the GBD returned to the center stage of global health metrics only when the IHME at the University of Washington started publishing its own GBD studies that offered direct competition to the WHO's metrics.88
The GBD origin story has been frequently narrated and I provide only the barebones here. For genealogy and controversies surrounding the GBD, see Williams (1999) and Smith (2017). The GBD is based on the DALY measure which is used to link health to economic models by quantifying the overall disease burden as number of years lost to ill health. For discussion of the strengths and weaknesses of DALYs, see Anand and Hanson (1997) and Arensen and Nord (1999); .
The GBD aims to provide a ‘health audit of the world’. It collects morbidity and mortality data, runs it through models, which then produce estimates on ‘the magnitude of health loss from diseases, injuries, and risk by age, sex, and population over time’ (Murray and Lopez, 2017). The IHME's study presents estimates of 291 diseases and injuries across 187 countries. Recently, it has also started providing estimates on patterns of health funding and donor aid to different countries. Through its cross‐country estimations and comparisons, it aims to provide international policy makers, donors, and program managers with health trends that can be used as the basis for decision‐making and resource allocation. The IHME's first GBD results were published in 2012. Since 2015, the IHME has published GBD updates every year.
The IHME publishes its GBD findings in the British journal Lancet which has become a key platform for the study's authorization and dissemination. The GBD estimates are also available on the IHME's website, www.healthdata.org. The website presents itself as an independent and neutral platform for health data and scientific studies, a one‐stop portal for health‐related metrics. It provides a dazzling interface with a slew of data visualization tools, education modules, and interactive interfaces that produce spectacular charts, maps, and tables. Various tabs on the website enable comparisons between different countries, regions, disease groups, and time periods. The site has a frequently updated menu of studies. In many ways, its interface is an elaborate invitation to the reader to explore the world of health metrics. The IHME's accessible interface, with its teaching modules, maps, and graphs, stands in contrast to the WHO's online data repository which is marked by varying methodologies and gaps for different countries, and updated episodically. ‘Getting comparable estimates of health trends from the WHO website is far more challenging’, pointed out a World Bank official, noting that very often pages of the WHO global observatory simply do not load.
Decades ago, the organisations responsible for data collection and analysis would be the statistical authorities in each country. National reporting would come from governments; international reporting would come from the UN. Now, however, the world has become more pluralistic. Although governments remain one of the most important actors in data collection, for many outcomes, such as ambient air pollution monitored by satellite, the dominant sources are likely to be remote or on‐the‐ground data collection from non‐governmental sources. As analytical techniques to produce robust data synthesis have become better but at the same time more computationally intensive, the role of organisations outside of government and UN agencies has grown and the UN no longer has a monopoly on international data reporting.(Murray, 2015, p. 1316)
This approach, where plural understandings of data are used to complement state‐produced statistics, stands in contrast with the WHO's traditional reliance on national statistics. As will be discussed later, this move to data plurality has repercussions on how to create representations of a state population, and who gets to be the credible authority on creating such representations.
One example of the size of the project can be represented with the cause of death database that we have used … we have included almost 800 million deaths from 1950 to 2010, and the data come from different sources. The goal was to incorporate ‘all the available data’… this is the biggest database for cause of death analysis in the world … Running the programs to map the data to our cause list of 291 causes and correcting the bias can take days, even using a powerful cluster of more than 100 computers. The data that we have to store after the modelling process can take 3 terabytes.(Das and Samarasekera, 2012, p. 2067)
Underlying this discourse of big data is an implicit view that more data are better for producing good estimates of health trends. Like the move to data plurality, this has proven to be a contentious assumption.
The GBD now tracks more than 1,000 health indicators for 188 countries covering 25 years, which are in turn double‐checked against 20–40 statistical models. The team also runs each model 1,000 times … The computations are run on the IHME's supercomputer, where 12,000 high performance processing cores churn away at the maths for 4 days to complete a single snapshot for the planet mortality rate for each age‐sex group as a function of SDI.(Murray, 2016, p. 40)
The IHME presents its modeling and computing prowess as its unique strength. In conversations, the IHME officials state that while the role of countries is to provide data, the IHME's contribution is in analyzing these data to produce trends and comparisons. The IHME underlines the methodological improvements enabled by its sophisticated models and computing power; unlike the WHO, its modeling is used to deploy a standardized methodology for all countries; it innovates to compensate for missing or poor data, often using methods from fields outside of epidemiology, such as computer science, engineering, weather science, and economics. The IHME emphasizes that its innovative models allow it to generate estimates that can then be used for comparability across regions and time periods.
The fact that IHME's data are run through models is in itself not innovative; after all global health estimates produced by the WHO and other international agencies are also invariably products of models. However, by all accounts, the degree of complexity and the scale of computing in the IHME models surpasses what is used by the WHO, other UN agencies, or the World Bank. With a staff of more than 300, the IHME employs a veritable army of computer scientists, mathematicians, epidemiologists, and economists devoted to the work of health metrics production. To put this in perspective, a UNICEF official remarked that her organization has about 10 people in the unit devoted to modeling and health metrics production. The WHO has about thirty core staff focused on modeling and metrics production, with a few more experts spread out through country offices (interviews).
The IHME's core staff does not even fully represent the full scale of its expertise. In its 2012 GBD publication, the IHME stated that it had worked with 420 collaborators. By 2015, its network had increased to over 1,000 collaborators. In his 2017 Director's Statement, Christopher Murray stated that the IHME had built a network ‘exceeding 2000 collaborators’. (Murray, 2017). The expanding network of partnerships and collaborations has become central to the IHME's discourse; it is featured in its publications, website, and conference presentations. These collaborations allow IHME to access expertise and perspectives from different countries. The relationships are also very attractive for the partners. As an IHME collaborator in South Africa explained to me, ‘Our partnership with IHME ensures frequent publications in top journals. It provides us access to global research networks and conferences. It facilitates access to training in the latest computing technologies … it also brings much needed funding to our group (which was upwards of two hundred thousand dollars for this particular research unit in South Africa)’. For groups based in resource strapped universities in the South, it is difficult to resist this package. The IHME and the South African research group had forged what anthropologist Johanna Crane has referred to as unequal but highly valuable partnership (Crane, 2010).
The IHME's work is also marked by its growing investment in education and its attempts to train students and technocrats. To this end, the IHME provides online courses and technical training workshops that introduce the GBD. More recently, it has started a PhD program that offers attractive 5‐year doctoral fellowships in order to attract the best students from around the world to the IHME. In effect, the institute aims to train the next generation of global health specialists in how to think of, generate and use health metrics.
Through its seductive web interface, Lancet publications, tightly controlled messaging in public conferences, discourse of big data, big computing, and thousands of collaborations, the IHME effectively deploys what the sociologist of science Stephen Hilgartner (2000) has called strategic ‘stagecraft’ in presenting an authoritative picture of global health. The enormous backstage work that is done to make disparate data sets commensurate, coordination with large number of interdisciplinary expert teams and partners, and management of large data gaps, is seldom visible to the public. Rather, there is a confident ‘front staging’ of certainty (and uncertainty which is defined in circumscribed quantitative terms), keeping much of the messiness to the backstage (Hilgartner, 2000).
Through its cumulative activities of collection of variegated data, complex modeling, far flung collaborations, and investments in education and training, the IHME has in effect produced what I call elsewhere a ‘data world’. This data world, animated by infrastructures of standardization, social relationships across expert groups, and styles of reasoning, reflects a ‘commitment to seeing, understanding and representing questions of health and well‐being through metrics’ (Mahajan, 2018). Moreover, as evident in the IHME's involvement in SDGs, the GBD's data world is a way not only of knowing but also acting upon questions of health and disease.
The SDGs and the GBD
The IHME was not very involved in the initial SDG negotiations. The views of its officials often echoed those of the Gates Foundation and some OECD governments that had wanted the SDGs to mimic the MDGs, which would have enabled relatively easy measurement of a small number of discrete goals, targets, and indicators (Buse and Hawkes 2014). By 2015, however, it became clear that the SDGs were going to be far more variegated and numerous than the MDGs. As expert committees were debating the choice of SDG indicators, Christopher Murray published an article in the Lancet that made a series of suggestions about health indicators appropriate for the SDGs. Many of the suggested indicators were already being measured by the IHME's GBD. Murray emphasized that since the IHME's GBD already did a lot of work on health indicators, it provided a ready‐made platform for evaluating SDGs, a task that could no longer be designated exclusively to the United Nations and state governments:
The IHME followed this up with a study in October 2016 in the Lancet which used the GBD as a platform for assessing health‐related SDGs. It examined 188 countries and how they were performing on 33 of the 47 SDG health indicators between 1990 and 2015. The study included SDG indicators such as maternal mortality and infant mortality but did not include indicators that examined social determinants of health, nor rights to health. The study produced a common scoring metric that facilitated comparisons between countries. In addition, it generated a summary health‐related SDG index, which provided a composite measure of progress achieved by each country on all 33 health indicators. Finally, it produced a sociodemographic index to judge to what extent a country's performance varied from what might have been expected. The IHME used the 2015 edition of the GBD data to produce its assessments. This work was aimed at providing ‘a baseline for all the health‐related indicators for which we have data. That baseline allows us to know where the world is and where each country is with regard to development goals and that knowledge is critical component of accountability’ (GBD 2015 SDG Collaborators 2016). The 2016 report was followed by an updated SDG study published in September 2017 in the Lancet (GBD 2016 SDG Collaborators 2017). The IHME also provided an extensive analysis of SDG health‐related indicators on its website. Like much of the IHME web interface, the SDG tab invites readers in and allows them to investigate an array of questions; it offers data visualization tools that produce beautiful graphs and maps; it provides estimates about a large number of SDG indicators in one place.
By providing this readymade data analytics center, the IHME addressed what has been one of the biggest concerns around the SDGs, namely the cost and capacity required to collect the data and do the work of evaluation for dozens of indicators. The IHME can and does claim that much of this work is already done, and at no additional cost to individual countries or international health organizations.
Moreover, the IHME underlined that it provided an ‘independent’ platform for measuring the progress that different countries were making toward achieving the SDGs. For the IHME, its independence was grounded in its not having any government or UN affiliation. As I have already indicated, IHME receives substantial funding and support from the Gates Foundation. But for IHME, its close link to the Gates Foundation does not in any way impugn its purported independence. Similarly, controversies, that will be detailed in the next section, have referred to the lack of transparency around IHME's data sources and the opacity of its complex models. These controversies revealed how IHME's metrics operate in a contested terrain, with the GBD representing one particular way of framing health burden which has ideological and epistemic commitments that are oriented toward standardizing and representing data in a way so as to facilitate what might be thought of as a global episteme. These commitments, which are built into the technical and conceptual infrastructure of IHME's GBD, are not acknowledged as being in their very nature political. Instead IHME defined its independence solely on account of it being free of the superintendence of governments and the United Nations.
It is not that the IHME's entry into the SDG terrain has displaced the WHO and other UN agencies, many of which have been designated ‘custodian agencies’ for the global compilation of data for SDG targets and indicators. The SDG monitoring process has required that these international agencies work closely with individual governments and provide direct technical assistance in addressing questions of what gets measured and how. The IHME is not involved in these processes within the SDG monitoring framework, which in turn has limited its influence. Indeed governments and National Statistical Offices (NSOs) are frequently resistant to modeling techniques that are not sufficiently transparent and data collected from outside their purview. They have far closer relationships with international agencies such as the WHO and UNICEF, rather than the IHME. However, while not always influential with governments and their NSOs, the IHME has been embraced and applauded by donors and particular agencies such as the World Bank. They applauded the IHME publications on the SDGs as a ‘landmark event’ and celebrated the evaluation tools and ease of access of SDG indicators on the IHME website as key to ‘holding governments accountable’ (Maurice, 2016). The salience of the IHME was underlined with the Chief Economist of the World Bank stating that the Bank would be using the health estimates from the IHME for its work on monitoring human capital. For these donors, the IHME's metrics made SDG monitoring and accountability of states a realizable goal. Moreover, in interviews, academics, and officials at agencies such as the UNICEF noted that they turned to the IHME numbers because there are often no alternatives.
Challenges of Transparency and Data Gaps
The reception to the IHME's work on the SDGs and the GBD more broadly, however, has by no means been unanimously celebratory. In the early years of the IHME's GBD, there were controversies regarding the significant difference between estimates provided by the IHME and the WHO around particular trends such as for maternal mortality, deaths due to malaria, and under‐5 mortality.99
See Murray et al. (2012) and WHO (2011), where the IHME's estimate of annual deaths due to malaria was almost twice as large as the WHO's estimate. Also see Cohen (2012) Victora and Boerma (2018), and Alkema and You (2012).
Critics pointed out that these differences were generally due to poor data from the ground. They further noted that the IHME source data were poorly described, not publicly shared, and therefore difficult to validate (Byass et al., 2013).
These initial controversies brought to the fore a larger concern with data gaps. Margaret Chan (2012, p. 2054), the former director general of the WHO, underlined the issue: ‘The real need is to close the data gaps, especially in low‐income and middle income countries, so that we no longer have to rely heavily on statistical modeling for data on disease burden’. She and others noted how data are typically missing from poor countries which have weak statistical systems. The total quantity of data used by IHME models occludes the fact that most of these data are from wealthy countries with large systemic silences from poorer parts of the world. When good data are not available from poor countries, the models are animated by proxies that are often deduced from trends in neighboring countries or middle income and high income countries. The systematic absence of good data from poor countries raises the issue of whether the estimates produced by models are accurate and appropriately representative of the phenomenon being discussed (Hickel, 2016; Mahajan, 2008). Indeed, several cases pointed to how sophisticated modeling could not compensate for poor data from the ground. And while the IHME has invested in building modeling capacities, it has not contributed to building statistical infrastructure in poor countries.
In fact, the IHME might be exacerbating the problem of data gaps from poor countries because of its move away from an exclusive reliance on government statistical data and toward collection of data from private sources. Anecdotes from officials in private hospitals and insurance companies in countries such as Nepal and South Africa revealed that an older tradition of sharing mortality and morbidity data with the government was occasionally overshadowed by a priority of sharing the data with the IHME. Officials at national statistical offices (NSOs) noted how the IHME's collection of piecemeal data from a slew of private and public sources undermined their credibility and central coordinating role in the national statistical system. NSOs have been the traditional purveyors of country data for the UN and the WHO. These offices are often understaffed and underfunded, especially in poor countries. They work under political pressure, and are often low in government hierarchy, with little ability to set agendas. The statistics they produce often have gaps and can be vulnerable to political vicissitudes. Indeed, the need for reforms in many NSOs is crucial. However, despite their problems, it is not clear that there is a credible alternative to the NSOs for they ultimately have a legal mandate from governments to create statistical representations of the country's population. However incomplete, at least in theory, their data aims to provide a complete picture of its people. The IHME's work may very well be compromising the long term viability of the infrastructure needed for statistical production. Bypassing the NSOs might, in the longer term, more deeply entrench the data silences from poorer parts of the world.
Despite the IHME's discourse of big data driving the GBD analysis, it is not always clear what these data are and how the IHME collates it. Much of the data that are used are not made publicly available; indeed this has been a source of consistent critique of the IHME (Byass). A member of the Lancet's editorial board commented on how sometimes experts were reluctant to review the GBD because they were not given access to the primary data. IHME officials explain that the data are often protected by memoranda of understanding (MOUs) that they have signed with private parties. Over the years, the IHME has taken steps toward increasing its transparency by complying with recently formulated international guidelines for health estimates reporting.1010
The IHME complies with the Guidelines for Accurate and Transparent Health Estimates Reporting (GATHER) which was collaboratively developed by the IHME, WHO, experts from universities such as the Harvard and Johns Hopkins Schools of Public Health, and two medical journals, The Lancet and PLOS Medicine. www.Gather-statement.org.
Nonetheless, it has far from satisfied critics who have pointed out that it is not much help to point to the enormity of the data used if these data are not open to scrutiny. The data, the assumptions, and the proxies that any model is built upon require transparency and scrutiny. Indeed it is this transparency that forms the basis of accountability and trust in metrics. The situation is not without irony: the IHME has often criticized the WHO and other UN agencies for their incomplete and opaque data (AbouZahr, 2011; Gibbs, 2016). However, the institute's own data, while awe‐inspiring in scale, are not necessarily presented any more transparently.
Concerns of transparency have also been raised about IHME's computing and modeling. The IHME's technological sophistication is typically presented as a self‐evident good that facilitates progress toward greater scientific achievement. However, the technological complexity has meant less transparency and accessibility, especially to experts within NSOs who have been at the forefront of collecting national data. ‘This is a fast movie in front of our eyes. We know we cannot be actors in this movie’, admitted the director of an NSO of an African country. Within the SDG monitoring process, NSOs have frequently resisted modeling techniques such as of the IHME that are not sufficiently transparent. They have demanded a greater role in the compilation of SDG rankings and metrics. One solution to this problem – which is partly being pursued by the IHME – is to train NSO experts and build a new generation of data analysts more familiar with complex modeling. While salutary in some ways, this raises some questions: it is not clear whether this is the most needed and appropriate technology for NSOs; whether the training is an option for the weakest NSOs; and whether this training will bolster or render obsolete existing national statistical capacities. After all, many NSOs receive paltry resources from their government and even less from international donors.1111
An important exception is the World Bank's Trust Fund and Global Financing Facility which is committed to build statistical capacity and supporting civil registration and vital statistics in selected countries.
For the weakest of NSOs, the unleashing of technology in statistics at a global level portends their growing irrelevance and disempowerment. The potentially troubling implications here are that national systems are being urged to catch up with a global technological norm with its distinctive metrics and standards; and yet these very metrics and technical ‘advances’ are expressly designed with an eye to global comparison, and not specifically to meet national needs.
IHME's computational work is opaque not only to national statisticians but also to traditional global health experts. The institute's significant investment in its computational machinery directs monetary and technical resources into analyzing health trends. But the discourse and actuality of big data and computing expertise also has the effect of preempting critical engagement. ‘It has an overwhelming effect’, admitted a WHO official. ‘The IHME throws these big numbers and computing figures at you … the response to critical comments is that they have bigger and better data … that they have improved computing methods … it shuts down the conversation pretty quickly’. This is confirmed by the fact that the initial controversies around the IHME's GBD were soon muted as many in the WHO and UNICEF admitted that it was difficult for them to compete with the IHME's massive resources. IHME and GBD have been clearly successful in normalizing and establishing new benchmarks in global health metrics.
The IHME's data and computing prowess has also imposed unintended constraints on peer review. At two separate meetings on global health, editors commented on the challenges they faced in finding reviewers who were both independent and expert enough to fully understand the intricacies of the IHME's modeling programs. They admitted that even their editorial council members had a tough time understanding the complex modeling. ‘The level of certification and legitimation we can provide is probably not as high as readers expect and presume’, admitted an editor.
The challenges of finding independent peer reviewers is exacerbated by the IHME's range of collaborators. Many of the key academic experts in a position to technically evaluate the GBD are increasingly part of the IHME's partnership network. ‘Our strongest reviews are from collaborators’, admitted an IHME director. Indeed, the IHME deflects criticisms about its patchy data from poor parts of the world by pointing to its vast network of collaborators. This large partnership network, while clearly valuable in checking assumptions that go into models, also means that the collaborators are likely to express their opinions in private interactions rather than in public debate or critical peer reviews. This is of course not a problem unique to the GBD and the IHME. Studies in sociology of science have long noted that in many fields marked by high levels of specialization, peer review is often difficult because different research groups are also often collaborators.1212
For instance, see Pickering (1984) for an account of how credibility and review are negotiated in the field of high energy physics where many of the research groups are also in collaboration.
And yet, few fields have the direct global policy impact of the GBD. The difficulties of getting expert reviews of the GBD from outside the IHME's ambit stands out as a matter of concern because of the study's potentially large impact on decision‐making resource allocations by governments and international health institutions.
As noted earlier, when the IHME's first GBD was published in 2012, there were controversies around differences between the IHME and WHO metrics. Around this time, in interviews, officials within the United Nations would raise not only technical but also larger political questions about the study. However, what became striking was that their concerns, while commonly voiced in private meetings, would be more muted in open public settings. In conversations, officials in UNICEF and WHO demurred from publishing their critiques because of ‘conflicts of interest’. Their organizations were receiving crucial funds from the Gates Foundation for key programs. As an official within UNICEF had explained, ‘We are receiving millions of dollars for our polio campaign in Afghanistan and Pakistan from the Gates Foundation. We cannot jeopardize that campaign. Publicly criticizing the work of the IHME could potentially alienate the Gates Foundation’ (interview).
Of course the IHME and the Gates Foundation are distinct organizations, and their messages and mandates are not always aligned. The Gates Foundation strongly supports and funds the IHME but it also has various other projects in myriad global health institutions and a broad mandate that exceeds the IHME's work. UNICEF's offering critiques of the 2012 GBD may very well not have led to it alienating the Gates Foundation. However, what was clear from interviews was the perception that criticizing the IHME's GBD could jeopardize the Gates Foundation's support for UNICEF's programs. There are substantial circumstantial reasons to think that this has led to an emerging culture of self‐censorship within international health organizations, including among those experts who were perhaps best placed to understand and critically assess the IHME's work.
Claims of self‐censorship are always difficult to firmly establish because they relate to that hazy domain of conscious and unconscious motivations. However, what cannot be denied is the growing clout of the Gates Foundation in the broad arena of global health.
The Gates Foundation provides funds to almost all the large international health organizations. It is the second largest funder of voluntary contributions to the WHO; with a recent US$80 million grant, it became the largest non‐state donor to UN Women; it is the third or fourth largest donor to UNICEF; and the largest non‐state donor to both the Global Fund and GAVI.1313
For the WHO, see who.int, ‘Source and Distribution of Funds Available/Contributors/BillandMelindaGatesFoundation’. For UN Women, see unwomen.org under the tab ‘Partnerships/Business and Philanthropies” which notes the Gates Foundation grant of US$80 million in 2017. The tab ‘Partnerships/Government Contributors/Top Contributions” notes Sweden as the top government donor in 2017 with a contribution of US$34 million.
It provides funds for global health projects to the World Bank and other philanthropic foundations such as the Ford Foundation and the Rockefeller Foundation. The scale and distribution of its grants have given it a central role in setting global health agendas.1414
Many articles in popular media and academic journals discuss the vast influence of the Gates Foundation in contemporary global health. Some examples include Birn (2014), Harman (2016); McCoy and McGoey (2011); and Kluger (2015).
The range of the Gates Foundation's funding reflects its considerable resources and its commitment to working on global health; however, its widespread influence has also had the unintended consequence of constraining open debate around crucial studies such as the IHME's GBD.
Conclusions
Over a short period of a decade, the IHME's GBD has gone from being a contested producer of global health metrics to becoming a default source for donors, academics and some international agencies. It is true that many governments rely on data and trends provided by their ministries and NSOs rather than estimates from IHME's GBD models. Similarly, for many within the UN system, WHO estimates still provide the final word. And yet, there is no denying that the IHME is a growing force in global health estimation, with long term implications for international agencies such as the WHO that did much of the same work.
Precisely because the IHME's metrics are now an important way of knowing and acting upon the world of health, it is imperative that they be accountable and that their innards be transparent – at a minimum to a range of experts and more maximally to all stakeholders. After all, metrics are frequently invoked as the basis of holding governments accountable; surely that claim comes with the imperative that these numbers themselves be accountable to a broad epistemic community. I want to use these concluding remarks to reflect on how the IHME case can help us think about questions of accountability in the realm of global metrics production.1515
Recent work on accountability in the context of data, health and the SDGs includes Williams and Hun (2017). This article conceptualizes accountability in a human rights framework with a focus on whether the SDGs are fulfilling the right to health. On a different register, Reubi (forthcoming) offers a concept of ‘epidemiological accountability’. He argues that the work of philanthropies in global health has introduced a new kind of accountability that combines the expertise of audit and epidemiology. Both these conceptualizations are complementary to my work; however, this article focuses on how a sociology of knowledge production, instantiated through the institutional and technological features of IHME, can facilitate or hinder accountability.
There are two broad questions that need to be articulated with regard to accountability: first, who are the experts and publics who should hold metrics and their purveyors accountable? And second, to what ends is this accountability important? These are clearly vast questions toward which this article can provide only abridged comments. However, these questions, at a minimum, interrupt the all too familiar perspective in which the proliferation of data and metrics are assumed to be self‐evident goods. The questions about accountability caution against the view in which an imagined global citizenry automatically celebrates metrics and data as facilitating the march toward better lives.
I want to suggest that the accountability of metrics to experts can be conceptualized at two levels: first is a proximate accountability that entails evaluation by domain experts who have the ability to determine the technical quality of the numbers and the methodology employed. These domain experts ascertain whether the metrics are accurate in representing the relevant phenomenon, and if the methodology is appropriate and ethical.1616
What counts as the relevant domain of course is not always self‐evident, especially in rapidly changing and interdisciplinary fields. See Chubin and Hackett (1990) on challenges of different forms of peer review.
Peer review would be an example of proximate accountability. In the case of global health metrics produced by IHME, the relevant domain experts would be epidemiologists, computer scientists, and others who can assess the science and mechanics of IHME's models and data.
A second level of accountability includes an assessment by a wider ranging realm of expertise that can weigh in on the broader work that the metrics are doing. These experts may not be conversant with the technical complexities of metrics production, and yet be in a significant position to assess the political, social, and epistemological origins and impacts of metrics. Such commentary is necessary to analyze if the presented global health metrics are answering the most relevant questions; how they are shaping knowledge about poverty and development; how they are affecting institutions and shaping policy making. Development metrics, after all, don't stay in data labs. They are intended to travel into diverse policy, social, and political realms. Correspondingly, it is crucial that the relevant experts that hold metrics accountable not be confined to specialized domain experts but extend to a broader community invested and interested in development. For the IHME work, this broader expertise would include historians, economists, and sociologists of global health, development policy specialists, and government and international agency technocrats who work with the metrics.
The IHME case reveals the challenges of achieving accountability at both these levels. On the one hand, the IHME metrics are evaluated by domain experts when the Lancet conducts peer review. In addition, IHME's many collaborators provide an internal review mechanism. And yet, there are clear indications that these reviews are constrained as it is difficult for journals to find peer reviewers who are both independent and expert enough. The technological complexity of the IHME's modeling and the shroud of private MOUs around data pose hurdles. The institute's vast network of collaborators makes the availability of independent reviewers even more challenging. Moreover, the depleted capacities of international agencies such as WHO and UNICEF has meant that alternative health metrics are increasingly sparse. Moreover, these agencies’ dependence on Gates Foundation funds muffles their willingness to openly critique IHME metrics.
To be sure, it is not that international health metrics generated by the WHO and other UN agencies were and are not plagued with problems of transparency and accountability. As discussed earlier in the article, the WHO metrics have long been criticized for not being transparent nor being subject to formal peer review. However, while the IHME has brought with it enormous technical competence, it is not apparent how its workings are significantly more accountable than the agencies it heavily criticizes. Indeed, the IHME offers a somewhat paradoxical picture of incredible access to estimates through its website, but it is an accessibility that sits alongside considerable opacity.
The IHME's accountability to a broader realm of development experts is also relatively elusive. While significant monies have been invested in building a computing juggernaut in the IHME, few new resources have been directed toward supporting social science experts who can study social and political meanings of the metrics. In the current context, a broader critique and accountability of global health metrics is situated only at the margins. But investing in the social studies of global metrics is crucial, not least because it might provide historical lessons from similar exercises conducted in the past, and warn against unintended consequences.1717
Adams (2016) makes a similar point when she emphasizes the role of ‘stories’ and ethnographic accounts to accompany global metrics.
For instance, one such unintended consequence that might be playing out is the weakening of already depleted national statistical capacities.
Officials at the IHME have stated that the work of building individual countries’ statistical capacities is not their mandate. After all, strengthening national statistical systems is expensive and long‐term work. As Murray stated in an interview in Nature, ‘the world needs to make local, national, regional, and global decisions before this happens’ (Butler, 2017). For the IHME, their work is to enable this decision‐making, even in the absence of good national data, by modeling, mapping, and creating comparisons across countries and time periods.
In this rendering, the national context is seen as providing raw material for the value‐added work that takes place in northern institutions that have a global orientation. This division of labor, of raw material from the poorer countries infused with a value‐added scientific respectability in Seattle and other similar sites, recalls the hierarchies and epistemological paternalism of an imperial era – even if the discursive packaging now studiously avoids any such reference. Perhaps the rapid growth of scientific knowledge and technological change simply cannot avoid such awkward antinomies even in a putatively democratic age. Still it must give us pause and at a minimum make us vigilant. There is after all, a long history to development in which knowledge was an instrument of power that legitimated itself by claiming to have a beneficent concern for those over whom it exercised power. These are not reasons for denying the beneficial effects of science and knowledge; but they are grounds for a vigilance that attends to the importance of accountability and transparency, and where these demands cannot simply be satisfied through a recursive self‐validation.
The IHME is undoubtedly producing important science. But it is also true that it is facilitating a particular kind of knowledge production, which is presented as global knowledge. Here the ‘global’ has particular contours: it is typically produced in Seattle – or other metropolises of the North; it claims to be independent of social and political context and commentary, which allows it to travel easily across the world; it satisfies the needs of donors and global planners who need to make cross‐country comparisons about cost‐effective interventions. The global in this case, encompasses systematic silences of data from poorer parts of the world. Because of the paucity of data from poor countries, the global knowledge may be of limited value in planning and decision‐making in the South. However, over time, national governments may nonetheless start turning to this global knowledge to aid their decision‐making in part because of the depleted nature of their own information systems. It potentially creates a situation where national governments’ ways of knowing their own populations is mediated via Seattle. Under the garb of neutral and scientific metrics, a global knowledge with systematic silences becomes a universal way of knowing and acting upon the world.
Notes
This paper benefited from support to the special issue project from Friedrich‐Ebert‐Stiftung New York Office; UNDP; University of Oslo Centre for Environment and Development and the Environment; Julien J. Studley Grant to The New School Graduate Programs in International Affairs.
Biography
Manjari Mahajan is Assistant Professor in the Graduate Program for International Affairs at The New School University, USA.
