High variability in the measurement of HIV primary prevention activities and outcomes

Abstract Introduction While there is a global consensus on monitoring Human Immunodeficiency Virus (HIV) treatment progress, there has been less attention to the degree of consistency of the measurement of HIV prevention programmes—and the global prevention response is not on‐track to achieve 2020 goals. In this paper, we assess the degree of variability in primary prevention indicators selected by national strategic plans (NSPs) and global stakeholder monitoring and evaluation (M&E) strategies. Methods We obtained the most recent NSPs from low and middle income Joint United Nations Programme on HIV/AIDS (UNAIDS) Fast‐Track countries, and M&E documents from The Global Fund to Fight AIDS, Tuberculosis and Malaria (The Global Fund), President’s Emergency Plan for AIDS Relief (PEPFAR), UNAIDS, the Global HIV Prevention Coalition and the World Health Organization (WHO). We extracted HIV primary prevention indicators from each document, standardized and aggregated them by age/ sex, categorized indicators by topic, and evaluated the frequency of matched indicators between countries and stakeholders. Data were collected between February and April of 2019. Results Twenty‐one NSPs and five global stakeholder documents were assessed; 736 primary prevention indicators were identified; 284 remained following standardization and aggregation. NSPs contained from 3 to 48 primary prevention indicators, with an average of 23; categories included: HIV education and outreach (17.6%), testing (17.3%) and condom use (16.2%). Of unique national indicators, only 34% was shared between two or more countries. Sixty‐nine per cent was applied in a single country only. 56% of NSP indicators did not appear in any global stakeholder document. Conversely, 42% of global indicators did not appear in any surveyed NSPs. Within global indicators, 63% was only measured by one global body, and no single indicator was measured by all five. Conclusions These analyses reveal a lack of consensus both between and within countries’ and global stakeholders' measurement of HIV prevention. Though some variability is expected, these findings point to a need to refocus attention on achieving greater consensus on a global measurement framework for HIV prevention.


| INTRODUCTION
Substantial progress has been made in responding to the HIV epidemic over the past decade, with new HIV infections reduced by an estimated 40% since their peak in 1996, and a 60% decrease in Human Immunodeficiency Virus/Acquired Immunodeficiency Syndrome (HIV/AIDS)-related mortality since 2005 [1]. However, HIV remains a global public health threat. An estimated 1.7 million people became newly infected with HIV in 2019, far above the United Nations General Assembly goal of fewer than 500,000 new HIV infections in 2020 [1,2].
In recent years, there has been a focus on expanding treatment access to improve patient outcomes and decrease the risk of onwards transmission; however, in some settings, there has been insufficient attention to primary HIV prevention. In 2012, after preventive benefits of HIV treatment were confirmed by the HPTN052 trial, models projected that HIV incidence would decrease by 35% to 54% within eight years in areas where population coverage levels reached 80% [3]. However, declines of this magnitude have not been observed in studies on a population level in the test and treat trials, highlighting the continued need for evidence-based combination prevention interventions [4][5][6].
One of the strengths of the treatment response is the clarity of programmatic goals and indicators. The 2014 UNAIDS 90-90-90 targets (90% of all people living with HIV know their status, 90% of all people with diagnosed HIV infection will receive sustained antiretroviral therapy and 90% of all people receiving antiretroviral therapy will have viral suppression by 2020) helped to streamline and prioritize monitoring and evaluation efforts for treatment and arguably served as an effective and powerful rallying and advocacy tool [7]. There is strong global alignment behind these targets, from the U.S. President's Emergency Plan for AIDS Relief (PEPFAR), Joint United Nations Programme on HIV/AIDS (UNAIDS), to The Global Fund to Fight AIDS, Tuberculosis and Malaria and the World Health Organization (WHO) [8][9][10][11][12]. Having a clear, focused treatment target has mobilized and influenced these decision makers, with substantial progress made in achieving the targets since their launch [13,14].
Although there have been efforts to establish clear prevention goals, such as the establishment of UNAIDS's five pillars [2], national governments and global stakeholders all have processes to produce monitoring and evaluation indicators [10,12]. There has been limited examination of the range of prevention-related indicators in use and the degree to which global stakeholders and governments are aligned in their measurement. Without this understanding, it is difficult to understand the feasibility and path forward to attain more streamlined indicators and global targets.
In order to understand the degree of consensus in HIV primary prevention measurement and evaluation, we review the variability in the primary prevention indicators selected in the country national strategic plans (NSPs) and global monitoring and evaluation documents. In addition, we examine the frequency of categories of primary prevention indicators used most consistently across stakeholders. . This analysis included only the low-and middle-income countries. The search strategy for the NSP documents included an internet search of the country name, "HIV" and "National Strategic Plan" or "National framework" or "Monitoring and Evaluation framework, " and the Ministry of Health website for each country. Representatives of each country's Ministry of Health were contacted through publicly available contact information found on the website. Data were collected between February and April of 2019. The most recent publicly available NSPs were compiled for each country. Any NSP that was published and applied (i.e. years in scope) prior to 2014 were excluded. Indicators were identified within each NSP by searching for tables containing the term "indicator"; these were recorded and subsequently evaluated based on inclusion/exclusion criteria.  [8][9][10][11][12]. These documents were obtained from each of the global organizations' websites.

| Inclusion/exclusion criteria for indicators
Each indicator within each NSP and global stakeholder document was reviewed and evaluated based on predefined inclusion/exclusion criteria that are detailed in Table 1. All indicators relevant to the primary prevention of incident HIV infections were included. Indicators related to antiretroviral treatment and prevention of mother to child transmission (PMTCT) were excluded. HIV testing indicators were included in the analysis because HIV testing is a necessary pre-condition for eligibility for most HIV prevention interventions, and serves as the entry point to the prevention cascade.
Each indicator included in our study was categorized within a measurement and evaluation indicator logical framework [16]. Indicators were included if they were defined as "activities, " "outputs, " or "outcomes, " and excluded if defined as "assessment and planning, " "inputs, " or "impacts" to capture indicators relevant to primary prevention [16].

| Data extraction and indicator standardization and aggregation
For each indicator, population type, age group (if available), numerator and denominator was extracted and recorded. These fields were then standardized across country NSP indicator lists and global indicator lists on these fields, such that exact language was not necessary to constitute a match. For example "men who receive voluntary male medical circumcision" and "men who undergo voluntary male medical circumcision, " would be standardized to "men who receive voluntary male medical circumcision. " Indicator language standardization was performed by SS and confirmed by JD.
Following extraction and standardization from the reviewed documents, indicators were aggregated by age and sex, to minimize over-reporting of variability based on minor variation in disaggregation of age and sex amongst nations and global stakeholders. They were also combined if they measured "percent of" or "number of". For example if one country measured "% of men aged 15 to 20 who are circumcised" and one global body measured "# of men aged 30 to 39 who are circumcised" this would be considered a match between the country and global body. However, indicators remained disaggregated by key populations (defined groups who are at increased risk of HIV, in any context), priority populations (groups who are at increased risk of HIV, in certain contexts) and other distinct populations ( Table 2).

| Data analysis
The indicators collected by each country were identified. For each country, indicators were evaluated by whether they appeared in any global stakeholder monitoring and evaluation (M&E) document. The proportion of indicators not in any evaluated global stakeholder M&E documents was calculated, by country and averaged across countries. The indicators listed by countries were evaluated by whether they were present in any global stakeholder prevention indicator set and the number of country NSPs. We conducted sub-analyses to examine variability in NSP indicators within the high burden region of Eastern and Southern Africa [17], and by countries with high vs. low burdens (countries with "high" burdens have >1% prevalence of HIV and countries with "low" burdens have <1% prevalence of HIV) [18]. We also compared HIV prevalence [18] to the number of indicators in a country NSP and calculated a Pearson's correlation coefficient (r).
Similarly, the number of indicators identified by each global stakeholder were compiled. Each global indicator was evaluated by whether it was documented in any country NSP. For each indicator, the number of countries that identified the indicator, as well as the number of global stakeholders that identified the indicator was assessed. The authors used Microsoft Excel (version 10) to house and analyse the extracted data.

| RESULTS
Of the 28 2015 LMIC Fast-Track countries, the NSP for 24 (86%) were obtained (Table 2a). Of these, seven were translated to English using online translation services: Angola (Portuguese), Cameroon (French), Chad (French), China (Mandarin), Cote d'Ivoire (French), Ukraine (Ukrainian) and Vietnam (Vietnamese). The NSPs for Jamaica, Vietnam and China did not include any indicators, so these three countries were also excluded from the analysis leaving 21 countries to be included. Documents were also evaluated for five global stakeholders (Table 2b). In total, 736 unique primary prevention indicators were extracted. After standardization and aggregation, 284 unique primary prevention indicators remained for analysis.

| Indicator alignment between countries
The 21 NSPs (Table 3) contained 247 unique indicators. Of these, 162 (66%) were measured by only one country. The indicators with the most country-level consensus were (in descending order of consensus): -"%/# of general population who report testing in last 12 months and know test results" (13 countries), -"%/# of general population who correctly identify main forms of HIV transmission and reject incorrect ones" (12 countries) -"%/# of general population who had more than one partner in the last 12 months and report condom use at last sexual encounter" (11 countries) The NSPs of the 11 high burden HIV countries of Eastern and Southern African included 158 indicators, with high variability between these countries; 107 (68%) of the indicators were listed in only one NSP. Of the 235 indicators measured by the 17 countries with high burden epidemics, 158 (67%) were measured by only one NSP, and the most commonly measured indicator ("%/# of general population who report testing in last 12 months and know test results") was included in 11 NSPs. Of the 59 indicators measured by five countries a. Indicator is in a table labeled with the term "indicator" b. Indicator is associated with HIV acquisition and incidence (including discrimination against people living with HIV and sero-discordant couples) c. Indicator is in "activity, " "output, " and "outcome" category of data collection a • Or, document categorizes indicator as a HIV prevention indicator Exclusion criteria • Indicator is associated with mother-to-child transmission • Indicator is in "assessment and planning, " "inputs, " or "impacts" category of data collection a with low HIV burdens, 42 (74%) were included in only one NSP and only 2 (3.5%) indicators were measured by four countries. Furthermore, only a weak positive correlation was found when comparing the number of indicators and HIV prevalence for each country (r = 0.30).

| Indicator alignment between global stakeholders
The M&E guidance documents of the five global stakeholders (Table 4) included 89 unique indicators. Of these, 56 (62.9%) were included by only one stakeholder, seven were included by four stakeholders and none were present in all five documents. The seven indicators included by four stakeholders were (in alphabetical order): -"%/# of general population who receive pre-exposure prophylaxis (PrEP)" -%/# of men who have sex with men (MSM) who report condom use during the last occurrence of anal sex" -"%/# of MSM reached with programmes for HIV prevention" -"%/# of people who inject drugs (PWID) reached with programmes for HIV prevention" -"%/# of sex workers (SW) who report condom use with last client" -"%/# of SW reached with programmes for HIV prevention" -"# of syringes distributed per PWID"     Figure 2) included 89 unique indicators. Of these, 52 (58.4%) were collected in at least one country, whereas 37 (41.6%) did not appear in any surveyed NSPs. By global stakeholder, the percentage of indicators not appearing in any NSP ranged from 13% (Global HIV Prevention Coalition) to 42% (GAM).

| Indicator alignment between countries and global stakeholders
Tables S1 and S2 show global stakeholder indicators that were absent from any surveyed NSP and indicators measured by at least one surveyed NSP and at least one global stakeholder respectively. Examples of global stakeholder indicators absent from any surveyed NSP include: -"%/# of general population diagnosed with gonorrhoea in past 12 months" -"%/# of health units with PrEP services" Examples of indicators measured by at least one surveyed NSP and at least one global stakeholder include: -"%/# of MSM who report testing in the last 12 months and know test results" -"%/# of PWID who report using sterilized equipment during last needle use" -"%/# of males circumcised"
However, outside the top five categories, there was relatively little alignment in countries and global bodies in indicator type measured (Figure 3). For instance South Africa's NSP and UNAIDS GAM were the only two documents that included hormonal contraceptives in any indicator. Likewise, four of the five global stakeholders (GAM, GF M&E, MER and

| DISCUSSION
In order to understand the degree of consensus in HIV primary prevention measurement and evaluation, we reviewed the primary prevention indicators included in national strategic plans and global stakeholder M&E documents. Our review found limited agreement between global stakeholders and countries measurement of HIV prevention, with no indicators measured by more than 13 of the 21 countries, and no indicators measured by all five global stakeholders. There was a high range (from 3 to 48) in the number of indicators measured within each country and by global stakeholder (from 15 to 45). The number of indicators measured by each country was only weakly correlated with HIV rate by country. The majority (66%) of national indicators for HIV prevention were measured only by a single country; this heterogeneity amongst countries remained even when comparing countries with similar epidemics and/or geographies. Similarly, 63% of indicators in global stakeholder M&E documents was only measured by one global stakeholder. Additionally, there was a high degree of discordance between and within countries and global stakeholders, with 56% of country indicators not appearing in any of the evaluated global stakeholder M&E documents when averaged across countries. There was also a high degree of discordance in terms of prevention categories measured and the number of indicators measured in each category.
Our findings suggest that global stakeholders need to harmonize to a set of indicators that have a firm empiric basis, are strongly linked to HIV prevention outcomes, are harmonized across geographies to the extent possible, and meet the rigour of a framework (e.g. the UNAIDS indicator framework) [19]. Some degree of variability among global stakeholders, among countries, and between global stakeholders and countries is expected for several reasons. First, there is heterogeneity in terms of how global stakeholder documents are intended to be used. For example UNAIDS GAM is meant to be for annual global reporting on outcome and impact indicators, as defined by political declarations [10]. The Global HIV Prevention Coalition aims to monitor national prevention programmes [11], and MER indicators reflect US government priorities [9]. There is heterogeneity in reporting frequency (e.g. some MER indicators are reported quarterly, some semi-annually and some annually). There are also differences in how often indicators are modified, with WHO SI guidelines updated every five years and Global Fund guidelines updated every three years, whereas MER indicators are updated yearly [8,12]. This variability in purpose naturally affects the variability in indicators collected. This is true of NSPs as well. Kenya states that their NSP should be used to "track progress and continuously measure results" and for "enhancing transparency of all players, " whereas India's NSP states that its aim is to "provide a solid framework to tailor the response to local needs based on context-specific evidence. " Second, countries have different dynamics in their HIV epidemic and response, and that variability could be reflected in their indicators. For instance high burden countries may be focused on more general populations and low-burden countries may focus more on key populations. There are substantial differences in the national HIV programmes and epidemiologic contexts of different countries. Third, indicators that are also measured by other programmes or sectors may not be reflected in HIV NSPs; for instance STI-related indicators may be recorded in Sexual and Reproductive Health monitoring and evaluation plans. Finally, the process for creating NSPs and global stakeholder documents varies by country and organization, with some bodies implementing a more rigorous process than others, engaging a wide range of stakeholders and experts, whereas other processes are more ad-hoc or political. Measuring prevention remains challenging because of the difficulty identifying high-risk individuals, determining which prevention interventions are effective, and then taking those interventions to scale. Some variability in measurement is useful as it provides complementary perspectives on the state of HIV prevention, but attention is needed on where variability is expected and where measurement should be aligned.
Despite some of the potential explanations for the lack of consensus, the analysis suggests a general lack of global alignment on measurement priorities and approaches within HIV prevention [20]. For example indicators measured by four of five global stakeholders were primarily measurements of key populations or PrEP, whereas the indicators most measured by countries were on testing, condom use and HIV education in general populations. There does not seem to be a clear explanation for these differences, but does suggest discordance and underscores the lack of a unified global prevention response.
One example where variability was particularly stark was in indicators that measure condom use. For example, Cote d'Ivoire listed 11 out of 31 indicators in their NSP within in the single category of condom use, whereas Nigeria and Ukraine did not measure condom use at all. In addition, there was a high variability in the language used between different condom indicators at both the country and global levels. Another notable example was in the measurement of stigma and discrimination, which was one of the top five categories in both countries and global stakeholders, yet had significant heterogeneity and definitions used.
Heterogeneity in indicators and definitions can have real costs; for example resources are required to collect indicators; funding may be at risk if indicators are not collected. As an example, the Norwegian Refugee Council found that if their nine largest donors used the same financial reporting format, the organization would save 11,000 hours per year on financial reporting [21]. Frontline health workers at health facilities are often the most impacted by large numbers of indicators, as described in an assessment performed by the WHO [22]. There are also costs at the programmatic level in terms of the human and financial resources required to collect and report large numbers of indicators [21]. This is especially the case in the HIV response, in which there are multiple global stakeholders and large financial investments. Of note, indicators that focused solely on testing or treatment of people living with HIV were not included in this analysis; the reporting burden is broader than prevention indicators alone. Larger than necessary indicators sets, particularly those with varying definitions for similar indicators, are likely to result in poorer data collection and challenges in assessing performance at the national and global level.
National strategic plans and global stakeholder documents can serve as a proxy for a country or global body's priorities, and some variability is expected and can be beneficial. Yet the degree of heterogeneity between selected indicators seems to indicate an opportunity to reevaluate and better understand the process for prioritizing and selecting indicators for HIV prevention at the national and global levels. The current UNAIDS-led Monitoring Technical Advisory Groupthe mandate of which is to harmonize national and global HIV indicatorsshould re-evaluate and strengthen the process of selecting and prioritizing indicators for HIV prevention. Concomitantly, donors with reporting requirements should commit to prioritizing alignment, and leverage existing metrics whenever possible (e.g. through the use of the UNAIDS Indicator Repository) [23]. Greater alignment behind clearer and more consistent goals and measurements could ultimately catalyse action for more effective HIV prevention programmes.

| LIMITATIONS
The potential limitations of this study include the availability and translation of surveyed NSPs. This analysis only includes 21 of 28 possible low-/middle-income 2015 Fast-Track countries, although the included NSPs account for over 90% of the estimated burden of new infections among the 28 countries [18]. Additionally, we assume that all NSPs and stakeholders use similar definitions for key populations or programmes (e.g. that the definition of "sex workers" is consistent across countries and global organizations). Furthermore, as mentioned earlier, we anticipate that some of the differences may be related to the differential timing of the development of the NSPs and global stakeholder's indicator sets, although we only included recent sets (2014 and onward) to mitigate this to the extent possible.

| CONCLUSIONS
Reaching ambitious global 2030 goals for ending the HIV epidemic will require a more effective HIV prevention response. This analysis suggests that the measurement of HIV prevention programmes is highly heterogeneous and can reasonably be considered one of the factors impeding progress. The vast number of indicators and their diverse formats create a high burden of collection on national systems and reflects the lack of clarity about priorities for HIV prevention in various settings and the best indicators for measurement. If more effective HIV prevention programmes are to be developed, greater attention to creating unified frameworks for measurement will be an important part of the process.

R E F E R E N C E S SUPPORTING INFORMATION
Additional information may be found under the Supporting Information tab for this article. Table S1. Global stakeholder indicators that are absent from any surveyed national strategic plans (NSPs) Table S2. Indicators measured by at least one surveyed national strategic plan (NSP) and at least one global stakeholder