Improving malaria treatment and prevention in India by aiding district managers to manage their programmes with local information: a trial assessing the impact of Lot Quality Assurance Sampling on programme outcomes

Authors


Abstract

Objectives

This paper reports the first trial of Lot Quality Assurance Sampling (LQAS) assessing associations between access to LQAS data and subsequent improvements in district programming. This trial concerns India's approach to addressing an increase in malaria-attributable deaths by training community health workers to diagnose, treat and prevent malaria, while using LQAS to monitor sub-district performance and make programme improvements.

Methods

The Ministry of Health introduced LQAS into four matched high malaria burden districts (Annual Parasite Incidence >5) (N > 5 million). In each sub-district, we sampled four populations in three 6-monthly surveys: households, children <5 years, people with fever in the last 2 weeks and community health workers. In three districts, trained local staff collected, analysed and used data for programme management; in one control district, non-local staff collected data and did not disseminate results. For eight indicators, we calculated the change in proportion from survey one to three and used a Difference-in-Differences test to compare the relative change between intervention and control districts.

Results

Coverage increased from survey one to three for 24 of 32 comparisons. Difference-in-Differences tests revealed that intervention districts exhibited significantly greater change in four of six vertical strategies (insecticide treated bed-nets and indoor residual spraying), one of six treatment-seeking behaviours and four of 12 health worker capacity indicators. The control district displayed greater improvement than two intervention districts for one health worker capacity indicator. One district with poor management did not improve.

Conclusions

In this study, LQAS results appeared to support district managers to increase coverage in underperforming areas, especially for vertical strategies in the presence of diligent managers.

Abstract

Objectifs

Ce document présente le premier essai de la méthode d’échantillonnage par Assurance de la Qualité du Lot (LQAS) évaluant les associations entre l'accès aux données LQAS et les améliorations ultérieures dans la programmation du district. Cette étude concerne l'approche de l'Inde en réponse à une augmentation des décès attribuables au paludisme par la formation des agents de santé communautaires à diagnostiquer, traiter et prévenir le paludisme, tout en utilisant la méthode LQAS pour surveiller les performances à l’échelle sous-district et apporter des améliorations au programme.

Méthodes: Le Ministère de la Santé a introduit la méthode LQAS dans quatre districts appariés à charge paludique élevée (incidence parasitaire annuelle > 5) (N > 5 millions). Dans chaque sous-district, nous avons échantillonné quatre populations lors de trois enquêtes effectuées tous les 6 mois: les ménages, les enfants de < 5 ans, les personnes atteintes de fièvre au cours des deux dernières semaines et les agents de santé communautaires. Dans trois districts, le personnel local formé a recueilli, analysé et utilisé les données pour la gestion du programme; dans un district contrôle, du personnel non local a recueilli des données et n'a pas diffusé les résultats. Pour huit indicateurs, nous avons calculé la variation de la proportion de la 1ère à la 3ème enquête et avons utilisé un test de la différence des différences pour comparer la variation relative entre les districts d'intervention et de contrôle.

Résultats: La couverture a augmenté de la 1ère à la 3ème enquête pour 24 sur 32 comparaisons. Les tests de la différence des différences ont révélé que les districts d'intervention présentaient significativement plus de variations dans 4 des 6 stratégies verticales (moustiquaires imprégnées d'insecticide et pulvérisation intérieur de résidu), 1 sur 6 comportements de recherche de traitement et 4 sur 12 indicateurs de la capacité des agents de santé. Le district contrôle a affiché une plus grande amélioration que deux districts d'intervention pour un indicateur de la capacité des agents de santé. Un district avec une mauvaise gestion n'a pas connu d'amélioration.

Conclusions: Dans cette étude, les résultats de la méthode LQAS se sont avérer appuyer les gestionnaires de district à augmenter la couverture dans les zones moins performantes, en particulier pour les stratégies verticales en présence de gestionnaires diligents.

Abstract

Objetivos

Este artículo informa sobre el primer ensayo de muestreo de aceptación de lotes (LQAS) para evaluar las asociaciones existentes entre el acceso a datos de LQAS y las mejoras posteriores en la programación del distrito. Este ensayo muestra la estrategia de la India para abordar el aumento en las muertes atribuibles a la malaria mediante el entrenamiento de los trabajadores sanitarios comunitarios, con el fin de que diagnostiquen, traten y prevengan la malaria, mientras que se utiliza LQAS para monitorizar el desempeño subdistrital y se realizan mejoras en los programas.

Métodos

El Ministerio de Salud introdujo el LQAS en cuatro distritos pareados con una alta carga por malaria (Incidencia Anual de Parásitos >5) (N>5 millones). En cada subdistrito se muestrearon cuatro poblaciones en tres estudios de 6 meses: hogares, niños menores de 5 años, personas con fiebre en las últimas dos semanas y trabajadores sanitarios comunitarios. Trabajadores locales entrenados pertenecientes a tres distritos recogieron, analizaron y utilizaron datos para la gestión del programa; en un distrito control, los datos fueron recogidos por personal que no era local y no se realizó una diseminación de los resultados. Para ocho indicadores hemos calculado los cambios en la proporción entre la encuesta uno y la tres y utilizamos una prueba de Diferencias en Diferencias (DID) para comparar el cambio relativo entre los distritos de intervención y control.

Resultados

La cobertura aumentó entre el primer y el tercer estudio en 24 de las 32 comparaciones. Las DID revelaron que los distritos en los que se había llevado a cabo la intervención mostraban cambios en 4 de 6 estrategias verticales (mosquiteras impregnadas con insecticida y rociamiento residual intradomiciliario), 1 de 6 comportamientos de búsqueda de tratamiento y 4 de 12 indicadores de capacidades de los trabajadores sanitarios. El distrito control mostró una mayor mejora que dos de los distritos intervenidos en un indicador de capacidad de los trabajadores sanitarios. En un distrito con una mala gestión no se observó mejora.

Conclusiones

En este estudio de LQAS los resultados parecen apoyar a los gestores distritales a la hora de aumentar la cobertura en áreas con un desempeño subóptimo, especialmente en lo que se refiere a estrategias verticales en presencia de gestores diligentes.

Introduction

In 2010, an independent estimate of 277 000 deaths in India caused by malaria exceeded previous estimates 10-fold, dwarfing WHO's estimate of 15 000 (WHO 2008) and the number of Indian certified malaria deaths (~1000) (National Vector Borne Disease Control Programme 2011b). That report is not the first to suggest that the malaria burden is considerably higher than earlier reported (Yadav et al. 2003; Kumar et al. 2007; Dash 2009; Dhingra et al. 2010; Hay et al. 2010). Although another study provided a lower annual mortality estimate (50 000) (Murray et al. 2012), adding to the uncertainty (Basnyat 2011; Shah et al. 2011; Valecha et al. 2011), what is certain it that malaria is a leading cause of mortality.

India's attempt to eradicate malaria during the 1950s resulted in a major reduction in the malaria burden. Although the prevalence subsequently increased in the 1970s, it never reached earlier levels. However, eastern states persistently report a high malaria burden, thereby prompting the Government of India (GoI) to introduce new control measures. The World Bank funded nine high malaria burden states (The World Bank 2008) under the leadership of the National Vector Borne Disease Control Programme (NVBDCP) to introduce these new approaches. They included (i) adopting rapid diagnostic tests (RDT) for point-of-care diagnosis of malaria including at the village level; (ii) introducing Artemisinin-based Combination Therapy (ACT) to treat falciparum malaria (P.f.); (iii) adopting long-lasting insecticidal nets (LLIN); (iv) creating the position of malaria technical supervisors (MTS) as staff dedicated to collecting and analysing malaria programme information at the sub-district level, presenting their conclusions to local managers, and resolving local logistical problems; and (v) using village health volunteers for malaria case management (National Vector Borne Disease Control Programme 2011a). These volunteers were primarily Accredited Social Health Activist (ASHA). The existing monitoring system consisted of ASHA maintaining a paper and pencil record of activities, which formed part of the recurrent health information system. NVBDCP managers recognised that this system was not appropriate for these new approaches and decided to introduce Lot Quality Assurance Sampling (LQAS), which permitted decentralised monitoring at both district and sub-district levels (called Block Community Health Centres: B-CHC), was low cost, and could be implemented by local health workers. Odisha, the focus of this paper, accounts for 25% of India's malaria cases and deaths (National Vector Borne Disease Control Programme 2011b), despite having <4% of the population.

In its elemental form, LQAS is a classification system categorising areas as ‘high’ or ‘low’ performing, thereby allowing managers to address problems. Due to the small sample size used for these classifications, LQAS has been in high demand; a recent LQAS literature review identified more than 800 studies (Robertson & Valadez 2006). Odisha VBDCP managers designed the first state application in November 2009 as a trial to appraise LQAS's utility for programme strengthening (Figure 1).

Figure 1.

Lot Quality Assurance Sampling Process Cycle.

The use of LQAS has burgeoned in that last 20 years (Robertson & Valadez 2006) along with the growing concern to address problems in health service delivery (Kim et al. 2013). While other studies have documented benefits of using LQAS data to guide decision-making (Valadez et al. 1995, 2005; Valadez & Devkota 2002), this study documents the first trial of LQAS as it affects decision-making at such a large scale.

Methods

This trial uses a Non-Equivalent Control Group Design (Campbell & Stanley 1963) in which three districts (Mayurbhanj, Sundargarh and Nabrangpur) had access to LQAS results and one district (Kandhamal) did not (Figure 2). Each district has a population exceeding 741 000 people (Table 1). A stakeholder group consisting of VBDCP managers, World Bank advisors, NGO managers and DFID programme consultants matched districts according to four criteria: a large proportion of low-income and tribal populations, >5 Annual Parasite Incidence (API) (>5 confirmed malaria cases reported per 1000 persons per year), received World Bank funding to support malaria prevention and treatment, and the presence of a functioning health system (inclusion criterion: each district's vaccination coverage approximated Odisha's average vaccination coverage) (Table 1). To reduce inter-district contamination, the districts were distributed throughout Odisha and did not share borders. The four selected districts exhibited the least inter-district variation among the selection criteria.

Table 1. Comparison of four districts in Odisha State used in this study
High burden districtsGeographical locationPopulation 2009Number of Block CHCNo. block CHC with >5 API*No. MTS working in block CHC with >5 APIFull routine immunization coverage, %
  1. Block CHC, Block Community Health Centre; LQAS, Lot Quality Assurance Sampling; API, Annual Parasite Index; MTS, Malaria Technical Supervisors are government technical staff responsible for 1–2 Block CHC. *API = Total positive slides for parasites in a year × 1000/Total population.

LQAS districts
1SundargarhNorth-west2 083 3661715860–75
2MayurbhanjNorth2 538 4762614745–60
3NabrangpurSouth-west1 168 705109530–45
Non-LQAS districts
1KandhamalCentral743 0601212645–60
Figure 2.

Map of Odisha State, India highlighting four districts included in this study.

Odisha VBDCP managers conducted three LQAS surveys in the intervention districts: November–December 2009, July–August 2010 and November–December 2010; the first and third surveys also were conducted in the control district in intervention districts. MTS were responsible for collecting and analysing data and presenting results to B-CHC and district managers. In the control, an outside survey team collected the data and did not share data or results with anyone until after the third survey in 2011. MTS collected the data for the third survey.

We used standard LQAS procedures (Valadez 1991; Robertson et al. 1997). LQAS is an established analysis technique originally developed as a classification method for industrial quality control during the 1920s and adapted to health sciences in the mid-1980s (Reinke 1988; Valadez 1991; Robertson et al. 1997) to classify management units, referred to as supervision areas (SA) according to a performance target. In each SA, a sample of ‘n’ individuals is assessed. A ‘d’ is selected that determines the cut-off number below which the area is classified as low performance for a specified indicator. The decision rule ‘d’ depends on the sample size, thresholds for classifying high and low performance, and selection of two misclassification errors: the risk of misclassifying an SA with very low coverage as high (β error), and the risk of misclassifying an SA with high coverage as low (α error). National stakeholders set the upper threshold, ‘pU’, for identifying acceptably performing SA while low performers, ‘pL’, are often 30% lower (Pagano & Valadez 2010). In our surveys, 19 villages were selected in each B-CHC, which is the largest administrative unit within a district, and which served as the SA for this study (Table 2). The SA sample size, n = 19, ensures that α and β errors do not exceed 10% for all ‘pU’, and all corresponding values of ‘pL’ that are 30 percentage points less than ‘pU’, as is the case for this trial. SA with pL < p <pU are also classified. A high classification is more likely the nearer p is to pU (and vice versa for pL); p located in the middle of this so-called grey area has an approximately equal chance of being classified in the high or low category.

Table 2. Surveys completed, sample sizes and use of results in four districts: November 2009–August 2011
DistrictSurvey 1 Nov–Dec 2009Survey 2 July–Aug 2010Survey 3 Nov–Dec 2010
Total sample size (no. SA)Used results for managementTotal sample size (no. SA)Used results for managementTotal sample size (no. SA)Used results for management
  1. SA, Supervision Area; API, Annual Parasite Index: Total positive slides for parasite in a year × 1000/Total population.

  2. a

    One additional Block CHC was added to Nabrangpur during 2010 as NVBDCP determined API >5.

Sundargarh (intervention)285 (15)Yes285 (15)Yes285 (15)Yes
Mayurbhanj (intervention)266 (14)Yes266 (14)Yes266 (14)Yes
Nabrangpur (intervention)171 (9)Yes190 (10)aYes190 (10)Yes
Kandhamal (control)228 (12)NoNot conductedNA228 (12)Yes

Other monitoring

Odisha, like other Indian States, does not have sufficient public health staff to monitor the malaria programme in all villages. MTS were intended to remedy this deficiency. However, they are too few. Nevertheless, MTS and district supervisors carried out regular monitoring and supervision, in addition to LQAS, in all four districts. For example, they visited ASHAs and health facilities to replenish supplies, collect recurrent information and solve problems.

Data collection

Surveys used a three-stage sampling design. At Stage 1, probability proportional to size sampling identified 19 villages in each of 51 B-CHC. At Stage 2, MTS used segmentation sampling (Turner et al. 1996; Milligan et al. 2004; Valadez et al. 2007) to randomly select a household. At Stage 3, interviewers randomly selected an eligible respondent in the sampled household. All stages were repeated for each sampling element in each survey. The survey questionnaire has four modules, each requiring different respondents: a household head, a caretaker of a child aged between 0 and 59 months, a person with fever in the last 2 weeks and ASHA. We designed the questionnaire to measure key NVBDCP's programme indicators; LLIN distribution and use, in-door residual spraying (IRS) and community knowledge and practices related to early detection and treatment of malaria. The questionnaire was designed collaboratively with Odisha VBDCP managers and approved by the State Secretariat for Health's Technical Working Group.

In October 2009, prior to the first data collection, the authors trained 27 MTS in the three intervention districts, 12 staff from an external data collection agency for the control district and 13 supervisors on the LQAS methodology and sampling procedures. Refresher training was carried out prior to each survey along with new MTS receiving a comprehensive training.

Data management, dissemination and analysis

Once data collection finished, we trained intervention district MTS to hand tabulate data and classify B-CHC as having reached or not reached state performance targets for key indicators. (Valadez et al. 2007). Following the hand tabulation, MTS shared the results with district public health officials to discuss: (i) reasons why targets were not met and (ii) strategies that address problems. We then aggregated data across all B-CHCs (weighted by B-CHC population sizes) to calculate district wide coverage estimates. Identical procedures were implemented in all intervention districts and facilitated by one or more of the authors.

After each survey round, questionnaire data were double entered into an ACCESS database to eliminate data entry errors and analysed using SPSS-19. We measured change in coverage estimates for key indicators within each district using the first and third survey time points. We calculated 95% confidence intervals (95% CI) for the change estimates with Stata-12.1 using logistic regression. Thereafter, we used difference-in-differences (DiD) tests. To use this procedure, we first calculated within each district the change between the baseline and final time point; then we compared the difference between each intervention district and the control district. We tested for significance using the 95% CI, which when crossing zero, indicates no evidence of difference. This trial design has three independent comparisons of intervention districts with the control district, with each district in the comparison having comparable sample sizes. We opted for this approach rather than collapsing the intervention districts into a single comparison group because districts in India have a large population size with a corresponding health system bureaucracy that are similar to regions in other countries. Collapsing the districts would have obfuscated the inter-district variation. District senior managers have styles that can vary considerably from other districts that may be relevant to this trial. Hence, a district level comparison is a fairer analysis than aggregating the intervention data as a single pool.

Ethics

This project was reviewed by the NVBDCP of India, the Technical Working Group of the World Bank, the Office of the Odisha Secretary of Health, and by the Odisha VBDCP. They reviewed the study design, questionnaires and informed consent procedures, and data security. The monitoring and evaluation methods included in this study are incorporated into GoI's contract with the World Bank.

Results

This study reports on eight indicators, NVBDCP used to monitor the malaria programme; each intervention district is compared separately with the control district (Table 3). Indicators 1 and 2 report on the proportion of people protected with either IRS or LLINs. Indicators 3 and 4 assess the treatment-seeking behaviour of people with fever and whether they received appropriate malaria treatment. Indicators 5–8 assess an ASHA's capacity to diagnose and treat malaria. Table 3 shows aggregated coverage figures for each indicator by district at three time points and the proportional change between surveys one and three with 95% CI.

Table 3. Coverage proportions for eight indicators, three time points, four districts with proportional change and Difference-in-Differences tests
IndicatorsDistrictSurveya 1 Nov 2009Survey 2 Aug 2010Survey 3 Dec 2010% Change95% CI ± of the Change: P-valueDiff in diff95% CI ± of the change: P-value
  1. ITN/LLIN, Insecticide Treated Net/Long-Lasting Insecticidal Net; IRS, Indoor Residual Spraying; RDT, Rapid Diagnostic Test; ACT, Artemisinin-based Combination Therapy; ASHA, Accredited Social Health Activist; P.f., Plasmodium falciparum; C, Control; NS, Not significant.

  2. a

    Targets (pU) are listed for each time point and indicator. All values of pL are 30 percentage points lower. Decision rules (DR) used: DR = 3 (30%), DR = 5 (40%), DR = 7 (50%), DR = 9 (60%), DR = 13 (80%).

Vertical InterventionSection Target30%40%50%      
1Proportion of people protected by either ITN/LLIN or IRS the night preceding the surveySundargarh23.835.248.925.17.70.00012.812.00.036
Mayurbhanj18.840.945.226.57.70.00014.211.90.02
Nabrangpur4.714.113.28.55.90.005−3.810.9NS
Kandhamal (c)32.745.012.39.20.000   
2Proportion of children protected by either ITN/LLIN or IRS the night preceding the surveySundargarh14.637.747.733.17.10.00015.111.40.01
Mayurbhanj8.645.442.734.16.60.00016.111.00.004
Nabrangpur1.812.111.59.75.10.000−8.310.2NS
Kandhamal (c)27.745.718.08.80.000   
Treatment-Seeking BehaviourSection target30%40%50%      
3Proportion of people with fever in the last 2 weeks whose blood was tested with an RDT within 1 day of the fever and told the result on same day as the testSundargarh11.724.721.910.26.00.0018.510.0NS
Mayurbhanj17.237.632.215.17.70.00013.411.20.019
Nabrangpur3.710.63.6−0.13.9NS−1.77.2NS
Kandhamal (c)16.317.81.58.0NS   
4Proportion of people with fever in the last 2 weeks who were RDT+ve and received ACTSundargarh1.65.61.4−0.29.6NS−0.212.1NS
Mayurbhanj1.65.30.0−1.611.0NS−1.613.3NS
Nabrangpur11.731.90.0−11.727.4NS−11.728.4NS
Kandhamal (c)0.00.00.07.4NS   
Community Health Workers/ASHA capacityTarget50%60%60%      
5Proportion of ASHA who have at least a 2-months RDT stockSundargarh8.416.020.612.25.70.000−2.914.2NS
Mayurbhanj22.042.842.420.48.00.0005.311.7NS
Nabrangpur61.252.925.3−35.9−26.00.000−51.0−12.70.000
Kandhamal (c)22.938.015.18.80.001   
 Target80%80%80%      
6Proportion of ASHA who have sufficient P.f. drugs on hand (per the current guidelines)Sundargarh1.51.27.96.43.40.0008.44.80.001
Mayurbhanj0.55.83.22.72.30.0214.74.30.031
Nabrangpur0.46.09.79.24.70.00011.25.90.000
Kandhamal (c)5.83.8−2.03.6NS   
 Target80%80%80%      
7Proportion of ASHA who know the waiting time prior to reading the RDTSundargarh81.887.291.69.85.70.0120.912.1NS
Mayurbhanj88.089.193.95.84.60.012−3.19.7NS
Nabrangpur82.775.281.6−1.17.8NS−10.011.6NS
Kandhamal (c)65.174.19.08.50.04   
 Target80%80%80%      
8Proportion of ASHA who know the correct ACT dose to treat P.f.Sundargarh66.771.586.019.36.80.000−17.4−9.90.001
Mayurbhanj43.088.292.549.66.70.00012.89.90.029
Nabrangpur41.965.370.829.010.00.000−7.812.4NS
Kandhamal (c)14.150.936.77.20.000   

The change in coverage proportion between surveys one and three ranged from −35.9% (95% CI ± −26.2%)(Nabrangpur, Indicator 5) to 49.6% (95% CI ± 6.7%) (Mayurbhanj, Indicator 8). Of the 32 within district changes (8 indicators × 4 districts), 23 (71.9%) showed a significant positive change in coverage from survey one to three. These changes included eight of eight prevention indicators, two of eight treatment-seeking behaviour indicators and 13 of 16 community health worker indicators. Significant change was detected in both intervention and control districts.

In DiD tests, intervention districts had significantly greater change in four of six (66.7%) prevention indicator comparisons with the control district, one of six (16.7%) treatment-seeking behaviour indicator comparisons, and four of 12 (33%) ASHA capacity indicator comparisons. Sundargarh and Mayurbhanj account for eight of these nine significant results; Nabrangpur accounts for one only. The control displayed greater improvement vis-a-vis Nabrangpur and Sundargarh for one indicator each among the ASHA capacity building indicators. At least one intervention district had a significantly greater change in coverage proportion compared to the control for five of the eight indicators.

For both prevention indicators (Indicator 1, Indicator 2), Sundargarh and Mayurbhanj exhibited improvements exceeding the control district. They displayed significantly more improvement for only one of the two treatment-seeking indicators (Indicator 3): people were tested with a RDT within 1 day of the onset of a fever and received the result. However, the data reveal no significant improvement for treating RDT+ve individuals with an appropriate antimalarial (ACT) (Indicator 4).

Sundargarh and Mayurbhanj displayed significantly larger increases than the control for one of four ASHA capacity indicators, namely, ASHA had sufficient ACT on hand to treat RDT+ve cases (Indicator-6); Nabrangpur also increased the value more than the control for this indicator, but this was the only occasion any of its indicators exceeded the control. Mayurbhanj also displayed one additional improvement as compared with the control district, namely, the proportion of ASHA who know the correct ACT dose to treat patients with a positive RDT result (Indicator-8). Interestingly, the control district, exhibited significantly more improvement vis-a-vis Sundargarh for Indicator-8; and Nabrangpur for Indicator-5 (the proportion of ASHA who have at least a 2-month RDT stock).

Indicator-7 (the proportion of ASHA knowing the waiting time prior to reading the RDT) was already a high value for all districts at time 1 and displayed no significant improvement by time 3.

Discussion

The analyses produced four main findings. Firstly, intervention districts, Sundargarh and Mayurbhanj, exhibited greater increases between survey periods time 1 and time 3, compared with the control district for 100% of the four prevention indicator comparisons, and 25% of the 12 health-seeking behaviour and ASHA capacity indicator comparisons. Secondly, the third intervention district, Nabrangpur, demonstrated significantly fewer improvements than other intervention districts. In only one of eight comparisons, that is measuring ASHA capacity, did Nabrangpur exceed the control. Thirdly, the control district exhibited greater increases relative to the intervention districts in only two instances: relative to Nabrangpur and Sundargarh. Fourthly, indicators concerning the supply chain (indicators 4–6) for RTDs and ACTs exhibited the poorest performance relative to the control with only 33% (three of nine comparisons) displaying a significant positive effect. This last finding is particularly important and may signal basic health system weaknesses; strengthening supply chain management in the periphery is a main aim of supervision. Odisha managers ought to examine why MTS were not able to improve logistics.

Although the trial was not designed to systematically study the role of good management, the findings concerning Nabrangpur suggest that the coverage figures for key indicators improved more in the presence of access to both LQAS information and committed managers in Sundargarh and Mayurbhanj; both had worked in those position at least 3 years prior to the study. Results suggest that in this study, locally produced LQAS data were effective when managers able to use the findings were present in the districts. Our interpretation is also due to the Nabrangpur's consistently lower coverage for indicators despite being an intervention site as compared to Sundargarh and Mayurbhanji. During the trial, the Nabrangpur District Malaria Officer (who had been in his position <1 year) was unable to formulate an acceptable management response to his district's low indicator results. This deficiency may explain the higher dropout rate of MTS during year 1 (60%). With their departure, no personnel were in place to use LQAS results.

By contrast, Odisha's VBDCP praised the control district's manager. Although he had been in his post <1 year, he had worked in other health sector management positions for a long time. Also he regularly visited the field to support MTS in problem solving. However, without data he could not perform as effectively as the two managers in Sundargarh and Mayurbhanj who had access to LQAS results.

The variation in intervention district performance across the three intervention categories is potentially important. The data suggest that the two districts with effective managers performed better than Nabrangpur for the prevention interventions and treatment-seeking interventions. They performed marginally better for ASHA capacity indicators. We know that during early 2010, shortly after dissemination of the baseline survey results that Odisha VBDCP managers initiated several system changes. For example, they prioritized low performing B-CHC for focused LLIN distribution, IRS spraying, and supply chain management reforms. They also, institutionalised LQAS surveys at the district level for evidence-based decision-making. However, these and other changes were effective in some but not all districts. This variation may suggest that in the study area strong central leadership ought to be complemented by equally strong district and B-CHC managers. However, the variation among indicator types may also be due to managers not being able to address all problematic areas simultaneously.

After the baseline survey, NVBDCP prioritised the prevention indicators. As IRS and LLIN coverage took place within a vertical programme, problems were addressed by a combination of strong state leadership and district responsiveness. Sundargarh and Mayurbhanj managers responded more effectively than Kandhamal, possibly because they had the LQAS data to target problematic B-CHCs. However, the Nabrangpur manager remained non-responsive as he had poor relations with MTS and with Medical Officers In-Charge in B-CHC. Kandhamal's managers reported a larger proportional change compared to Nabrangpur although this difference was not significant.

Between the two treatment-seeking behaviours, Indicator 3 (the proportion of people with fever in the last 2 weeks whose blood was tested with an RDT within 1 day of the fever and told the result on the same day as the test) displayed a significant increase over three time points whereas, Indicator 4 (the proportion of people with fever in the last 2 weeks who were RDT+ve and received ACT) did not. The difference between the two indicators is that the former requires district and B-CHC managers to train ASHA to correctly use RDTs, and maintain adequate supply chain management for RDT provision, as well as good supervision to monitor their appropriate use. Indicator 3 increased in Sundargarh and Mayurbhanj, but not in Nabrangpur and the control district. This deficiency may be because district managers were not engaged in using the LQAS data or in mobilising B-CHC staff. Indicator 4 is controversial as the use of ACTs had two systemic problems: Firstly, ASHA were newly introduced into the malaria programme for diagnosis and treatment of uncomplicated malaria cases. ACT was not adequately supplied from higher levels at the time, as a supply chain had not linked Federal state and district health systems. Secondly, not all B-CHC clinicians supported ASHA using ACT; during 2010, for example, some clinicians restricted ASHAs from using ACT. Once informed, NVBDCP began rectifying the problem.

Earlier research in Peru (Lanata et al. 1990), albeit not organised as a trial, also concluded that LQAS usage increased coverage by alerting programme implementers to underperforming areas. In that research, immunisation coverage increased from 72% (59–85%) to 88% (84–92%) over 3 months when LQAS was used promptly after each immunisation campaign. However, the presence of LQAS results alone does not produce improvement. Only results combined with an understanding of performance problems and timely corrective action foster change. When LQAS was integrated into community-based health care in Nepal to measure maternal and child health programmes, the programme displayed a significant improvement across all indicators over four time points (Valadez & Devkota 2002). A key part of the LQAS process highlighted in that project was a semi-annual planning meeting to share results across SAs and jointly solve problems. Other longitudinal examples exist in Nicaragua (Valadez et al. 2005), Costa Rica (Valadez et al. 1995), and Uganda (Sprague et al. 2012).

While the results of this trial cannot be generalised to other settings, when considered with other longitudinal applications of LQAS, they suggest the following testable hypothesis: managers’ use of LQAS data to prioritize problems and formulate programme solutions are associated with improvements in health indicators.

Other non-LQAS related research conducted among mid-level health managers (Loevinsohn 1994) in an unnamed developing country revealed a serious lack of confidence and skill in the analysis and use of data; only 24% of the respondents in that study could identify their best and worst performing districts for a specific indicator. The conclusions suggested that in addition to appropriate training, data must be presented simply and managers need an enabling environment promoting the routine use of data. Other research in India (Moreland et al. 2010) exploring barriers to effective data use found the majority of respondents not having timely access to survey results and limited computer skills or access to this technology.

Lot Quality Assurance Sampling is a process and a classification method (Pagano & Valadez 2010); health workers are trained to analyse and interpret data to identify substantially underperforming SA. Using these classifications they can develop plans to address SA problems while maintaining the programme intervention in areas classified as reaching targets. This process removes the need for computer data entry as hand tabulation is often preferred by local health workers, and allows prompt local use of results. In this trial the total per district cost for this use of LQAS per round, assuming a district with 10 B-CHC, was approximately $4000 (2011 US$) (including: training, data collection and hand tabulation); this cost does not include international technical assistance.

Although LQAS has been used for more than 20 years in health sector M&E, this study is the first trial of LQAS that indicates benefits obtained by competent managers in Odisha using LQAS data to make strategic and tactical changes to their programmes (also see (Myatt et al. 2003)).

Following the first-round LQAS surveys, intervention district results were shared with the state VBDCP who, acting upon the low coverage, improved the IRS and LLIN activities which they rolled-out to all districts, thereby contaminating the control. Further, in 2009 Kandhamal reported the highest number of deaths, thereby instigating NVBDCP to take precautions during 2010 by supplying higher quantities of LLINs, intensifying IRS, and intensifying early diagnosis and treatment by ASHA.

An important limitation is the inclusion of only four districts in the trial. But in the case of India, using few districts may be inevitable given the nature of the research question, which involves assessing district level use of LQAS results. The four districts comprise a population exceeding five million people. Although the population is enormous, the few districts used in the design lead us to treat this study as a proof of principle; we do not generalise the results to other settings. Yet, while we are cautious about our statistical findings, LQAS did unmask a range of system weaknesses the VBDCP found useful such as: programme improvements were dependent on the district manager, the MTS workforce was unstable, supervision of district supply chain management was ineffective, some clinicians working at the B-CHC level undermined the strengthening of curative services by restricting ASHA from treating P.f. malaria with ACT.

Another concern was the weak district leadership and attrition of MTS in Nabrangpur, which the State attributed to its distant location away from the State headquarters, and fewer health facilities. While these characteristics are a limitation, these deficiencies also indicate the importance of good management for successful use of M&E information. The study duration of 12 months was a limitation. A longer intervention may have resulted in larger intervention impacts and produced more stable trend data than this short study.

Conclusion

The study suggests that when functioning programme management was present in an intervention district in this trial, the use of LQAS for M&E was associated with programme strengthening and improved outputs and outcomes; in eight of 16 comparisons with the control district Sundargarh and Mayurbhanj had significant improvements. This result needs replication in a larger study and in diverse settings, due to the small number of districts used in the current study. By design, LQAS data inform local health. Using classifications, programme staff can quickly identify underperforming areas and, through structured problem analysis with committed district and state leaders, achieve improvements. These improvements can be made while maintaining areas that are classified as not currently a priority. However, state leadership was also essential to address systemic problems impeding the programme, especially for the prevention interventions.

By 2013, Odisha's LQAS application had become a national showcase despite the need for many improvements (Hussain et al. 2013). Representatives from low resource countries visit Odisha to learn about its integration of LQAS into the routine work of field workers. LQAS activities expanded to all 21 high malaria-burden districts (population = 26 116 630). In India, Odisha VBDCP managers have a unique capacity to compare results within and across districts, and assess state performance relative to other states, making it a standard for excellence. This may serve other states or countries, which can learn from the Odisha experience. While this is a small study (covering a large population), it revealed systemic problems within Odisha –problems that can potentially be resolved by a confluence of efforts by state, district and B-CHC managers. Although this trial produced results that were relevant to Odisha, we do not generalise them to other settings. Rather we recommend replication of this trial in other settings with a larger number of districts.

Acknowledgements

We are grateful for the guidance and support of Anu Garg without whom this project would not have been possible. We also are grateful to our colleagues at DFID including Billy Stewart, Jyoti Tiwari and Manjula Singh for their continuous support. We also thank Michelle Stanton for her highly useful comments on earlier version of the manuscript. We are also deeply grateful to Marcello Pagano and Caroline Jeffery for their essential statistical advice and support. This work was supported by UK aid from the Department for International Development. However, the content does not necessarily reflect that of DFID or official UK government policy.

Ancillary