Improving community health worker performance through supportive supervision: a randomised controlled implementation trial in Pakistan

To assess the effect of enhanced supportive supervision of lady health workers (LHWs) by lady health supervisors on integrated community case management of childhood pneumonia and diarrhoea.


INTRODUCTION
Diarrhoea and pneumonia are two leading causes of childhood mortality and morbidity among children under 5 in Pakistan (1). The World Health Organization (WHO) and the United Nations Children's Fund (UNICEF) recommend management of diarrhoea and pneumonia by community health workers (CHWs) to increase the access to care (2). CHW-driven integrated community case management (iCCM) can substantially reduce childhood mortality as evidenced in Niger (65% reduction) and Ethiopia (67%) between 1990 and 2012 (3,4).
Pakistan's under-5 mortality declined from 117/1000 live births in 1991 (5) to 89/1000 by 2012 (6): a reduction of 23%. Pakistan's Program for Family Planning and Primary Care (also called the Lady Health Worker Program [LHWP]) was initiated in 1994 and covers 60% of the rural population. The LHWP's mandate includes iCCM of childhood diarrhoea and pneumonia. Still, only 42% of under-5 Pakistani children receive antibiotics for pneumonia, and only 42% receive either oral rehydration solution (ORS) or recommended home fluid for diarrhoea (6). An independent evaluation of the LHWP showed skill deficits among lady health workers (LHWs) and their supervisorslady health supervisors (LHSs)-and concluded that improved supervision could lead to better LHW performance (7).
Maintaining iCCM skills among CHWs is challenging in the low-and middle-income country context. Infrequent training and refresher sessions lead to deterioration of acquired knowledge and skills (8), and dysfunctional supervision systems fail to maintain competencies (9). Finding more efficient and effective ways of supervising CHWs has emerged as a crucial area of interest for global iCCM implementation. Additionally, variable results from similar iCCM interventions underline the need for more in-depth reporting of contextual factors (10).
This study was conceived in a national health research priority-setting workshop for researchers and policymakers, supported by the Ministry of Health, Pakistan, WHO and the Alliance for Health Policy and Systems Research. We report here the effect of improved, structured supportive supervision of LHWs by LHSs on iCCM of pneumonia and diarrhoea in children under 5 years of age.

Study design and setting
Nigraan (named for an Urdu word meaning supervisor), a cluster randomised controlled trial, was conducted in Badin district in Pakistan (11). Badin is predominantly rural, spans 6726 km 2 and has a population of 1.35 million. A total of 1100 LHWs work in Badin to provide maternal and child health services at community level supervised by 34 LHSs. Primary health facilities include 11 rural health centres and 34 basic health units. Secondary-level facilities include one district headquarters hospital and four taluka headquarters hospitals at subdistrict level. A large number of private care providers also exist in the district.

Lady Health Worker Program supervision system in Badin
Each LHW is responsible for 150-200 households, and each LHS supervises 25-30 LHWs. An LHS is supposed to pay a supervisory visit to each LHW at least twice monthly. During this visit, the LHS observes the LHW performance and provides verbal feedback. LHWs report case data to their LHSs in a monthly meeting. Each LHS compiles these data and sends it to the district LHWP.

Nigraan intervention
Nigraan studied the effect of enhanced supportive supervision by LHSs on iCCM of childhood diarrhoea and pneumonia by LHWs and care-seeking practices by family caregivers (primarily mothers).
All LHSs regularly conducting and reporting field visits in Badin were eligible for inclusion in this study. The intervention consisted of i) training intervention arm LHSs to enhance their clinical, mentorship and supervision skills and ii) written feedback to LHWs during field visits. Comparison arm LHSs received standard LHWP refresher training. For real-time communication between LHWs and LHSs, all 34 LHSs were given simple mobile phones and a small communication allowance (Box 1a).

Lady health supervisor training
Nigraan trainers provided two days of refresher training to all 34 LHSs on iCCM of diarrhoea and pneumonia based on the existing LHW curriculum. Intervention arm LHSs also received an additional four days of training to improve their clinical and supervisory skills, including provision of verbal and written feedback. A detailed manual describing the training content and pedagogies was created. The intervention complemented face-to-face learning with hands-on training in a hospital setting. Simulation exercises, audiovisual aids and role-plays were used to reinforce skills.

Improved communication between LHWs and LHSs and case surveillance
A short message service (SMS) was added within the existing management information system of the LHWP to track diarrhoea and pneumonia cases. Upon case identification, LHWs informed LHSs so that a supervisory visit could take place, preferably within 24 hours. During these visits, LHSs reviewed LHWs' clinical findings and case management plans. The surveillance system tracked case reporting and LHS follow-ups. The SMS-based communication system was also used for the identification of cases for field-based skill assessments.

Implementing enhanced supportive supervision and performance feedback
In the intervention arm, three tools were used during supervision visits to increase the direct observation of LHW performance (Box 1b). Intervention arm LHSs provided written feedback to LHWs using a structured written feedback card. LHSs in both arms provided verbal feedback as per LHWP specifications. Written feedback provision added approximately ten minutes to each visit's duration.

Sampling and data collection
Overall sample size An overall sample size was calculated of 17 LHSs per arm to achieve 80% power to detect an increase of 15% among the different parameters of iCCM skill levels between 30% and 70%. The test statistic used was the two-sided Z-test (unpooled). The significance level of the test was 0.05. A total sample of 2040 households in each study arm was thus required.
Sample size for household cluster survey A cluster was defined as the coverage area of one LHS. We randomised 34 LHSs: 17 each into intervention and comparison study arms. Of the 25-30 LHWs working under each LHS, five were randomly recruited, to total 85 LHWs in each arm. Assuming that (on an average) each LHW covers about 100 households, this provided an estimate of 8500 households in each arm. Randomly 64 sampling every fourth household resulted in surveying 2125 households in each arm. Thus, 5 LHWs/LHS provided the right sampling frame to come up with the required estimate of 2040 households. The slight oversampling was conducted to account for refusals and incomplete information.

Skill assessments
In each assessment, information from surveillance data was used to randomly choose 25 LHWs for diarrhoea and 25 for pneumonia skill assessments from the cases reported in each arm.
To select LHWs for skill assessment, a log of cases reported by designated LHWs in the last 24 hours was created. All reported cases were given unique IDs and entered into Microsoft Excel, and reporting LHWs were randomly chosen for skill assessment. To ensure 25 unique LHWs for diarrhoea and 25 for pneumonia in each assessment period, no LHW was assessed more than once within one disease category. LHW iCCM skills and supervisory skills of their respective LHSs were assessed in the field during follow-up visits.
Skill assessment scorecards An iCCM Clinical skill assessment scorecard designed according to the LHWP curriculum was used to assess history taking, physical examination, diagnosis, treatment and counselling skills of LHWs and LHSs (Table S1).
A supervision skill assessment scorecard assessed LHSs' supervision and mentoring skills. The clinical mentoring section assessed LHSs' ability to evaluate LHWs' clinical skills. A supervision section scored each LHS based on the provision of verbal and written feedback and her demeanor towards the LHW (Table S2).

Baseline and endline surveys
Household survey questionnaire Baseline and endline household surveys were conducted utilising a structured questionnaire adapted from the 2012-2013 Pakistan Demographic and Health Survey (6). Data from family caregivers regarding their knowledge and practices during an episode of diarrhoea/pneumonia in their children under 5 were collected. A module gauging perceptions and practices related to the LHWP was included.

Qualitative research
In-depth interviews and focus group discussions were conducted with various stakeholders (community caregivers, policy-makers, LHWs and LHSs) to understand the context and effect of the Nigraan intervention.

Ethics, consent and permissions
The ethical approval was obtained from the Aga Khan University Ethical Review Committee and the WHO Research Ethics Review Committee. Written informed consent was obtained from all individual participants in the community. The informed consent statement was read out to illiterate individuals, comprehension was assured, and thumbprint was taken for those who consented to participate. The process of consent for all community participants was witnessed (signatures of witness obtained) by a member of the community trusted by the participant. A formal agreement was made with the provincial government for the involvement of LHWs and LHSs in the research. An informed written consent was obtained from each LHS and LHW for the involvement at the beginning of the research.

Data management and analysis
Quantitative data were analysed using IBM SPSS version 19. Descriptive statistics were calculated for knowledge and skill levels of LHWs and LHSs. For sociodemographic data (highlighting education, years of experience, etc., of health workers), descriptive statistics were calculated and differences between intervention and comparison groups assessed using various tests of statistical significance. Descriptive statistics were also calculated for household survey data.
Qualitative data were audio-recorded, transcribed and translated into English and then analysed by the thematic content analysis using QSR NVivo version 10.

RESULTS
LHSs and LHWs in the two study arms were comparable in most sociodemographic characteristics except for a higher median age in comparison arm LHWs and LHSs and higher median years of experience in comparison arm LHSs (Table 1).
Lady health worker diarrhoea and pneumonia iCCM performance Lady health worker pneumonia iCCM performance All indicators of LHW performance (history taking, danger sign assessment, examination, diagnosis, management and counselling) for iCCM of pneumonia showed marked improvement between the first and second assessments. Even though the comparison group also showed better performance, the improvement was more prominent in intervention LHWs. However, in the third assessment, six months after the end of the intervention, comparison group performance also improved to the extent that there were few differences between the two groups. Additionally, intervention LHW performance deteriorated between the second and third assessments for measuring respiratory rate correctly (52% [n = 25] vs 44% [n = 25]) and correct disease classification (64% [n = 25] vs 44% [n = 25]) ( Table 2).
Lady health worker diarrhoea iCCM performance Intervention arm LHW performance for diarrhoea iCCM improved considerably between the first and second assessments, while comparison arm LHW performance showed little improvement for most indicators. In the third assessment, intervention LHWs performed a great deal better than comparison LHWs in assessing dehydration (92% [n = 25] vs 64% [n = 25]) and in classifying the disease correctly (68% [n = 25] vs 40% [n = 25]). The difference between intervention and comparison groups diminished in the third assessment because the comparison group improved in some indicators. The performance of LHSs in both arms regarding correct treatment advice deteriorated in the third assessment compared to the second ( Table 2). Monthly house-to-house and catch-up polio campaigns (inactivated polio vaccine to be added soon) All Expanded Program of Immunization vaccinations Measles and tetanus crash vaccination campaigns Tuberculosis-Directly Observed Treatment-Short Course Biannual deworming campaigns Diarrhoea and pneumonia community case management in children under 5 Contraceptive counselling, provision and referral for intrauterine contraceptive devices and tubal ligation Referral of pregnant women for danger signs and antenatal check-ups Reproductive and nutrition education and vaccination of unmarried girls and women Consultation and referral for gynaecological problems Screening, referral and follow-up for severe-and-moderate acute malnutrition Other field activities such as iodine survey and so on.

66
Lady health supervisor diarrhoea and pneumonia iCCM performance Lady health supervisor pneumonia iCCM performance Substantial improvement in intervention arm LHS performance was observed for the assessment and classification of pneumonia between the first and second surveys. For instance, correctly counting respiratory rate (18% [n = 16] to 94% [n = 16]), correctly classifying the illness (0% to 62%), referral for correct indications (0% to 62%) and explaining signs that should prompt the caregiver to seek further care (0% to 81%) improved. Correctly detecting the presence or absence of chest indrawing reached 100%. In the post-intervention assessment, the proportion of intervention LHSs performing these indicators correctly was maintained or increased (correct respiratory rate 80%; chest indrawing 100%; correct illness classification 100% [n = 15]). Performance was similarly maintained in the comparison arm for most skills.
Lady health supervisor diarrhoea iCCM performance Lady health supervisor iCCM skills for diarrhoea management for both intervention and comparison arms improved considerably between the pre-intervention and second assessments and were sustained in the post-intervention assessment. In the pre-intervention survey, only 6% (n = 16) of intervention arm LHSs assessed the sick child for dehydration, which improved to 100% in the second (n = 16) and post-intervention (n = 15) assessments. In the intervention arm, 62% (n = 16) of LHSs correctly classified the illness in the pre-intervention assessment. This proportion rose to 93% (n = 16) in the second assessment but dropped to 73% (n = 15) six months postintervention. In the comparison arm, in the first, second and third assessments, 50% (n = 16), 93% (n = 16) and 87% (n = 16) of LHSs correctly classified the illness, respectively.
Lady health supervisor supervisory performance Among intervention arm LHSs, supervisory performance showed steep improvement between the first and second assessments. This improvement was maintained for most indicators post-intervention. Supervisory performance of comparison arm LHS also improved between the first and second assessments, although to a lesser extent than that in the intervention arm, and continued to improve in the postintervention assessment. Pre-intervention, no LHS provided written feedback. In the second assessment, the proportion of LHSs providing this feedback in the intervention arm increased to 76% (n = 21), while only one (5% [n = 21]) comparison arm LHS did so. Although intervention LHSs still performed better than comparison LHSs (16% [n = 25] vs 0% [n = 25]) in the post-intervention assessment, there was a substantial reduction compared to the second assessment. Almost equal proportions of intervention and comparison LHSs provided verbal feedback to their LHWs at the postintervention assessment (80% [n = 25] vs 76% [n = 25]) ( Table 3).
In the post-intervention assessment, more intervention Family caregiver knowledge and practices for pneumonia and diarrhoea care and perception of LHWP In the endline survey, 85% (intervention n = 2182; comparison arm n = 2197) of caregivers in both arms recognised loose stools as a sign of diarrhoea. A larger proportion of caregivers in the intervention arm (50%, n = 2182) recognised breathing difficulty as a key symptom of pneumonia compared to 47% (n = 2197) in the comparison arm. Home-made ORS was given to children with diarrhoea by 51% (n = 667) of intervention arm caregivers compared to 45% (n = 697) in the comparison arm.
In the baseline survey, 71% of intervention arm (n = 3494) and 72% (n = 3531) of comparison arm caregivers reported that an LHW had visited their household in the last 30 days. When asked about services provided during the visit, caregivers most frequently reported polio vaccine administration (79% intervention arm [n = 3494] vs 78% comparison arm [n = 3531]) and vaccination (34% intervention arm [n = 3494] vs 32% comparison arm [n = 3531]). This finding corroborates LHWs' statements in qualitative interviews that 'we are best seen as polio workers' (12). Less than 1% of caregivers in both arms (intervention n = 3494, comparison n = 3531) said that LHWs provide ORS, while less than 2% said that the LHWs provide any medicines. Caregiver perceptions about LHW responsibilities did not improve in the endline survey. *LHWs did not have antibiotics or ORS packets most of the time during the intervention. LHWs educated family caregivers on how to make home-made ORS.

68
Given these findings, it is not surprising that the utilisation of LHW services for diarrhoea and pneumonia management in children under 5 remains low. In the baseline survey, only about 16% (n = 3494) of intervention arm and 21% (n = 3531) of comparison arm caregivers considered LHWs capable of providing diarrhoea and pneumonia care. Less than 1% of caregivers in either study arm reported utilising LHW services for the two conditions. Demonstrating marginal improvement, in the endline survey, 18% (n = 2182) of intervention and 23% (n = 2197) of comparison arm caregivers considered LHWs competent to provide diarrhoea and pneumonia iCCM. However, this apparent trust in LHWs' ability does not compel caregivers to seek care from them. Even though in the endline survey, 12% (n = 2182) of intervention and 17% (n = 2197) of comparison arm caregivers said that they would prefer LHWs as the first point of contact for diarrhoea and pneumonia care in children, less than 1% chose to do so when their child was sick. Surveillance data show that at 72-hour follow-up, only 3% (n = 2500) of cases were recovering after treatment prescribed by LHWs; the rest had sought care from other sources (13).

DISCUSSION
Our results show considerable improvement in LHW and LHS diarrhoea and pneumonia iCCM performance between the pre-intervention and second assessments, especially for diarrhoea. At six months post-intervention, most skills showed stability or continued improvement. LHS clinical mentoring and supervision skills also improved, particularly in the intervention arm.
Without regular training and refresher courses, CHW skills deteriorate quickly (8). Outside the intervention, LHWs and LHSs in Badin have not had diarrhoea and pneumonia iCCM training in over 10 years. This could explain the low level of baseline skills as well as the substantial improvement in response to training. Previous studies have shown that training health workers considerably improved knowledge about childhood acute respiratory infections (14,15) and accuracy of diarrhoeal disease diagnosis (16).
As there were no other relevant interventions in the study district, improvement in LHW knowledge and skills was likely a consequence of enhanced LHS supervision and increased interactions with LHWs. This finding accords with research, which suggests that supervision and feedback significantly improve CHW performance (9). Higher quality of supervision contributes to better CHW performance just as much as increased frequency of supervision (17). Therefore, improving the clinical skills of LHSs along with more frequent field visits where they could repeatedly demonstrate correct clinical practices was likely to have improved LHW skills (9). Similar performance trends in the two study arms may be due to a number of reasons. First, in the LHWP's administrative set-up, all district LHSs gather monthly for a collective performance appraisal meeting. These meetings may have allowed exchange of information between intervention and comparison LHSs. Second, LHWs and LHSs working in contiguous areas often know each other and could have discussed the enhanced supervisory skills. Third, the refresher training for comparison LHSs could have improved performance as it came after many years. Fourth, LHSs and LHWs often carry out their work in communities without much supervisory support and attention from the program. Repeated observation and assessment could have motivated comparison arm LHSs and LHWs to refresh their knowledge using the LHWP resources available to them, for example LHW curriculum and iCCM management flow charts. A number of logistic and management factors prevented the achievement of greater impact of the intervention. LHWs provide many promotive, preventive and curative services (Box 1c), which require mastery of a broad range of skills which is challenging for LHWs, even more so when the program does not regularly provide training and refresher courses. Apart from their routine responsibilities, LHWs are involved in multiple ancillary community-level activities. For instance, LHWs routinely participate in monthly house-to-house polio immunisation campaigns. During these campaigns, they do not provide routine services, thus making their availability unreliable for the community. Policy-makers and LHWP managers admit that LHWs are overburdened and that this affects the quality of services they provide (18). Lack of functioning vehicles and fuel means that supervision visits often cannot take place; lack of tools such as feedback cards means that LHSs cannot provide written feedback; and lack of antibiotics for pneumonia and ORS and zinc for diarrhoea implies that LHWs cannot provide treatment. This situation diminishes their prestige in the community (18,19). Salaries are often delayed for months, which also demotivates LHWs. As LHWs often do not have medicines, they refer most of the cases they see. Nigraan surveillance data reveal that 76% of diarrhoea and 88% of pneumonia consultations with LHWs resulted in referral. During the intervention period, after the Nigraan team advocated with the LHWP to ensure supplies, LHWs were able to provide ORS to 23% of diarrhoea cases, but still antibiotics to only 2% of pneumonia cases (13).
Nigraan findings have crucial administrative and policy implications. If better LHS supervision ability and frequent supervisory visits can improve or maintain LHW skills, the LHWP could enhance LHW performance within existing resource constraints. Since the 2010 18th constitutional amendment, health services have become a provincial remit. The LHWP will be funded by the federal government until the end of 2017 with funding frozen at 2010 allocation levels. Afterwards, provinces will have to pay for the programme from their own budgets. Therefore, interventions that offer performance gains without substantial additional resources are more likely to be sustainably implemented. The Nigraan supervision curriculum, improved communication and case reporting, and more frequent supervisory visits can be readily implemented through efficient and transparent utilisation of existing resources. A number of funded mobile phone-based initiatives are being considered in the district by development agencies. Coordination with them can offset the cost of communication allowances for Nigraan implementation.
Finally, the LHWP can meaningfully provide improved management of diarrhoea and pneumonia only if caregivers utilise its services. Improvement in LHW iCCM skills and better logistic support, such as availability of medicines, will enhance the community's trust (20). Reactivation of languishing LHWP village committees could improve community engagement and awareness of LHW capabilities. If disruptions to routine LHW activities, such as frequent polio campaigns, are reduced, caregivers can rely on LHWs to be available when needed.
One of the key factors that facilitated this implementation research was integrating views of policy-makers and implementers in the process. The programme implementers were involved from the beginning to the end. The research question for this study was conceived by a combined team of researchers and policy-makers at a national-level research priority-setting exercise. Provincial and districtlevel policy-makers, particularly LHWP stakeholders, were involved in the process of framing the intervention and determining monitoring mechanisms. Field-based research staff worked from the district LHWP office. Trainings and meetings were also conducted in the programme office.
To maintain policy-maker buy-in of study findings, multiple dissemination meetings were conducted throughout the intervention period. Frequent transfers of higherlevel program officials presented a challenge. However, close working relationships with second-tier LHWP and health department officials allowed for more consistent engagement with policy-makers. As a result of this collaboration, LHWP officials agreed to incorporate the Nigraan training into the routine LHWP curriculum for LHSs. The process of that integration is under way.

LIMITATIONS
As all LHWs and LHSs worked under the same district administration, some spread of the intervention to the comparison arm probably occurred. Some inter-rater bias in skill assessment is also possible. Repeated assessments using the same tools may have resulted in inflated performance scores. The findings could be more generalisable if the Nigraan intervention were to be executed in more than one district.

CONCLUSION
Supportive supervision has the potential to improve LHW performance through robust communication, monitoring and accountability mechanisms. However, substantial improvement is unlikely without concurrent political will, training, logistics and smooth supply of commodities. A scale-up of the Nigraan trial is required to establish the impact of supportive supervision.

CONFLICT OF INTEREST
None of the authors reports any conflict of interest.

DISCLAIMER
Shamin A Qazi is a former staff member of the World Health Organization. The author alone is responsible for 70 the views expressed in this publication and they do not necessarily represent the views, decisions or policies of the World Health Organization.