Using the RE‐AIM framework to evaluate the implementation of integrated community case management in Kenya

To evaluate an integrated community case management programme for sick children aged 2 to 59 months in western Kenya using the Research, Effectiveness, Adoption, Implementation and Maintenance (RE‐AIM) framework.


INTRODUCTION
Globally, approximately 5.6 million children under the age of 5 years die annually (1). In sub-Saharan Africa in 2015, one in 13 children died before their fifth birthday compared to one in 189 children in high-income areas (2). Over one half of the deaths were due to infectious diseases such as pneumonia (17%), diarrhoea (10%) and malaria (10%) (1).
Appropriate treatment of these diseases involves use of oral antibiotics, oral rehydration salts (ORS) plus zinc and artemesinin-based combination therapy respectively to reduce mortality (3). Unfortunately the coverage of these well-known and essential interventions remains low due to limited access to and poor quality of health services, weak supply chain management, an inadequately trained and supported health workforce, provider shortages at health facilities, inadequate knowledge about interventions by clients, and failure to convert national policies into action plans (4,5). Integrating facility-and community-based health services using strategies such as integrated community case management (iCCM) can expand the capacity to treat children with poor access to health facilities in lowresource settings (6)(7)(8). iCCM is promoted by the World Health Organization (WHO) and United Nations Children's Fund (UNICEF) for settings where well-trained and supervised community health workers (CHWs) can provide prompt and adequate treatment for pneumonia, malaria and diarrhoea to help reduce morbidity and mortality among under-five children (9).
Despite the evidence demonstrating that trained and supported CHWs can increase access to appropriate management of pneumonia, diarrhoea and malaria, many countries, including Kenya, have not scaled up iCCM in the areas that could benefit the most (10). The Kenyan Ministry of Health (MOH) and WHO supported research priority-setting exercise in 2011 with Kenyan experts (academics, clinicians and policy-makers) identified iCCM implementation, scale up and sustainability as a high priority. The main purpose of the project was to demonstrate the feasibility and effectiveness of training CHWs and supplying them with ORS, zinc, rapid diagnostic tests (RDTs) for malaria, artemether-lumefantrin (AL) and amoxicillin to treat diarrhoea, pneumonia and malaria at the community level. We conducted this project in the government county health system, which at the time was only just being established.
This paper presents an evaluation of the implementation process of the iCCM programme in western Kenya using the RE-AIM framework (11). The RE-AIM framework facilitates the evaluation of the Reach, Effectiveness/Efficacy, Adoption, Implementation and Maintenance of an intervention (11). This framework is particularly useful in contributing to the understanding and learning about scaling up effective community-based health services such as iCCM (12). Without proper implementation of iCCM, it is likely that the anticipated impact of this promising intervention will be diminished and the results not easily generalisable.

Study design
This was a prospective observational study utilising quantitative methods. Owing to the large scope and size of the project, we implemented iCCM in a stepwise manner, which allowed us to make sure that all community units (CUs) in Homa Bay received the intervention. As stated above, we use the RE-AIM framework to evaluate the implementation process and outcomes as outlined in the Measures section. The RE-AIM framework allowed the flexibility to use data already collected within the organisational setting as well as through surveys before, during and at end of implementation (13).

Study setting
The project was conducted in Homa Bay County in western Kenya between December 2013 and February 2016. Homa Bay is a rural county of approximately one million people and has eight administrative sub-counties (Homa Bay, Kabondo, Kasipul, Mbita, Ndhiwa, Rachuonyo North, Rangwe and Suba). Children under 5 years account for 16% of the population. The county is characterised by high levels of poverty (14), and one of the highest under-five mortality rates in Kenya (15). Despite an adequate number of health facilities in Homa Bay, nearly one half of the population finds distance and poor road access serious barriers to health care. This situation especially applies to those living on islands in Lake Victoria, who must travel very long distances over water and land to reach health facilities.

Intervention description
The Kenya Medical Research Institute (KEMRI) conducted this iCCM research project in Homa Bay County from October 2013 to February 2016. The study trained, equipped and supervised unpaid CHWs in the county to: (i) identify and treat children aged 2 to 59 months with pneumonia, malaria and diarrhoea at home; (ii) refer severely ill children to health facilities for further timely management; and (iii) counsel caregivers on disease prevention (e.g. immunisations, family planning, nutrition and preventing HIV infection) and on the continued care of their sick children at home. As this was the first time CHWs were involved in curative services, particularly for pneumonia, they were trained on how to send a 'Please Call Me' text message to a hotline number to report all suspected pneumonia cases. Each CHW owned a personal mobile phone that was not provided by the project. The CHWs were taken through an initial accreditation process eight weeks post-training by Community Health Extension Workers (CHEWs) and health facility clinicians. The CHWs were supplied with treatment and tracking registers for documentation, timers for counting respiratory rate in children and RDT kits for malaria diagnosis. CHWs were reimbursed a fee of 500 Kenya Shillings (US$ 5) per month to facilitate their transport to the health facility to collect commodities (drugs and diagnostics) and submit reports. The project supplied all CHEWs with smartphones for purposes of data aggregation and transmission as well as commodity tracking.
Children aged 2 to 59 months with diarrhoea, fever and/or cough and/or difficult breathing identified during home visits by CHWs, or whose caregiver sought care from a CHW, were assessed, classified and treated. The CHW recorded the findings on a standardised sick child recording form adapted from WHO (16). Any child with danger signs of severe illness was referred as per the standard of care.

Implementation procedure
We implemented iCCM using a five-pronged approach: (i) sensitisation of national, county, sub-county, health centre and community teams on iCCM; (ii) training of CHWs, CHEWS, quality assurance (QA) nurses and other healthcare workers; (iii) commodity provision and accountability; (iv) field-and facility-based supervision of CHWs; and (v) monitoring and evaluation. Figure 1 depicts the implementation process and conceptual framework.
We used mobile phone technology to: (i) ascertain quality of care for pneumonia home case management; (ii) track commodity management and inventory; (iii) compile data; and (iv) monitor performance. We created a free 24/7 telephone-based decision support hotline that provided an effective way to listen to CHWs, verify the pneumonia diagnosis using signs identified by the CHW and link the pneumonia case to a trained QA nurse for physical case confirmation within 24 hours. The hotline utilised the widespread cellular phone coverage linked to a web-based platform, which aggregated all reports and displayed the information on a dashboard through which MOH at national, county, sub-county and facility level could access and review reports as well as track commodities.
Additional implementation details are available in the supplemental material. Table 1 presents RE-AIM dimensions, definitions, key outcome variables and the source of data.

Reach
To determine the participation rates and representativeness of children and their caregivers who benefited from accessed iCCM, we provide a description of the existing community health services Homa Bay county at baseline, the CUs and CHWs. This data were abstracted from the MOH CHW treatment and tracking register, CHW household register, CHEW supervision checklist and CHEW stock records. Other important sources of routine service delivery data included the county health management team (CHMT) supervision checklist and government databases on training. The key tools used for collecting this information are summarised in Table 2. The tools are available as part of the Kenya iCCM implementation framework (17) and monitoring and evaluation plan (18) guideline documents.

Effectiveness
We assessed the effectiveness of iCCM on the change in proportion of children under five years old who received early and appropriate (i) antibiotic treatment for pneumonia, (ii) management for diarrhoea and (iii) treatment for malaria from baseline. These were measured through a set of household surveys of mothers/caretakers of children who had experienced iCCM conditions in the previous four weeks. The surveys were cross-sectional, population-based surveys done at baseline (August 2013) and at the end of   study (December 2015) among a sample of 9124 households within 107 clusters. Sample size calculation was done to ensure that we had adequate number of sick children to evaluate. A sample size based on two-week pneumonia prevalence would yield the largest sample which would be able to accommodate the other illnesses under survey namely malaria and diarrhoea appropriately which have higher 2-week period prevalence than pneumonia. We made the following assumptions: (i) that there is no seasonality of pneumonia incidence and (ii) that the duration of pneumonia symptoms is one week on average. The two-week period prevalence was based on an estimated pneumonia incidence of 300 cases per 1000 children per year in low-and middle-income countries based on a systematic review of population-based cohort studies using standard case definitions consistent with the World Health Organization Integrated Management of Childhood Illness case definition of pneumonia (19,20) This translated to a 2week period prevalence of approximately 2%. We used multistage random sampling to arrive at clusters and women who participated in the surveys. The clusters were randomly drawn from all 8 implementation sub-counties and were the same at baseline and at the end of study. Women resident in the study area aged 15 to 45 years who had at least one child less than 5 years were and enrolled for this survey. Independent enumerators conducted the household surveys. Data were collected using passwordprotected smartphones. The phones had in-built logic and skip patters to enhance data quality. Descriptive univariate and bivariate analyses showing the impact of iCCM in Homa Bay during pre-and post-iCCM implementation was done.

Adoption
Adoption of iCCM was analysed by an analysis of the coverage of iCCM in Homa Bay postimplementation including the number of children who utilised and benefitted from implementing of iCCM.

Implementation
As proposed by Proctor et al. (21) and Dane and Schneider (22), we examined the fidelity of the implementation process as an outcome using four dimensions of: (i) adherence, (ii) coverage, (iii) details of content and (iv) quality of delivery. Fidelity of implementation is the degree to which a programme is implemented as intended (21). In order to assess the intervention fidelity, we designed a midline crosssectional survey eight months into implementation when all the CUs and CHWs had been trained, accredited and implementing iCCM.

Midline survey protocol and tools
The midline survey protocol and tools were adapted and appropriately modified from those developed by the Institute for International Programs -Johns Hopkins University (IIP-JHU) study team in Ethiopia (23).

Data collection
Six survey teams were deployed for data collection. Each team was made up of one supervisor and two data collectors. The two data collectors were separated into observers and re-examiners. Re-examiners were clinicians (nurses or clinical officers) who were trained on iCCM so they can carry out clinical re-examination of sick children for the assessment of quality of care. The survey involved directly observed CHW consultations with sick children for common illnesses (signs of pneumonia, fever/malaria, and diarrhoea) and danger signs, compared with a clinician trained in iCCM.

Sample size and sampling
To determine the sample size for the midline survey, we made assumptions of correct assessment, diagnosis and treatment prevalence of 50% and allowed an error margin of 8% at the 95% confidence level. Using these assumptions, the required sample size was 151 CHWs. We used multistage random sampling to arrive at the CUs and CHWs who participated in the survey. The number of CHWs selected from each sub-county was proportional to the total number in that sub-county. This ensured that all sub-counties, CUs and CHWs were given an equal chance of being selected. Sick children brought by their caregivers for treatment were screened to identify those with the following conditions: signs or symptoms of severe illness (change in consciousness/lethargy, convulsions, vomiting everything, not eating or drinking); fever/malaria; cough, fast/difficult breathing; pneumonia; diarrhoea/vomiting; nutrition or feeding problems.

Analysis
Continuous variables, such as the number of community activities carried out by CHWs in the last three months, were reported as either means and standard deviations or medians and interquartile ranges (IQRs). Binary measures, such as supervision or training in the last three months (yes/no), were calculated as proportions. All analyses were carried out in SPSS version 18.

Maintenance
We assessed maintenance by measuring the average posttraining knowledge level and retention rates of CHWs implementing iCCM in the different sub-counties at least 6 months from their initial training.

Ethical clearance
The study protocol was reviewed and approved by the KEMRI National Ethical Review Committee, Kenya National Pharmacy and Poisons Board and WHO. Monitoring of adherence to the protocols, including human subject's protection, was conducted by a designated KEMRI ethics officer not directly involved in the study as well as by an independent WHO monitor experienced in performing monitoring in low-resource settings. All serious adverse events were reported to the KEMRI National Ethical Review Committee and WHO. Community leaders and village headmen were approached through the CHCs and briefed on the project, and their verbal consent and approval solicited. Written consent in the preferred local language (Dholuo or Swahili) was obtained from caregivers of children who were followed up after receiving treatment from the CHWs to allow collection of data to assess the effectiveness of the intervention.

CHW training
One half of the CHWs had been recruited and trained within the four years preceding the survey (IQR: 2-8). The median number of months since completing iCCM training was six (IQR: 5-7), and 83.4% (126/151) of CHWs reported having received follow-up training six weeks after their iCCM training.

CHW time commitment
CHWs reported spending a median of 3.5 hours per day (IQR: 1-6) providing clinical services in the community and three hours (IQR: 2-5) conducting community education per week in the month preceding the interview. Most CHWs (73.5%; 111/151) visited their health facilities more than five times in the three months before the survey.

CHW supervision
A majority of CHWs (94%; 142/151) reported having received at least one supervisory visit in the three months preceding the survey, mostly from CHEWs (40.1%; 57/ 142), nongovernmental organisation supervisors (16.9%; 24/142) and supervisors supporting the linked health facilities (16.2%; 23/142). Over 91% (138/151) of the CHWs said they had attended iCCM performance review and clinical accreditation meetings organised by the CHEWs and linked health facilities at least once in the last three months.

Quality of care
Most of children's presenting conditions were correctly classified by CHWs, meaning their assessments were concordant with a clinical re-examiner's classification as the gold standard. Based on this assessment criterion, over 80% of pneumonia and diarrhoea and over 95% of malaria cases were correctly classified. Overall correct classification of target iCCM diseases, regardless of their nature, was over 85% (Table 4).
Nearly one-third (35%; n = 434/1241) of the children seen by the CHWs were referred to the linked health facilities for further management. About 80% (n = 347/434) had danger signs (66%; n = 229) or the CHW did not have the indicated drugs (34%; n = 118). Nearly 90% (n = 386/ 434) of those referred were given notes to present to the health facilities where they were referred. In over one half (n = 241/434) of the cases, transport arrangements were made for the caregiver. Data on compliance with referral were not collected.

Counselling of caregivers
Except for vitamin A, which was administered at the facility, the majority of caregivers received demonstrations on how to administer the medicines given. For instance, 86.6% (n = 131/151) of the CHWs demonstrated to the caretakers how to administer amoxicillin and 67.6% (n = 102/151) AL. However, only 50.2% (n = 76/151) asked the caretaker to repeat back the instructions to assess their understanding. Over 60% (n = 91/151) of the CHWs asked the caretakers to administer the first dose of medicine to the child before they left the health facilities.

Maintenance
Knowledge retention post-training The mean score of the pretest examination of the CHWs before training was 54.7% (n = 2604). This increased to 73.7% after the first training (n = 2604). At accreditation at eight weeks (n = 2558) and six months after the training (n = 2432), the mean score remained 73.3%.

DISCUSSION
Good implementation is a prerequisite for intervention effectiveness (21). This paper describes the implementation process and outcomes of an iCCM programme in one county in Kenya as part of countrywide implementation. One of the documented challenges to access and coverage of wellknown essential interventions for pneumonia, malaria and diarrhoea is the failure to convert national policies into action plans (4,5). This study, therefore, addresses an important iCCM knowledge implementation gap. By using RE-AIM metrics, we demonstrate that when CHWs are trained, supervised and supported, they are able to have adequate knowledge, skills and competency to provide and increase coverage of quality services including treatment of sick children. These results are similar with previous findings in programmes implementing iCCM in Ethiopia, Pakistan and Bangladesh. Bari et al. (24) found almost similar findings in a community in a cluster randomised controlled trial in Haripur District of Pakistan when Lady Health Workers were asked to treat children with pneumonia at home. The role of CHWs in promotion of health at the community level is well known (25)(26)(27). CHWs can play a critical role in an over-burdened healthcare system, filling the information and distribution gaps between people who need care and the health facilities that provide a range of health services to large populations (28). CHWs cannot effectively perform iCCM without adequate training, robust efforts to retain human resources, uninterrupted drug supply, supportive and quality assurance supervision and active involvement of the communities in which they work (29,30). There also should be mechanisms of performance review and clinical accreditation of skills after basic training in order to produce and maintain competency among CHWs. Accreditations should assess knowledge of different components of the iCCM algorithm, using case-based questionnaires, simulations, videos with physiological readings or similar tools as practical predictors of the quality of individual CHW performance. These methods of evaluation have been used successfully to improve frontline healthcare worker competence in management of sick children (31). Accreditations are not only important for monitoring and standardising CHWs' performance, but also for providing mentorship and encouragement to them. They can also be valuable for establishing the minimum number of cases CHWs need to see to maintain competency. Friedman, for instance, detailed the importance of mentorship in assisting South African CHWs carry out their work effectively in the communities (32).This frequent contact with the CHWs coupled with affirmation of competency serves as a strong nonmonetary incentive, which is important for their retention within the programme as well as fidelity to programme implementation. Health facility-incharges participated in monthly community dialogue days and provided feedback to CHWs on cases they had received and interpreted service statistics. We believe these strong facility-community linkages are necessary to improve the quality of services provided and to foster trust between health facility providers and CHWs.
The RE-AIM framework allows policy and programme managers to divide the implementation and its evaluation into five separate components, providing focus on the most important aspects of implementation for sustainable adoption (12). While many resource-poor settings now have policies on iCCM, there is still a resistance to scaling up iCCM (33). This resistance to scaling up of iCCM is not an apathy to what iCCM promises, but rather reflects the fact that health system effects of iCCM are broad ranging, requiring strategic analysis and resourceful management; skill sets that are often not readily available in low-resource settings (34). In 2012, the global iCCM Task Force developed a set of 47 iCCM benchmark indicators to guide implementers to improve monitoring and evaluation (M&E) systems for iCCM (33). Unfortunately, some indicators may not fit country priorities, and others may be difficult to collect due to the fragmented nature of evaluation systems or increased burden of reporting (35). The RE-AIM framework as outlined in this paper can be used to identify metrics and indicators that are readily available from public sources (e.g. treatment and tracking registers, sick child recording), and identify means by which behaviours can be tracked routinely and efficiently (e.g. routine demographic household health surveys, government training databases). Each dimension of the RE-AIM framework can be used in planning the components of iCCM that span the different health system blocks, and outline various steps that managers can follow throughout the course of design, implementation and evaluation.
This project was not without challenges. First, the study was rolled out at around the same time that the national health system had been devolved to the county level. Neither a national nor a county level implementation strategy and monitoring and evaluation framework for iCCM existed at the time. The project worked with the MOH, the national iCCM technical working group and the County Health Management Teams to develop these tools. Despite these challenges, we feel that this precise lack of health system infrastructure for iCCM implementation and evaluation are ubiquitous in many low-resource settings in sub-Saharan countries and that this paper provides generalisable approaches to implementation and results. Secondly, the success of iCCM could not be assured without proper functioning mechanisms of essential commodity and job aids provision to CHWs. Although the project leveraged mobile phone technology to track commodities and notify stock-outs and reorder levels, a significant proportion of CHWs still reported stock-outs because the linked health facilities lacked commodities, especially malaria-related ones. Lastly, we only used quantitative methods in this evaluation. While quantitative methods are effective in measuring changes over time, they are less effective in providing a deep understanding of the processes and how those changes actually occur. A mixed methods approach allowing triangulation of qualitative and quantitative findings would have provided a deeper understanding of the implementation process and its outcomes. Despite these limitations, the design, methods and implementation approach of the iCCM project in Homa Bay was robust.

CONCLUSION
The experience of this implementation project has important implications for the design and expansion of iCCM in Kenya. Our detailed report of the implementation process and outcomes using the RE-AIM framework offers pragmatic information that the Kenyan MOH as well as the CHMTs can use for the adoption and scale-up of iCCM.