Objectives To investigate community health workers’ (CHW) adherence over time to guidelines for treating ill children and to assess the effect of refresher training on adherence.
Methods Analysis of 7151 ill-child consultations performed by 114 CHWs in their communities from March 1997–May 2002. Adherence was assessed with a score (percentage of recommended treatments that were prescribed), calculated for each consultation. Recommended treatments were those that were indicated based on CHW assessments. We used piecewise regression models to evaluate adherence before and after training.
Results The average adherence score was 79.4%. Multivariable analyses indicate that immediately after the first refresher training, the mean adherence level improved for patients with a severe illness, but worsened for patients without severe illness. Adherence scores declined rapidly during the 6 months after the second refresher training.
Conclusions The first refresher was partially effective, the second refresher had an effect contrary to that intended, and patient characteristics had a strong influence on adherence patterns. Longitudinal studies are useful for monitoring the dynamics of CHW performance and evaluating effects of quality improvement interventions.
Objectifs Investiguer l'adhérence au cours du temps des agents de santé communautaires (ASC) aux directives pour le traitement des enfants malades et évaluer l'effet du stage de recyclage sur l'adhérence.
Méthodes Nous avons analysé 7151 consultations d'enfants malades effectuées par 114 ASC dans leurs communautés, de mars 1997 à mai 2002. L'adhérence a étéévaluée avec des scores (pourcentage des traitements recommandés qui ont été prescrits) calculés pour chaque consultation. Les traitements recommandés étaient ceux qui étaient indiqués sur base de l’évaluation des ASC. Nous avons utilisé des modèles de régression par morceaux pour évaluer l'adhérence avant et après la formation.
Résultats Le score moyen d'adhérence était de 79,4%. Les analyses multivariées indiquent que juste après le premier stage de recyclage, le niveau moyen d'adhérence a été amélioré pour les patients présentant une maladie grave mais s'est empiré pour les patients sans maladie grave. Les scores d'adhérence ont diminué rapidement au cours des 6 mois après le deuxième stage de recyclage.
Conclusions Á notre connaissance, cette étude est le premier examen longitudinal des tendances à long terme de l'adhérence des ASC aux directives de traitement. Les résultats suggèrent que le premier stage de recyclage a été partiellement efficace, le deuxième par contre a eu un effet inverse à l ‘attente et les caractéristiques des patients ont une influence forte sur les formes d'adhérence. Les études longitudinales sont utiles pour surveiller la dynamique de la performance des ASC et évaluer les effets des interventions sur l'amélioration de la qualité.
Objetivos Estudiar la adherencia a las guías para el tratamiento de niños enfermos, de trabajadores sanitarios comunitarios (TSC), a lo largo del tiempo; y evaluar el efecto que sobre la adherencia tienen los cursos de actualización.
Métodos Hemos analizado las consultas de 7,151 niños enfermos, realizadas entre Marzo 1997 y Mayo 2002, por 114 TSC en sus comunidades. La adherencia se evaluó mediante un puntaje (porcentaje de tratamientos recomendados que fueron prescritos), calculado después de cada consulta. Un tratamiento recomendado era aquel considerado indicado, basándose en la evaluación del TSC. Se utilizaron modelos de regresión ‘‘por piezas’’, para evaluar la adherencia antes y después del tratamiento.
Resultados El puntaje promedio de adherencia fue 79.4%. Un análisis multivariado indicó que inmediatamente después del primer curso de actualización, el nivel medio de adherencia mejoró con pacientes que presentaban una patología severa, pero empeoró con pacientes no graves. Los puntajes de adherencia declinaron rápidamente durante los 6 meses siguientes al segundo curso de actualización.
Conclusiones Hasta donde sabemos, este es el primer estudio longitudinal de las tendencias a largo plazo en la adherencia de TSC a las guías de tratamiento. Los resultados sugieren que el primer curso de actualización fue parcialmente efectivo. El segundo, en cambio, tenía un efecto contrario al que se buscaba. Las características de los pacientes tenían una influencia importante sobre los patrones de adherencia. Los estudios longitudinales son útiles para monitorizar la dinámica del desempeño de TSC y evaluar los efectos que tiene el mejorar la calidad de las intervenciones.
In 1995, the non-governmental organization CARE initiated a CHW program in Siaya District, Kenya, which is located in a province with a high childhood mortality rate (198.8 deaths before 5 years of age per 1000 live births) (Macro International 2002). The program trained CHW volunteers to assess, diagnose and treat children under age 5 years according to the CARE Management of the Sick Child (MSC) guidelines, a simplified version of the WHO/UNICEF Integrated Management of Childhood Illness guidelines (Gove 1997). The program covered 81 890 persons in Siaya in 1999 (Central Bureau of Statistics 2000).
To improve and maintain CHW performance (where performance means adherence to MSC guidelines), the program implemented several interventions. First, the program provided in-service training. The program required that all CHWs attend initial training and recommended that CHWs attend refresher training. Three group training sessions were conducted: the first (January–July 1997) served as initial training, and the subsequent trainings (November 1998–February 1999 and September–November 2000) served as initial or refresher training. Activities of the first training included: lectures, case scenarios, role playing and clinical practice. In addition to these activities, the second and third trainings focussed on weaknesses in CHW clinical skills that were identified by performance assessments and included practical sessions in small groups and review of videotaped exam findings. The same trainer facilitated the second and third trainings, and the same educational methods were used. However, as each of these trainings focussed on weaknesses identified in assessments conducted beforehand, some of the content of the training material was different between these two trainings. Other interventions [described in detail elsewhere (Kelly et al. 2001)] to improve CHW performance included: criteria for selecting CHW, community involvement in CHW selection, job aids, provision of medicines, and supervision. Of note, the MSC guidelines changed in November 1998, and training materials and job aids were revised to reflect these changes.
Despite these interventions to improve performance, a subsequent evaluation showed that CHW frequently did not adhere to treatment guidelines (Kelly et al. 2001). An important question therefore became: why are CHW frequently making errors after several years of quality improvement interventions?
Results from several cross-sectional studies suggest that CHW performance declines after the completion of basic training (Mangelsdorf 1988; Zeitz et al. 1993; Ashwell & Freeman 1995). Refresher training is widely regarded as ‘necessary’ to improve and maintain CHW performance (Winch et al. 2005). However, the scientific evidence for the effectiveness of refresher training is mixed. A cross-sectional study in Bolivia showed that refresher training had an immediate beneficial impact on CHWs’ clinical skills (Zeitz et al. 1993), but our initial investigation of determinants of CHW performance in Siaya revealed that increased exposure to refresher training was not associated with better adherence to treatment guidelines (Rowe et al. 2007). A possible explanation for this finding was that the effect of refresher training had waned and was no longer detectable by the time of our study. In our initial investigation, we were unable to explore this hypothesis because we studied CHWs at only one point in time.
Cross-sectional surveys, such as our initial investigation, are useful for studying causal relationships involving variables whose values are unalterable (e.g. highest level of education CHW attained, patient age) (Hennekens & Buring 1987). However, cross-sectional surveys are limited in their ability to study relationships involving variables that change over time or those that might be subject to alteration subsequent to care (e.g. number of refresher trainings CHW attended). As randomized, controlled trials may not be feasible for evaluating large-scale interventions (Victora et al. 2004), a longitudinal study in a non-randomized setting is the best way to examine reasons for changes in adherence over time (Habicht et al. 1999).
To date, we know of no longitudinal studies that have examined long-term trends in CHW performance. As a continuation of our investigation of factors that influence CHW practices (Rowe et al. 2006, 2007), we conducted a longitudinal study to investigate the changes in CHWs’ adherence to treatment guidelines over time and to assess whether refresher training had an immediate or enduring effect on adherence.
We analysed data on consultations performed by 114 CHWs in their communities from March 1997 to May 2002. These CHWs were the same randomly selected CHWs who were evaluated in February–March 2001 (Kelly et al. 2001). In the survey in 2001, we selected CHWs by systematic sampling, in which we compiled a list of all CHWs who were active at the time of the study (n = 255), selected a random starting point in the list, and selected every second CHW.
We abstracted data from CHWs’ clinical registers, which are books that contain columns that prompt the CHW to record for each consultation: (1) the presence or absence of specific symptoms; (2) classifications (i.e. diagnoses); and (3) treatments. One author reviewed registers to gather data on the timing of supervisory visits recorded by CHW supervisors. We gathered data on CHW characteristics from a questionnaire administered to CHWs in February–March 2001 (Kelly et al. 2001).
In efforts to obtain results that were representative of the CHWs’ skills in following the MSC guidelines for ill-child consultations conducted in their usual work place, we instituted several inclusion criteria. We analysed consultations that (1) were recorded in registers up to 5 years after the CHW became active; (2) involved an ill child aged 0–59 months; (3) were initial consultations; (4) occurred in the village; (5) were recorded by the CHW and not by his/her supervisor (supervisors sometimes recorded consultation findings during supervisory visits); (6) involved an ill child for whom the documentation of assessment findings was sufficient to allow the determination of recommended treatments; and (7) involved an ill child for whom one or more treatments should have been prescribed (quality of treatment was only assessed among patients for whom a drug or referral was indicated).
The outcome was an adherence score, defined as the percentage of guideline-recommended treatments prescribed for each patient. This score was a continuous variable ranging from 0% to 100% and encompassed seven treatments: five oral drugs, one antibiotic eye ointment and referral to a health facility if indicated. Every patient did not require all seven treatments. By applying the MSC guidelines’ definitions of classifications to CHWs’ findings of signs and symptoms, we identified for each patient the classification(s) the CHW should have identified and the treatment(s) the CHW should have prescribed. In effect, the adherence score measured the CHW's ability to process clinical information to choose recommended treatments. The adherence score for each ill child was calculated according to the guidelines that existed at the time of the consultation; thus, the score took into account the changes in the guidelines that occurred in November 1998.
We hypothesized that several patient and CHW factors may influence adherence over time (i.e. interaction between covariate and time) or confound adherence patterns (see covariates listed in Table 1). Nearly all CHWs were female, of Luo ethnicity, or spoke the Luo language well. Thus, we did not consider CHW sex, ethnicity or language-speaking skills to be covariates.
Table 1. Distributions [mean (standard error)] of the treatment adherence score† according to selected covariates
Number of consultations (unweighted)
Treatment adherence score (weighted)
*P value < 0.05 for the Wald test that difference between adherence scores is different from 0.
**P value < 0.05 for test for linear trend.
†Treatment score defined as per cent of all recommended treatments prescribed per child, where recommended treatments are those recommended for disease classifications that are derived from the CHW's assessment findings. Scores were weighted to adjust for probability of selection and CHW non-response.
‡Severe classifications included very severe disease, severe pneumonia, persistent cough, dehydration, severe dehydration, persistent diarrhoea, bloody diarrhoea, complicated measles and ‘measles or complicated measles’ (classification only applicable to consultations recorded on original version of clinical register). Classifications were derived from the CHW's assessment findings.
§Moderate classifications included pneumonia, some dehydration, malaria and measles. Classifications were derived from the CHW's assessment findings.
¶Danger signs included child had fever (for children <2 months old only), child was unable to drink or breastfeed, child vomitted all that he/she consumed, child had convulsions, or child was lethargic.
††CHW, community health worker.
Patient's age (months)
Patient had ≥1 severe classification‡
Patient had ≥1 moderate classification§
Number of classifications
Patient had fever
Patient had cough
Patient had diarrhoea
Patient had ≥1 danger sign¶
Number of supervisory visits recorded in clinical register as of date of consultation
Version of clinical register onto which consultation was recorded
CHW's†† age (years)
Women in community had strong influence in CHW selection
Levels of education CHW completed
We hypothesized that (1) adherence declined with time; (2) refresher training had an immediate, positive impact on the mean adherence level after refresher training (i.e. the intercept of the adherence score curve increased at the time of refresher training); and (3) refresher training had a positive impact on trends in adherence (i.e., the difference between the slopes of the post- and pre-refresher adherence score curves was greater than zero).
We used scatterplots and robust locally weighted smoothing techniques (Cleveland 1979; Friendly 2004) to visualize patterns in adherence. As a preliminary analysis of the impact of refresher training, we calculated the mean adherence scores for consultations performed by CHWs who had attended zero, one or two refresher trainings by the time of the consultation. We also reported mean adherence scores across categorical levels of covariates. We performed descriptive analyses with SUDAAN (Research Triangle Institute 2001), which accounts for sampling weights and the correlation between adherence scores for patients seen by the same CHW. The sampling weights accounted for the fact that we studied a sample of all active CHW, and that CHW participation rates varied by geographic areas of the Siaya District (Rowe et al. 2006a).
To test statistical hypotheses regarding the impact of refresher training, we developed piecewise random effects linear regression models (Cnaan et al. 1997; Naumova et al. 2001; Wagner et al. 2002; Ramsay et al. 2003). In these models, a knot represents the time at which a change occurs (e.g. time of refresher training), the intercept is the mean adherence score at the beginning of a time period (e.g. at time = 0 or at a knot), and the slope is the rate of change in adherence scores over time. The model allowed for the estimation of the change in the intercept at each knot and the change in the slope after the knot. We included random effects (i.e. random intercepts and slopes) to account for the variability in treatment adherence levels and trends between CHWs.
To analyse the impact of the first refresher training, we assumed each CHW had a two-piece linear spline adherence score curve, which was discontinuous at a knot at the time of the first refresher training. This allowed the intercept and slope to change after the first refresher training. The descriptive analysis results revealed a downward curvilinear trend in the post-first-refresher training slope; thus, we also allowed for non-linearity in the post-first-refresher training slope. In the analysis of the first refresher training, we excluded consultations performed by CHWs who did not attend any refresher training and consultations performed by CHWs who had attended two refresher trainings by the time of the consultation.
To analyse the impact of the second refresher training, we assumed each CHW had a three-piece linear spline adherence score curve, which was discontinuous at a knot at 22 months before the second refresher training (‘knot 1’) and at another knot at the time of the second refresher training (‘knot 2’). We positioned knot 1 at 22 months before the second refresher training because the earliest consultation performed by a CHW who had attended one refresher training by the time of the consultation occurred approximately 22 months before the second refresher training. Thus, with this model, we allowed the intercept and slope to change after the first refresher training and to change again after the second refresher training. The time period before knot 1, between knots 1 and 2, and after knot 2 is defined as period A, period B and period C, respectively. The descriptive analysis results revealed a curvilinear trend in the slopes for periods B and C; thus, we allowed for non-linearity in the slope in periods B and C. In the analysis of the second refresher training, we excluded consultations performed by CHW who did not attend two refresher trainings.
We assessed interactions between covariates and the change in the intercept and between covariates and the slope. If an interaction exists between the change in the intercept and a covariate, this means that the change in the mean adherence level at a knot (e.g. at the time of refresher training) varies according to different values of the covariate. If an interaction exists between the slope and a covariate, this means that the month-to-month change in adherence scores varies according to different values of the covariate. We maintained only interaction terms that were significant at the alpha = 0.05 level in the model. Our analysis of interactions was exploratory, and we did not adjust P-values for multiple comparisons. We assessed for confounders using a hierarchical backwards elimination approach (Kleinbaum et al. 1982). This approach involved first including all covariates and statistically significant interaction terms into a model (the ‘base model’), then sequentially removing each covariate from the model that was not a component of an interaction term to create ‘comparison’ models. Covariates that, when removed from the base model, resulted in slopes for the pre- and post-refresher training time periods that differed by 20% or more were considered to be confounders. We conducted regression analyses using the MIXED procedure in SAS, which accounts for correlated data (SAS Institute 2001). To account for sampling weights, we adjusted results obtained from SAS using jackknife repeated replication techniques (Raghunathan et al. 2002). To test statistical hypotheses for fixed effects, we used Wald tests (i.e. ratio of an estimated parameter to its standard error) (Cnaan et al. 1997). All tests were two tailed.
To visualize the effect of refresher training after adjusting for potential confounding factors, we calculated population predicted adherence scores from the piecewise random effects linear regression models and plotted these predicted values against time relative to refresher training.
CARE/Kenya and the Institutional Review Boards at the US Centers for Disease Control and Prevention and Emory University reviewed and approved the study protocol.
We abstracted data on 10 060 consultations. Of these, we excluded 2909 consultations: 1743 (17.3%) were follow-up visits; 461 (4.6%) were initial consultations for patients 60 months old or older; 465 (4.6%) were initial consultations for ill children whose documentation of assessment findings was incomplete and did not allow for the determination of recommended treatments; 139 (1.4%) were initial consultations for ill children who did not require treatment; and 101 (1.0%) occurred at a health facility or was not recorded by the CHW. Thus, 7151 consultations remained for analysis.
The data from this study were highly unbalanced. For each CHW who became active in 1997 (n = 95), we collected clinical register data on an average of 69 (range 9–334) initial consultations involving children 0–59 months old, while for each CHW who became active after 1997 (n = 19), we collected an average of 30 (range 5–134) consultations.
Smoothed adherence score curves over time since initial training are displayed in Figure 1. We see that on average, the mean adherence scores initially improved but eventually worsened after initial training. The trends in adherence after initial training were similar for CHWs who attended one (n = 23) or two (n = 85) refresher trainings. However, the adherence pattern for CHWs who attended no refresher trainings (n = 6) clearly diverged from that for CHWs who attended any refresher training.
When we plotted adherence scores vs. time relative to the date of refresher training, the pattern of adherence scores with respect to refresher training was clearer. Figures 2 and 3 show plots of adherence scores grouped and averaged over 3-month periods. Based on the smoothed adherence curve in Figure 2, had the first refresher training not occurred, we would have expected the mean adherence level to continue at about 78% per patient per 3-month period. However, immediately after the first refresher training, the mean adherence level was about 85%. Furthermore, adherence continued to rise during the post-first-refresher period, flattened, and then curved downwards (Figure 2). In contrast, the mean adherence level did not improve immediately after the second refresher training, and adherence scores declined rapidly after the second refresher training (Figure 3).
For the entire sample of consultations (n = 7151), the mean adherence score was 79.4% [standard error (SE) = 1.01]. Of the entire sample of consultations, 55.8% were prescribed all recommended treatments. The mean adherence score for consultations performed by CHWs who had attended zero, one or two refresher trainings by the time of the consultation was 75.5%, 84.3% and 79.1%, respectively. This preliminary analysis suggests that the adherence scores improved after the first refresher training but worsened after the second refresher training.
Several patient and CHW characteristics were related to adherence scores in univariate analysis (Table 1). The adherence scores were higher in consultations in which the patient did not have a severe classification, diarrhoea, cough or a danger sign; the patient had a moderate classification; the consultation was recorded on the revised version of the register; and the patient was treated by a CHW who was under 50 years of age. The adherence scores increased as the number of supervisory visits recorded in the register increased.
Random effects analysis
Table 2 shows the estimates of the fixed effects in the multivariable model of the impact of the first refresher training on adherence (model 1). Model 1 excluded consultations performed by CHWs who did not attend any refresher training and excluded consultations performed by CHWs who had attended two refresher trainings by the time of the consultation; 6006 consultations remained for analysis. The first refresher training's effects varied across different types of consultations, as shown by the significant interactions between (1) the pre- or post-first-refresher slopes and patient age (where age is coded as continuous and centered around zero months); and (2) the change in the intercept at the first refresher and patient-level covariates (i.e. patient had one or more severe classifications, fever or cough). For example, as shown in the estimate for the ‘interaction term for patient had ≥1 severe classification and change in intercept at event’ for model 1, the change in the adherence level immediately after the first refresher training was 17.03 percentage points higher for patients with a severe classification than that for patients without severe classifications.
Table 2. Estimates of fixed effects from multivariable models 1 and 2 for treatment adherence scores
Model 1† Estimate ± SE§
Model 2‡ Estimate ± SE
*P value for the Wald test for the hypothesis that the estimate is different from zero.
†The ‘event’ is the first refresher training. Empty spaces indicate the variable was not included in the model.
‡The ‘event’ is the second refresher training. Empty spaces indicate the variable was not included in the model.
§SE, standard error.
¶For model 1, the estimate assumes the consultation (1) did not involve a patient with a severe or moderate classification, fever or cough; (2) was recorded on the revised version of the clinical register; and (3) involved a patient who was of average age (19.4 months), had the average number of classifications (1.3 classifications), and was seen by a CHW who had the average number of initial consultations with ill children 0–59 months old recorded in the clinical register as of the date of the consultation (49.9 consultations).
For model 2, the estimate assumes the consultation (1) did not involve a patient with a severe classification; (2) was recorded on the revised version of the clinical register; and (3) involved a patient who was of average age (19.1 months), had the average number of classifications (1.3 classifications), and was seen by a CHW who had the average number of initial consultations with ill children 0–59 months old recorded in the clinical register as of the date of the consultation (51.2 consultations).
**For model 2, knot 1 occurred at 22 months before the second refresher training. The estimate assumes the consultation (1) did not involve a patient with a severe classification; (2) was recorded on the revised version of the clinical register; and (3) involved a patient who was of average age (19.1 months), had the average number of classifications (1.3 classifications), and was seen by a CHW who had the average number of initial consultations with ill children 0–59 months old recorded in the clinical register as of the date of the consultation (51.2 consultations).
††Time period 23 or more months before second refresher training.
‡‡Time period 22 or fewer months before second refresher training.
§§Time period after second refresher training.
¶¶Severe classifications included very severe disease, severe pneumonia, persistent cough, dehydration, severe dehydration, persistent diarrhoea, bloody diarrhoea, complicated measles and ‘measles or complicated measles’ (classification only applicable to consultations recorded on original version of clinical register). Classifications were derived from the CHW's assessment findings.
***Moderate classifications included pneumonia, some dehydration, malaria and measles. Classifications were derived from the CHW's assessment findings.
Mean treatment guideline adherence just before event¶
85.96 ± 3.33
84.78 ± 1.93
Mean treatment guideline adherence just before knot 1**
88.09 ± 3.02
Change in intercept at event¶
−18.26 ± 3.57
0.84 ± 2.06
Change in intercept at knot 1**
−8.39 ± 3.74
−0.07 ± 0.13
Period A††: −0.29 ± 0.15
Period B‡‡: 0.78 ± 0.43
0.61 ± 0.34
Period C§§: −1.59 ± 0.99
Quadratic term for pre-event slope
Period B: −0.02 ± 0.02
Quadratic term for post-event slope
−0.02 ± 0.01
Period C: 0.14 ± 0.04
Interaction term for patient's age and pre-event slope
−0.02 ± 0.004
Interaction term for patient's age and post-event slope
0.01 ± 0.004
Interaction term for patient had fever and change in intercept at event
9.99 ± 2.67
Interaction term for patient had cough and change in intercept at event
8.39 ± 1.75
Interaction term for patient had ≥1 severe classification¶¶ and change in intercept at event
17.03 ± 2.98
Interaction term for patient's age and change in intercept at knot 1
−0.16 ± 0.05
Interaction term for patient had ≥1 severe classification and change in intercept at knot 1
19.77 ± 3.09
Interaction term for number of classifications and change in intercept at knot 1
8.81 ± 1.45
Patient age in months (centered)
−0.22 ± 0.05
0.05 ± 0.04
Number of classifications (centered)
−5.76 ± 0.89
−11.64 ± 1.45
Patient had fever
−8.49 ± 2.14
Patient had cough
−9.61 ± 1.38
Patient had ≥1 severe classification
−22.68 ± 2.72
−34.67 ± 2.39
Patient had ≥1 moderate classification***
14.72 ± 2.43
Consultation was recorded on original version of clinical register
−5.53 ± 2.25
−6.36 ± 2.91
Number of initial consultations with ill children 0–59 months old recorded in the CHW's clinical register as of the date of the consultation
0.02 ± 0.02
0.05 ± 0.03
On average, the post-first-refresher training slope was higher than the pre-first-refresher slope [post-event slope for model 1 – pre-event slope for model 1 = 0.61 − (−0.07) = 0.68]. However, this finding was not statistically significant (P = 0.06).
In multivariable analysis, several covariates were significantly associated with adherence. The adherence scores were significantly higher for patients with one or more moderate classifications and consultations recorded in the revised version of the register. We excluded from multivariable analysis the variable for the number of supervisory visits recorded in the register because we found that supervision did not influence patterns in adherence (i.e., no interaction), and the effects of the first refresher training were not confounded by supervision. When forced into the multivariable model, supervision was not significantly associated with adherence (results not shown).
Population-predicted adherence scores based on model 1 vs. time relative to the first refresher training are displayed in Figure 4. This figure illustrates that the change in the mean adherence level immediately after the first refresher training differed importantly by whether the patient had ≥1 severe classification.
Table 2 shows the estimates of the fixed effects in the multivariable model of the impact of the second refresher training on adherence (model 2). Model 2 excluded consultations performed by CHWs who did not attend two refresher trainings; 5812 consultations remained for analysis. On average, the post-second-refresher slope was significantly lower than the pre-second-refresher slope in period B (model 2: period C slope − period B slope = −1.59–0.18 = −2.37, SE = 0.87, P = 0.01).
In multivariable analysis, one covariate was significantly associated with adherence: adherence scores were significantly higher for consultations recorded in the revised version of the register. Again, we excluded the variable for the number of supervisory visits recorded in the register from multivariable analysis because we found that supervision did not influence patterns in adherence, and the effects of the second refresher training were not confounded by supervision. When forced into the multivariable model, supervision was not significantly associated with adherence (results not shown).
Population-predicted adherence scores based on model 2 vs. time relative to the second refresher training are displayed in Figure 5. This figure illustrates that the second refresher training had no immediate impact on the mean adherence level, and adherence declined rapidly after the second refresher training.
Our study presents important clues to understanding how CHWs’ treatment skills change over time. To our knowledge, our study is the first longitudinal examination of long-term time trends in CHWs’ adherence to treatment guidelines and factors that influence these trends. While a series of cross-sectional studies highlights overall population-based improvements in adherence, they miss the changes in adherence for individual CHWs that contribute to the overall population-based change (Weiner and Long 2004). In the current study, results from the preliminary, cross-sectional assessment of the impact of refresher training suggested that the adherence scores improved after the first refresher training. However, our longitudinal analysis revealed that adherence scores for only certain types of consultations improved. This demonstrates that overall population-based improvement in adherence does not necessarily mean that all CHWs’ treatment skills are improving uniformly.
The treatment skills of CHWs in this study were comparable with those of other health workers in developing countries. For example, the estimated overall mean treatment adherence score just after the first refresher training was 76% in this study. In an evaluation in Bolivia, where CHWs were given case histories of children with acute respiratory infections and asked to specify treatments, the overall mean treatment score was 76% just 2 weeks after refresher training (Zeitz et al. 1993).
The finding that the first refresher training was helpful in improving CHWs’ skills in treating severely ill patients is consistent with one of the intended purposes of refresher training: to improve CHWs’ skills in managing severely ill patients. This result corroborates evidence from some reviews of health worker programs in developing countries, which suggests that interventions targeted at specific causes of problems are more effective than those that are not (Marquez 2001; Grol & Grimshaw 2003). However, we also gathered compelling evidence that refresher training adversely affected CHW treatment practices. The first refresher training had an immediate negative impact on skills for treating non-severely ill patients, and adherence scores declined rapidly after the second refresher training.
Our finding of declining adherence scores after the second refresher training is consistent with several explanations. First, the difference between the focussed content of the first and second refresher trainings may have confused CHWs. Second, the post-second-refresher training slope estimate might have been biased because the slope for the period beyond 6 months after the second refresher training was based on data from only 12 CHWs, and their results might not be representative of all CHWs. However, this explanation is less likely because these few CHWs did not differ from the entire sample of CHWs by demographic characteristics, and after excluding consultations that occurred beyond 6 months after the second refresher training, the post-second-refresher slope was still less than the pre-second-refresher training slope (results not shown). Third, CHWs who had poor performance after the first refresher training may have been more motivated to attend a second refresher training, and CHWs with poor performance after one refresher training might be less likely to improve after a second refresher training. This reasoning is less likely, as among the CHWs who attended the second refresher training, the estimated overall mean adherence score before the second refresher training was fairly good (about 80%). Fourth, the second refresher training might have had no effect, and the decline in scores reflected that which would have occurred naturally in the absence of refresher training.
In short, we do not have a clear explanation for the contrasting effects of the first and second refresher training. Our mixed results agree with those from numerous cross-sectional studies that have examined the relationship between refresher training and the quality of treatment given by health workers in developing countries; results range from a beneficial effect (Zeitz et al. 1993; Zurovac 2004; Naimoli et al. 2006) to no effect (Ashwell & Freeman 1995; Ofori-Adjei & Arhinful 1996; Rowe et al. 2000, 2003). Taking all these results together, the challenge for CHW programs is to find quality-improvement strategies that work for their particular setting. In the setting of the Siaya CHW program, we did not find that repeated targeted refresher training was an effective strategy for sustaining high-quality treatment practices over time.
In our longitudinal analysis of the impact of refresher training, we excluded consultations performed by CHWs who did not attend any refresher training, as the inclusion of these consultations might have resulted in biased estimates of the difference between pre- and post-refresher slopes. Although based on data from only six CHWs (and therefore might not be reliable), the ‘U-shaped’ trend in adherence scores for CHWs who did not attend any refresher training suggested that these CHWs were different from those who attended refresher trainings. A possible explanation for the improvement in adherence among the CHWs who attended no refresher training was the clinical guidelines and job aids were revised approximately 15 months after initial training, and following this revision, guidelines might have been easier to follow and the job aids might have been easier to use. The implications of excluding data from CHWs who attended no refresher training is that our measured impact of refresher training on adherence might not be generalizable to all CHWs in Siaya. However, as we evaluated refresher training implemented in a ‘real life’ (i.e. non-clinical trial) setting, where the intent may be for all CHWs to receive the intervention, but actual reception of interventions often can be imperfect, our study still provides valuable insights to the impact of refresher training.
Our study had several limitations. First, our measurements of CHW characteristics might be affected by mis-classification owing to recall error. We attempted to reduce misclassification by verifying information recalled by CHWs concerning training experience with program records. Second, our measurement of the number of consultations that CHW performed in their villages and the number of supervisory visits might be inaccurate, as we relied on the documentation of this information in registers. Third, as there was no clinical re-examination of consultations recorded in registers, we were unable to evaluate the ‘true’ quality of treatment (i.e. errors based on the treatment recommended according to a gold standard clinician's assessments) given by CHWs. Fourth, we could not investigate the influence of several potentially important factors because all CHWs were exposed to these factors (e.g. complexity of MSC guidelines). Fifth, our statistical analysis involved testing numerous hypotheses regarding associations between treatment adherence and several exposure factors. Statistically significant results might be spurious and significant owing to chance alone.
In conclusion, we have demonstrated that CHWs’ adherence to treatment guidelines exhibits a dynamic pattern over time. We encourage CHW programs to collect longitudinal data to determine whether CHWs achieve benchmark levels of performance and to evaluate immediate and long-term effects of interventions, such as in-service training, that are implemented only once over long time intervals. Furthermore, our study results add to a growing evidence base that discounts the widely held assumption that refresher training is effective in improving and maintaining CHW performance. There is an urgent need to re-examine commonly used interventions aimed at improving CHW performance and to investigate the effectiveness of alternative interventions. In the Siaya CHW program, and perhaps in others, this re-examination should focus on the CHW's ability to manage severely ill children. In Siaya, treatment adherence scores for severely ill children were consistently sub-optimal over time, despite the implementation of several quality improvement interventions. Additionally, our study evaluated CHWs that served a relatively small area in Kenya. As CHWs will likely be involved in interventions that are scaled up to a national level, further efforts are needed to investigate the effect of quality improvement strategies in settings with wide coverage and utilization of CHWs. Moreover, as interest in the use of CHWs in developing countries is increasing (Bhattacharyya et al. 2001; Chen et al. 2004; Sachs 2005) and more tasks may be integrated into CHWs’ responsibilities, further efforts are needed to examine the hypothesis that increasing complexity of tasks leads to lower adherence. Without interventions that are effective in maintaining high performance for all CHW tasks, programs risk exposing large populations to inappropriate care.