- Top of page
Objective Although essential for understanding the reasons for success or failure of large scale nutritional interventions, process evaluation results are rarely reported. Our objective was to assess whether the process output objectives of the Community Nutrition Project (CNP) in Senegal, West Africa, were adequately met.
Methods An adequacy assessment study based on monitoring data for individuals collected during the CNP was used to assess ‘fidelity’, ‘extent’ and ‘reach’ of participants recruitment and of the services provided. The CNP provided underweight or nutritionally at risk 6- to 35-month-old children in poor districts with monthly growth monitoring and promotion and a weekly food supplementation for 6 month periods, provided that mothers attended weekly nutrition education sessions. An exhaustive sample of the participating children (n = 4084) in Diourbel was used for evaluation over the first 2 years.
Results At recruitment, only 66% of children were underweight (vs. 90% expected) varying with the CNP center and cohort, and the child's sex and age. Attendance at growth monitoring reached expected levels (93%vs. 90%) whereas numbers of food supplements distributed and education sessions attended were lower than expected (45%vs. 90% and 62%vs. 80%, respectively). At the end of follow-up, 61% of underweight children recovered vs. 80% expected.
Conclusions Because of CNP design for underweight diagnosis and bias in the targeting process, respect for selection criteria was low and consequently under coverage and leakage occurred. Besides a globally satisfactory process, wide discrepancies were observed between CNP centres concerning the utilization and effectiveness of services. This formative evaluation helped diagnose weaknesses; ongoing feedback enabled the CNP to improve targeting and supply of supplements. It also informed a larger impact evaluation. Some generalizable lessons for similar programmes have been highlighted.
Objectif Bien qu’étant essentiels pour comprendre les raisons associées à l’échec ou au succès des programmes d'intervention nutritionelle à large échelle, les résultats d’évaluation de fonctionnement sont rarement rapportés. Notre étude avait pour but d’évaluer si les objectifs de fonctionnement du Projet de Nutrition Communautaire (PNC) du Sénégal, en Afrique de l'ouest, ont été atteints.
Methodes Une évaluation de l'adéquation aux objectifs, basée sur les données de suivi individuel des bénéficaires collectées dans le PNC, a été utilisée pour estimer les critères de ‘fidélité’, ‘étendue de couverture’ et ‘atteinte’ du recrutement des participants et des services procurés. Le PNC procédait, dans les quartiers pauvres, au suivi-promotion mensuel de la croissance des enfants de 6 à 35 mois présentant une insuffisance pondérale ou un risque nutritionnel, et leur fournissait un complément alimentaire hebdomadaire pendant 6 mois, à condition que les mères participent à des sessions hebdomadaires d’éducation nutritionnelle. Un échantillon exhaustif des enfants ayant participé au PNC à Diourbel a été utilisé pour l’évaluation des deux premières années du projet (n = 4084).
Resultats Au moment du recrutement, seulement 66% des enfants présentaient une insuffisance pondérale, contre 90% attendus, et cela variait selon le centre du PNC ou la cohorte, mais aussi selon le sexe et l’âge des enfants. La participation au suivi de la croissance a atteint le niveau attendu, soit 93%vs. 90%, alors que le nombre de compléments alimentaires distribués et la participation aux sessions d’éducation étaient inférieurs aux attentes, soit 45%vs. 90% et 62%vs. 80% respectivement. A la fin du suivi, 61% des enfants qui avaient une insuffisance pondérale au départ ont récupéré, contre 80% attendus.
Conclusions A cause du mode de diagnostic de l'insuffisance pondérale choisi par le PNC et d'un biais dans le procédé de ciblage, l'adhésion aux critères de recrutement était faible, ayant pour conséquence une faible couverture des enfants ciblés ainsi qu'un détournement du projet vers des non ciblés. Au delà d'un fonctionnement qui s'est avéré globalement satisfaisant, des discordances ont été observées entre les différents centres du PNC pour ce qui était de l'utilisation et de l'efficacité des services. Cette évaluation ‘formative’ a contribuéà identifier les faiblesses, et des rétroactions régulières ont permis au PNC d'améliorer le ciblage et la fourniture de complément alimentaire. Elle a aussi fourni des informations nécessaires à une évaluation de l'impact du PNC à un niveau plus large. Des leçons de portée plus générale ont été tirées pour des programmes similaires.
Objetivo A pesar de ser esencial para entender los motivos detrás del éxito o el fracaso de intervenciones nutricionales a gran escala, los procesos de evaluación son rara vez reportados. Nuestro objetivo fue el valorar si los objetivos y resultados esperados del proceso del Proyecto de Nutrición Comunitaria (Community Nutrition Project (CNP)) en Senegal, África Occidental, habían sido alcanzados de forma adecuada.
Métodos Se utilizó un estudio de valoración de la adecuación basado en la monitorización de datos de individuos recolectados durante el CNP, para evaluar la ‘‘fidelidad’’, la ‘‘extensión’’ y el ‘‘alcance’’ del reclutamiento de participantes y los servicios provistos. El CNP proveía a niños con bajo peso o riesgo nutricional de entre 6 y 35 meses de edad y provenientes de distritos de bajos recursos, con una promoción y monitorización mensual del crecimiento así como una suplementación alimenticia semanal durante 6 meses, siempre y cuando las madres asistieran a las sesiones de educación nutricional que se impartían semanalmente. Con el fin de llevar a cabo la evaluación durante los primeros dos años, se utilizó una muestra exhaustiva (n = 4084) de los niños que participaron en Diourbel.
Resultados Al ser reclutados, solo un 66% de los niños tenían bajo peso (vs. 90% esperado) encontrándose variaciones entre los diferentes centros CNP y las cohortes, el sexo y la edad del niño. La asistencia a la monitorización de crecimiento alcanzó los niveles esperados (93%vs. 90%), mientras que el número de suplementos alimenticios distribuidos y las sesiones de educación atendidas fueron menores de lo esperado (45%vs. 90%, y 62%vs. 80%, respectivamente). Al final del seguimiento, 61% de los niños con bajo peso se habían recuperado vs. un 80% esperado.
Conclusiones Debido al diseño del CNP para el diagnóstico de bajo peso y el sesgo en el proceso de asignación, el respeto a los criterios de selección fue bajo y consecuentemente se encontró una baja cobertura y pérdidas. Además de un proceso global satisfactorio, se observaron amplias discrepancias entre los centros CNP con respecto a la utilización y efectividad de los servicios. Esta evaluación formativa ayudó a realizar el diagnóstico de una flaqueza; la retroalimentación constante permitió que el CNP mejorase la asignación y el suministro de suplementos alimenticios. También ha servido de información para una evaluación de mayor impacto. Se resaltan algunas lecciones extrapolables a otros programas similares.
- Top of page
The purpose of process evaluation is to relate impact and outcome data to intervention activities. Process evaluation results are not frequently reported, although they are essential for understanding the failures and successes of large scale interventions (Habicht et al. 1999). Three components of process evaluation can be examined in a process evaluation study: ‘fidelity’, ‘extent’ and ‘reach’ (Baranowski & Stables 2000). Fidelity concerns the adherence of the project's delivery to its design and stated guidelines of operation, extent provides information on the level of implementation of the intervention and reach informs on the probability of the process outcomes to be met. First, the quality of recruitment reflects the fidelity of delivery. It influences the degree to which the program was received by the targeted group (reach) and can therefore influence global reach, i.e. the coverage of the targeted group in the overall population. Second, the quality of services provided by an intervention can influence participants’ attendance. This attendance reflects, on the one hand, the amount of services delivered (extent), and, on the other hand, the maintenance of participants (reach). The level of attendance of participants has a direct influence on the level of the expected process outcome, e.g. nutritional recovery among malnourished participants.
The main focus of the present study was the process evaluation of the Community Nutrition Project (CNP) in Senegal, West Africa. As part of evidence based practice, descriptive results of process evaluation were considered as a third level of useful data on the intervention implementation (Rychetnik et al. 2004) after the nutritional problem to be addressed was identified and the CNP design was chosen. The main question addressed in this adequacy assessment (Habicht et al. 1999) was ‘to what extent were the CNP activities implemented and the expected process objectives met?’ Besides the aim of improving CNP operations whenever possible, the process evaluation aimed at informing a larger impact evaluation.
Overview of the Community Nutrition Project in Senegal
In Senegal in the beginning of the 90s, the prevalence of underweight among children under-five increased in urban areas (WHO 2005) and marked differences in prevalence were to be expected between neighbourhoods owing to their contrasting socio-economic levels. In response to this situation, the Government of Senegal initiated the nation-wide CNP funded by the World Bank (World Bank 1995) and implemented by a private agency AGETIP (Agence d'Exécution des Travaux d'Intérêt Public; Marek et al. 1999).
The CNP featured both a nutrition component and a water component. The objectives of the nutrition component were: (i) to halt further deterioration in the nutritional status of children <3 years of age in targeted poor urban neighbourhoods; (ii) through nutrition education, to initiate changes in attitudes and feeding practices of mothers. Services were provided in specific buildings called Community Nutrition Centres (CNC), by legal entities specifically created for the purpose which consisted of four people (generally with a bachelor's degree), usually previously unemployed and living in the targeted neighbourhood (Diallo et al. 1997). Physicians specially employed for the project supervised the delivery of services.
To reach the most vulnerable groups among the poor, the first level of targeting was geographical, focusing on districts comprising a high rate of poor households (Sadio & Diop 1994). Poor households were defined as those whose average per capita monthly expenditure level was below the cost of a food basket equivalent to 2400 kcal/day. In the selected districts, every 6- to 36-month-old child was theoretically eligible. However, as the priority outcome was to bring malnourished children back to normal growth, a second level of targeting was the children's nutritional status in order to select eligible children for participating in the CNP nutrition component. According to CNP terms of reference, malnutrition was defined as underweight (low weight-for-age) identified with a colour-based growth chart which used the threshold of approximately 80% of the median of the National Centre for Health Statistics (NCHS) reference (WHO 1983). A child aged 6–36 months was eligible if s/he was: malnourished; the sibling of a malnourished child; or well nourished but had not gained weight during the last 2 months, when previously participating in CNP.
The project design has been documented in detail elsewhere (World Bank 1995; République du Sénégal, AGETIP, Projet de Nutrition Communautaire 1997). Basically, the CNP provided targeted children with monthly growth monitoring and promotion and a weekly food supplementation (a flour mix made of local ingredients), provided that their mothers attended weekly nutrition and health education sessions, for a period of 6 months. CNP also provided targeted children with: referral to health services for unvaccinated children, for severely malnourished children, for children who failed to gain weight over 2 months and for sick children; and home visits to follow-up participating children who were referred or who did not come to the services.
Expected processes of the Community Nutrition Project in Senegal
Process objectives were developed and agreed upon with the Government, AGETIP, and key stakeholders during an objectives-oriented project planning (Ziel-Orientierte Project Planung: ZOPP) participatory workshop held in July 1994 in Dakar, Senegal. The objectives of the project were then clarified, project activities and intended results identified and indicators to monitor results defined (World Bank 1995).
The CNP planned to offer a total number of seven weightings (monthly), 24 rations of supplement (weekly) and 24 educational sessions (weekly) per beneficiary during each cohort of 6 month duration. Expected processes comprised the recruitment of at least 90% of underweight children in each cohort. Moreover, the CNP assumed that 20% of participating children would still present criteria to be eligible at the end of a 6-month period and offered them the opportunity of being recruited again. CNP's objectives were to weigh 90% of participating children monthly, to deliver 90% of the planned food supplements and to reach an attendance of 80% of the mothers at education sessions. CNP objectives also included target values for the main process outcome that is 80% of nutritional recovery at the end of a cohort in children underweight at recruitment (World Bank 1995; République du Sénégal, AGETIP, Projet de Nutrition Communautaire 1997).
- Top of page
The real value of a process evaluation is its ability to compare observed programme processes with expected processes. Moreover, our results were used to attempt to improve the operations. Management of the data collected from CNP monitoring system needed for the present study was carried out after the end of the second year of the programme. During the third year, analyses were performed and results reported, notably at ‘mid-term workshop of the CNP and World Bank in Senegal’ (Kameli & Gartner 1998). Then, during all the period of the CNP evaluation, i.e. until 2003, we worked in permanent collaboration with CNP leaders and workers and ongoing feedback to CNP was performed via short regular (in average monthly) meetings. Another interest of process evaluation is to speculate about potential programme impact. For that purpose, Table 5 contains some elements of results and discussion in terms of process outputs and outcome.
Table 5. Comparison of expected and observed processes, and potential contribution to CNP impact
| || Expected processes (%)||Observed processes||Possible reasons||Potential contribution to impact in the overall population (positive or negative effect)|
| Underweight diagnosis|| ||Overestimation at recruitment, underestimation at the end*||‘Chart’ and ‘yield’ bias||Under coverage of the targeted group prevents direct and immediate effect on prevalence of underweight||(−)|
|Lower rate of underweight girls recruited vs. boys|
| Recruitment of children|
| Underweight||90||66%† (from 47% to 89%)‡||‘Yield’, ‘humanitarian’, and ‘social’ bias||Leakage because of the recruitment of non-underweight children||(−)|
| At risk|| ||6% (from 0% to 9%)‡|
| Children previous participant||20||30% (from 3% to 65%)‡|
| Underweight||90||62% (from 29% to 88%)‡|
| At risk|| ||0.4% (from 0% to 4%)‡|
| Attendance at growth monitoring and promotion (monthly)||90||93%§ (from 85% to 99%)‡||Other services are weekly||Raising mother's awareness of the child growth||(+)|
| Effective distribution of food supplement (weekly)||90||45%§ (from 19% to 94%)‡||Irregular supply of supplement to CNP staff||Low coverage of beneficiaries plus dilution within the household can limit effectiveness of supplementary feeding||(−)|
| Mothers’ attendance at education sessions (weekly)||80||62%§ (from 45% to 96%)‡||Lack of food supplement||Insufficient knowledge improvement prevents mid-term impact on child growth||(−)|
|Conflicts with income generating activities||Context dependent (e.g. household resources)||(−)|
| Recovery from underweight among beneficiaries||80||61%† (from 43% to 91%)‡||Wrong underweight diagnosis||Too low rate of nutritional recovery prevents immediate impact||(−)|
|Wasting or stunting not differentiated by using weight-for-age||Prevents specific action addressing each problem||(−)|
Underweight diagnosis and recruitment
The explanation of the incorrect selection of targeted children may be found in a combination of several biases. First, the main explanation of a ‘chart’ bias may be found in the use of a single reference for boys and girls together. The CNCs’ growth chart had a threshold determined from a ‘mean’ value of the two NCHS references of body weight for the age of boys and of girls (WHO 1983), thus giving an intermediate value between the correct one for boys and the correct one for girls (FAO 1992; Latham 1997). This threshold was too low for boys and too high for girls. Therefore, it is likely that an underweight boy with a weight just below the correct reference value for boys would not be classified as underweight, and a normal weight girl with a weight value just above the proper reference for girls would be classified as underweight by CNP workers. Many community nutrition programmes do use a single chart for both sexes. The bias introduced by this practice is important enough to underline that it is a significant weakness of those programmes for at least two reasons. First, it leads to an assessment of underweight that differs from the diagnosis based on sex-specific references largely used to evaluate underweight status in population or impact of nutritional intervention. Second, it leads to under coverage of targeted underweight girls in contexts where female children could be already challenged with other disadvantages. However, there was no disadvantage towards girls concerning delivery of services from CNP.
Another source of bias could be that the objective of CNC workers was to include as many underweight children as possible at the beginning and to retrieve as few as possible at the end. We could therefore suggest the hypothesis of a ‘yield’ bias at the group (cohort) level. Third, this kind of error could also be due to a ‘humanitarian’ bias at the individual (child) level to avoid excluding borderline normal weight children, as previously reported (Soeters 1986). One reason for this systematic bias could be that with the chart method, the measurer knows immediately whether or not the child is malnourished and tends, at recruitment, to classify children who are slightly above the threshold point as being under it. Finally, implementing the eligibility criteria can lead to perceptions of unfairness (Marchione 2005). The individual nutritional targeting among the population who came to the CNCs was not always well accepted by mothers and the CNC workers may have felt obliged to include non-target children (‘social’ bias). Better accounting for such problems met by programme workers is critical for the improvement of program targeting (Lee et al. 2005).
For comparison purpose, sensitivity/specificity of the assessment of very low weight-for-age by using the growth chart in Africa when compared with weight-for-age<-3 z-scores was of 62.0%/98.7% under conditions which exclude some bias inducing constraints e.g. those from targeting or meeting programme objectives (Hamer et al. 2004).
Inclusion of normal weight children could be considered potentially useful in an intervention in terms of prevention of malnutrition. However, a too low proportion of underweight participants has to be taken seriously in the case where all malnourished children in the overall population would not be recruited: CNP services are delivered at the level of participating children and their mothers while improvement in nutritional status is expected as an impact at the level of the overall population.
Process outputs and outcome: attendance at the services and recovery from underweight
One can assume that high attendance rate at the monthly weightings should be considerably helped by the fact that the other two services were delivered weekly. Moreover, the weekly food supplement for the child could be one of the reasons for the mother's regular attendance at the weekly education sessions. However, attendance at education sessions was better than attendance at food supplement delivery, which was only half the CNP objective. This may be explained by a low fidelity of implementation: the supplement was not delivered to project staff on time or in sufficient quantities, especially at the beginning of the programme. Over time, these practical points improved and the two weekly services showed higher attendance. Moreover, in the three cohorts that followed the cohorts reported here, the attendance at the education sessions reached 71–86% of the total number of sessions planned and in five new CNCs opened in Diourbel, the coverage of the education sessions in their first three cohorts increased from 65% to 81% (République du Sénégal, AGETIP, Projet de Nutrition Communautaire 1996–1999).
The higher rate of initially underweight children in the groups of mothers who presented the best weekly attendance suggests that CNC workers were efficient in raising mothers’ awareness of the nutritional status of their children and of the importance of attending the sessions. However, out of the 100 g/child/day of food supplement given to each child by the CNP, a child on average ate only 25 g (Treche 1998), the food supplement being diverted to the family. It was not possible to confirm the output hypothesis, namely, that the supplement will be consumed in addition to the normal daily diet and thereby increase total daily caloric intake. Generally, mothers were satisfied with the CNC's services, but they expected more activities, mainly diversification of the food supplement, integration of health care, literacy programs and income-generating activities (Ndiaye 1999). Indeed, whatever the quality of the services provided, their use by the mothers is often determined by conflicts with income generating activities. Even in the most efficient CNC or cohort, the expected recovery rate of 80% was not reached during the first 2 years of the CNP in Diourbel.
Potential contribution to the Community Nutrition Project's impact
The expected impact of CNP is at least a stabilization, or at the best, a decrease of the prevalence of malnutrition of children in the population of the target neighbourhood. Among process outcomes, recovery from underweight in children under programme can have a direct effect immediately on impact, whereas changes in feeding practices of the mothers could have a less direct or immediate effect provided they can have benefit on child growth. A wrong underweight diagnosis, as well as applying incorrect criteria for recruitment, harmed the quality of individual targeting and therefore the CNP reach of underweight children in the overall population, thus compromising the potential extent of the direct impact. Irregular supplies of supplement could certainly have reduced, first, the direct impact through supplement consumption, if any, and, second, the attendance of mothers to education sessions; this problem needed to be addressed through appropriate action. High attendance of mothers to growth monitoring and promotion probably improved mothers’ awareness of the child's growth, but cannot be considered to have a positive impact alone as its efficiency remains to be shown (Save the Children UK 2003; Hossain et al. 2005). Good attendance to growth promotion and temperate attendance to education sessions cannot by themselves ensure effective performance in mothers training. However, we showed, in another city in Senegal, that changes in knowledge, attitudes and practices of mothers were generally favourable during the first 2-year period (Méjean et al. 2004). Therefore, a benefit for impact could be expected. It has to be noted, further, that behaviour changes resulting from education could strongly depend on the context, e.g. household resources.
Finally, beyond process results, an explanation for lack of recovery could rely on the design of choosing underweight as nutritional criterion. Indeed, many underweight children are stunted but not necessarily wasted. The fact that wasting or stunting are not differentiated by weight-for-age index could prevent the right action to be carried out toward the right problem.
The CNP leaders and workers took heed of process evaluation results thus improving implementation successfully as demonstrated also by process results from following cohorts carried out after the four ones studied here. Such accountability is likely to be one of the ways that could contribute to enhance potential of impact.
This study concerns one of the more recent large scale community nutrition programs. Evaluation was integrated into the CNP at the design phase (World Bank 1995) as now clearly recommended (Habicht et al. 1999). Independent and transparent presentation of the process evaluation of such interventions is needed to show what works in different contexts. From our study, it appeared that: a wrong design of underweight diagnosis could largely compromise reach of targeting and, moreover, could imply a risk of disadvantage towards girls; even with clearly defined design and objectives, incorrect criteria for enrolment can be frequently applied; in this context of program follow-up, growth monitoring could be effectively implemented in routine settings and on a large scale; dependence on the supplement supplier led to low fidelity of implementation. As a more in-depth analysis of data from CNP monitoring system, moreover done by an independent source from CNP, our study allowed assessing to what extent could the performances of an intervention be misinterpreted or exaggerated, as also already reported (Save the Children UK 2003). Training and supportive supervision are at heart of a well functioning program. Contrary to inadequate training and supervision of nutrition workers reported for two other large scale nutrition projects in Africa (Save the Children UK 2003), CNP ensured training of all CNC workers by local consultants or training institutions, and only the best supervisors among those who entered the training were selected. Each supervisor looks after five teams each managing a CNC. This positive point is favourable to a possible generalization of the implementation as described here.
- Top of page
The present findings showed to what extent the process moved in the expected direction. The results were used during the course of the intervention to provide recommendations to CNP leaders and workers for subsequent phases where changes occurred in the process, mainly in underweight diagnosis and targeting, and in the provision and delivery of the supplement. Beyond a globally satisfactory process, the programme had to analyse the different causes of the marked discrepancies observed between CNCs concerning the utilization and the effectiveness of the services, either because of CNCs’ workers initial or further training, the quality of their work, or how they dealt with the acceptance of the targeting selection by the population.
Process evaluation results could reflect the effectiveness of an intervention provided that delivery was adequately carried through, and therefore it has an effect on the interpretation that can be made of the final results (Save the Children UK 2003). From CNP, that implemented the contracting approach as a strategy to provide quality services on a large scale (Marek et al. 1999), lessons can be learned about factors affecting targeting, service delivery and a hypothesis can be formulated about the obstacles to nutritional recovery. In conclusion, this study confirms that process evaluation, based on clear terms of reference and ongoing monitoring data, is essential to assess potential of impact of an intervention. On the basis of this study, it could be recommended that large scale nutrition programme focus on targeting design and process to avoid under coverage and leakage, and it is suggested to choosing an appropriate anthropometric indicator to ensure each specific nutritional problem to be adequately addressed.