Experiences and lessons learned for programme improvement of micronutrient powders interventions

Abstract Continual course correction during implementation of nutrition programmes is critical to address factors that might limit coverage and potential for impact. Programme improvement requires rigorous scientific inquiry to identify and address implementation pathways and the factors that affect them. Under the auspices of “The Micronutrient Powders Consultation: Lessons Learned for Operational Guidance,” 3 working groups were formed to summarize experiences and lessons across countries regarding micronutrient powder (MNP) interventions for young children. This paper focuses on how MNP interventions undertook key elements of programme improvement, specifically, the use of programme theory, monitoring, process evaluation, and supportive supervision. Methods included a review of published and grey literature, interviews with key informants, and deliberations throughout the consultation process. We found that although much has been written and published about the use of monitoring and process evaluation to inform MNP interventions at small scale, there has been little formal documentation of lessons for the transition from pilot to scaled implementation. Supervision processes and experiences are not documented, and to our knowledge, there is no evidence of whether they have been effective to improve implementation. Improving the efficiency and effectiveness of interventions requires identification of critical indicators for detecting implementation challenges and drivers of impact, integration with existing programmes and systems, strengthened technical capacity, and financing for implementation of effective monitoring systems. Our understanding of programme improvement for MNP interventions is still incomplete, especially outside of the pilot stage, and we propose a set of implementation research questions that require further investigation.

23 months of age (De-Regil, Suchdev, Vist, Walleser, & Peña-Rosas, 2013). However, results from programme evaluations in a number of countries have highlighted challenges in coverage, appropriate use, and adherence and show only modest biological impact (Rah et al., 2012).
There is increasing recognition in the field of nutrition that more and better information is needed to guide the design and delivery of programmes (Habicht & Pelto, 2014;Neufeld, Piwoz, & Vasta, 2016).

| METHODS
A consultative group consisting of 49 practitioners with knowledge in the implementation of MNP interventions was formed. The process is described in the executive summary of this series . Briefly, under the auspices of "The Micronutrient Powders Consultation Lessons Learned for Operational Guidance," three working groups (WGs) were established: planning and supply (WG1); delivery, social and behaviour change communication, and training (WG2); and monitoring, process evaluation, and supportive supervision for continual programme improvement (WG3). The focus of the consultation was to review interventions that were fairly well established and scaled, targeting children 6-23 months of age. However, as the consultative process unfolded, learnings from pilots and programmes with a wider target age (up to 59 months of age) were included, as well as some relevant lessons from emergency settings.
Each WG was charged with synthesizing available evidence from programmatic settings. The outcomes of this effort are presented in this paper for WG3 and elsewhere in this series for WG1 (Schauer et al., 2017) and WG2 (Reerink et al., 2017). WG3 consisted of 2 co-chairs (LMN and AT) and 12 participants working for governmental institutions, multilateral and international organizations, universities, as well as independent consultants. WG members were based in Indonesia, Kyrgyzstan, Mexico, Mongolia, Peru, the Philippines, Switzerland, Tanzania, and the United States. WG3 participated in a yearlong (July 2015-July 2016) consultative process. It held two teleconferences to define the scope of the WG topic, participated in a meeting on October 19 and 20, 2015, in Washington, D.C., United States, exchanged emails, conducted key informant interviews, and reviewed literature.
The WG obtained primary data from key informants identified using purposive and snowball sampling (Table 1). Key informants either completed a questionnaire or were interviewed using the same structured questionnaire (Supporting Information S1). Follow-up with key informants to confirm data and seek additional information was performed as necessary. WG members involved in implementation also completed questionnaires or were interviewed. Data were analysed by collating the information into a spreadsheet and identifying relevant information.
We also identified key informants to provide information for case studies to take a more in-depth look at context-specific learning. Key informants provided expert opinion as part of their professional capacity and regular public health practice. Thus, the activities involved in the consultative process did not meet the human subjects research definition and were considered exempt by the John Snow, Inc. Institutional Review Board.
Interview participants were told that their names would be confidential in all reports and manuscripts and that any information gathered would be summarized in manuscripts submitted for peer-review publication.

Key messages
• The ability to make evidence-based decisions to improve micronutrient powders programme implementation is hindered by the lack of documented micronutrient powders experience, particularly for programmes implemented at scale.
• Monitoring and process evaluation more effectively inform programme improvements when based on programme theories operationalized through tools such as programme impact pathways and a shared learning agenda.
• Financial resources and technical capacity for monitoring and evaluation activities rely heavily on external institutions for support; support is often discontinued after the initial research stages of a programme.
• For effective programme improvement throughout the progression of the programme to scale, monitoring systems require prioritization of the information to be collected, and explicit feedback loops for viable data compilation, interpretation and utilization.
• Making evidence-based programme improvement continual and effective requires close collaboration among micronutrient powders programme implementers and technical/research partners, careful planning, and adequate budget.
• A deeper understanding of factors that drive micronutrient powders coverage, utilization and impact, and at the same time are feasible to measure in resource constrained settings, is required to improve the efficiency and effectiveness of programme improvement activities.
The WG obtained secondary data from a systematic search of peer-reviewed and grey literature. The search inclusion criteria were implementation learning on MNP from inception through December 2015 and included a screening of abstracts, along with full texts when required, as described in more detail in the executive summary of this series . A broad interpretation of relevance was applied when selecting literature to maximize the potential secondary data.
In this paper, we identify and summarize barriers and factors that facilitate adequate implementation of MNP interventions, highlighting examples from individual or multiple countries as appropriate. This analysis is not designed to provide results from any individual country. This analysis is also not designed to provide a detailed review of programme design, nor does it delve into specific monitoring indicators for programmes to use, which are covered elsewhere (HF-TAG, 2013) and to a limited extent discussed in the delivery paper (Reerink et al., 2017). Rather, we focus on the existence and quality of systems for collecting, compiling, analysing, and using monitoring and research data for programme improvement. The findings from this review are presented as a series of statements that relate to current practice, followed by details of the findings from countries on which these statements are founded. Terms and working definitions for the content of this paper, defined based on literature and key informants, are presented in Box 1. The authors acknowledge that other definitions may apply outside the context of this paper.

| RESULTS
Sixty-six peer-reviewed articles, 16 guidance documents, and 45 programme reports or conference presentations with information on MNP programme implementation experiences were identified and reviewed . Twenty-one documents were identified as relevant for monitoring, evaluation, supervision, or programme improvement. Twenty key informants (KIs) were interviewed, completed questionnaires, or participated in the development of case studies (Table 1). Lessons from 22 countries in all six WHO geographic regions were included, some with multiple experiences with MNP pilots and programmes; case studies from Bangladesh (Box 2; KI 11) and Kyrgyzstan (Box 3; KIs 4 and 13) illustrate key experiences on the topics of this paper.
3.1 | A clearly articulated programme theory has been developed in many MNP interventions but has rarely been used throughout design and implementation to track progress and make course corrections A common characteristic of MNP interventions considered to be effective is the clear mapping of programme theory, also known as a theory of change, through the use of a logic model. A logic model is essential to clearly articulate how activities will lead to impact. Moreover, the logic model often sits with programme evaluators or Box 1: Definitions of terms used in programmatic research by working group 3 (WG3): "Monitoring, Process Evaluation and Supportive Supervision for Continual Program Improvement" a Efficacy: demonstration that an intervention and/or a product "does more good than harm under optimum conditions" (Flay, 1986).
Evidence of efficacy is demonstrated through randomized controlled trials, and is necessary but not sufficient for effectiveness.
Effectiveness: demonstration that an intervention and/or product "does more good than harm when delivered via a real-world program" (Flay, 1986), i.e. whether it can produce the desired results when delivered in a programmatic context. Effectiveness is generated through program impact evaluation and given the diversity in logistical, administrative, political, social, and other factors across settings, results of impact evaluation in one setting may not be generalizable to others.
Logic model: a depiction (usually as table or figure) of the logical sequence and intended relationships between program inputs, activities, outputs, and outcomes (Kim et al., 2011;UNICEF, 2002).
Logical framework (logframe): transforms the logic model into specific indicators of process, output and outcome, intended to measure the progression and impact of the program as planned (Kim et al., 2011;UNICEF, 2002).
Program impact pathway (PIP): provides an explicit representation of the pathways by which the program (activities) achieve intended outcomes, taking into consideration non-program factors (biological and/or contextual) that might facilitate or impede such impact (Kim et al., 2011;Rawat et al., 2013;Rossi et al., 2004).
Monitoring: continual tracking of inputs, activities and sometimes outputs of a program to assess performance against plans and identify areas for improvement. Monitoring may be internal (often referred to as routine monitoring), as part of continual systems, or external to the programs. For MNP programs, a detailed monitoring manual is available that lays out information to be collected and why, as well as alternatives of systems and processes to collect and utilize this information (HF-TAG, 2013).
Process evaluation: looks inside the so-called 'black box' of program implementation to see what happened in the program and how that could affect program outcomes or impacts (Saunders et al., 2005).
Supervision: review of documents and/or observing performance; often compared to check lists; standards of care, or other tools.
Supportive supervision: collaborative effort that involves discussion and joint problem-solving, specifically designed to create opportunities to improve performance and gain confidence of workers (Also: mentoring).
Quality improvement (QI): systematic and continuous actions designed to lead to measurable improvement in program delivery and the health status of targeted groups. May utilize routine monitoring data and will include supportive supervision, but goes beyond with specific strategies to assess and improve performance through accreditation, performance-based incentives, or others.  (Table 2) and the Kyrgyzstan Gulazyk Home Fortification Programme (Table 3).
Programme impact pathways (PIPs) go a step further than logic models by explicitly mapping the causal pathway of how an intervention results in biological impact (Neufeld et al., 2016)  timelines, and engaging with the programme implementation and management teams . The PIP approach to guiding process evaluation was also used to inform nutrition and WASH interventions (Mbuya et al., 2015).
3.2 | Monitoring systems that rely heavily on external support may not be viable when transitioned to scale Target population 4 million children 6-59 months of age Monitoring system The monitoring system is financially supported by CIFF and was developed jointly by CIFF, GAIN, and BRAC, with on-going data collection led by BRAC. Shasthya Shabekas and their supervisors complete monthly reports on supply, distribution, training, and social mobilization to promote good IYCF practices. Local reports are compiled and submitted on a monthly basis. These monitoring data are reviewed and findings fed back into the system to address identified challenges and improve performance.
A research and learning agenda for program improvement In addition to the strong monitoring led by BRAC with technical inputs from GAIN, the program has benefited from a robust set of research activities, including in-depth formative research, to understand barriers to improved IYCF practices and utilization of MNP. This includes coverage, utilization, barriers and opportunities for improvement, process and impact evaluation, and testing of alternative models as part of small implementation research studies. In an effort to consolidate all monitoring, research, and evaluation findings, and facilitate their interpretation and utilization for program improvement, GAIN with CIFF, BRAC, and icddr,b worked together to develop a learning agenda for the Bangladesh MIYCN Home Fortification Program Phase II. Continual update and discussion of the learning agenda permits triangulation of information across multiple sources, the consolidation of lessons learnt and their implications for program design and implementation and for emerging research priorities. It also serves as a repository to capture key messages and program modifications made as a result of those. It is a "living document" that all program partners routinely update with details of surveys or studies, key recommendations, and any program modifications that resulted from the research. The document is managed by GAIN and is in the process of developing a web-based system soon, all partners are actively involved in keeping information up-to-date and in interpretation.  (Suchdev et al., 2010). After the study period when external financing and technical support from international partners was withdrawn, monitoring activities were scaled-back substantially by the implementing non-governmental organization. After research support ended, continued monitoring activities relied on reporting systems maintained by the implementer, rather than household-level data.
These continued activities focused on the supply and distribution of MNP and promotional materials, as well as promotional activities undertaken by distributors  Target population Children 6-24 months of age. Reached around 250,000 children at national scale.
Monitoring system Data collection by health care providers through health records. Supply data, coverage, and reasons for refusal were collected locally and then aggregated up the chain to the oblast-level.
Process evaluation Household surveys carried out by specially trained and hired survey staff, and health care personnel completed a short questionnaire. Indicators included product availability, coverage, adherence, KAP regarding MNP, receipt communications materials; knowledge and skills of medical workers, the quality of reporting documents.
How monitoring and process evaluation complement each other The health system required rigorous recordkeeping for transparency and accountability purposes. This local data collection allowed decision makers to pinpoint where problems existed, something that surveys could not do. However, the records were not detailed enough to track the moving denominator (a result of children aging in/out of the program or migrating). Household surveys, on the other hand, were able to explore more in-depth questions and provide representative estimates of complex indicators.
Transition from pilot to national scale Both monitoring and process evaluation activities took place throughout the pilot and into the national scale-up. Collecting similar data during the pilot allowed implementers and researchers to triangulate findings, and consistency between estimates indicated that internal monitoring systems were performing well. Process evaluation activities were slowly rolled back as the program scaled, with household surveys sampled using the Lot Quality Assurance Sampling method, instead of the more resource-intensive proportional to size sampling of the pilot, before they ended with the study in 2013. The monitoring work continues, although supply difficulties have hampered its use in recent years.   and 15). This lack of engagement with programme staff, along with limited continuity with clients, makes reporting and monitoring of side effects, dropout, and intake adherence essentially non-existent. This problem was partially overcome in Somaliland through the creation of a subgroup of 200 caregivers. Active follow-up with these caregivers was done to inquire about any issues they were experiencing (e.g., side effects), and the information was used to modify the intervention. The Somaliland social marketing programme also created a free call-in hotline where caregivers could access support and additional information on the MNP (PSI, n.d.).

| Adequate allocation of financial and human resources for monitoring activities is essential but limited in many countries
In the most recent UNICEF NutriDash Global Report (2015) to ensure programme progress were identified as gaps (KIs 1, 2, 4, and 16). In Tanzania, for example, community health volunteers were expected to monitor consumption in addition to distribution (KI 1).
The data on consumption monitoring were considered unreliable in this programme due to the complexity of the topic, the low capacity and inadequate training of the health workers, and the lack of systems to ensure accountability.
3.4 | To be effective for programme improvement, monitoring systems require prioritization of the information to be collected, and explicit feedback loops for data compilation, interpretation, and utilization 3.5 | Process evaluations are intended to complement monitoring systems and have been used in formative stages of a new programme, but more learning on how to improve implementation at scale is needed Process evaluation activities are used for course correction of implementation (formative use) and to explain programme outcomes (summative use) (Habicht & Pelto, 2012;Saunders, Evans, & Joshi, 2005).
Process evaluation is not intended to replace routine monitoring for timely feedback and corrective actions (Kim et al., 2015), as it generally requires extensive fieldwork, has a longer duration, and provides periodic versus continual feedback. Instead, process evaluation provides a limited-duration opportunity for understanding the "how" of programme implementation in a way that allows for more meaningful discussion of changes needed to improve effectiveness.  (Afsana, Haque, Sobhan, & Shahin, 2014;Angdembe, Choudhury, Haque, & Ahmed) and three large-scale MNP distribution in refugee camps and in emergency contexts (de Pee et al., 2007;Kodish, Rah, Kraemer, de Pee, & Gittelsohn, 2011;Rah et al., 2012).
Programmes in the pilot phase tend to have a stronger research component including greater process evaluation with external surveys and more intense data collection, whereas full-scale programmes tend to switch their focus to routine monitoring alone. In situations where a programme is well established and many of the key process questions have already been examined and answered, a routine monitoring system may be sufficient to ensure regular updates on progress.
However, many implementers stated the value to continuing process evaluation beyond initial pilot stages or the beginning of a national programme, but the burden of data collection and analysis makes this difficult (KI 13).
An example of a programme with a strong research plan for the pilot stage is the MNP programme in Kyrgyzstan (Box 3), which included internal monitoring activities complemented by extensive external impact and process evaluations (Table 3  Bangladesh (Angdembe et al., 2015) Implemented in 61 districts of the country at the time of study by BRAC, a national NGO (2012) To assess adherence to MNP intake regime and associated factors in a community setting Cross-sectional study using quantitative methods Interviews with caregiver using a semistructured questionnaire Bangladesh (Afsana et al., 2014) Implemented in 61 districts of the country at the time of study by BRAC, a national NGO (2013) To describe BRAC 0 s experience and achievements in scaling-up a nationwide MNP programme Mixed quantitative and qualitative methods Periodic monitoring surveys, process evaluation survey, and rapid qualitative assessment China (Sun et al., 2011) Pilot [2 counties in Shan 0 xi province] (2008 & 2010) To test the concept of public-private partnership to deliver MNP+ and to evaluate the effectiveness of marketing MNP+ through publicprivate partnership Two cross-sectional studies, convenience sample using quantitative methods

Cross-sectional household surveys
Haiti (Loechl et al., 2009) To assess the feasibility and acceptability of distributing MNP through a food-assisted maternal and child health and nutrition programme using a programme theory framework in order to document programme processes  (Creed-Kanashiro et al., 2015;Kodish et al., 2011;Korenromp et al., 2015;Loechl et al., 2009), and three conducted in-depth interviews with key informants or focus groups (Kodish et al., 2011;Loechl et al., 2009;Suchdev et al., 2010).

| Supervisory systems and how they should be used to improve implementation is poorly documented for MNP interventions
Despite the general consensus among key informants that supervision for the purpose of quality improvement is important, we were unable to locate documentation related to supervision systems or processes for MNP interventions. Nor did process evaluations specifically address the issue of supervision and its role in improving programme implementation. Key informants reported that programmes infrequently collect data during supervisory visits and the potential role of supervision in programme improvement in general does not appear to be prioritized in MNP programmes (KIs 5,11,12,16,and 17  the awareness of the importance of such measures by stakeholders and decision makers. The findings presented here are similar to those of a recent review of government information systems to monitor complementary feeding programmes for young children, which concludes that all programmes need internal monitoring to implement effective programmes. The review highlights the use of programme description and conceptual models to develop a monitoring system, including indicators and tools, that is feasible to implement and maintain throughout the programme and can be used for programme improvement. Complementary feeding indicators should fit their context and provide data that can be part of a decision making and programme improvement process (Jefferds, 2017 with representatives from all stakeholder groups. When possible, the authors attempted to verify details with information from another data source. Although many countries and international organizations were included in this process, the authors of these papers acknowledge that some countries and experiences that may have added to the learning were not included. This type of summative process on programme experiences is still fairly methodologically new (Green, 2008) and subject to a certain amount of subjectivity. The subjectivity in our approach includes our snowball sampling, the use of expert opinion as the primary data and subject to author interpretation and biases. The lack of published or documented experience, particularly for at-scale programmes and supportive supervision, means that the results presented here are indicative of the situation but cannot be considered a definitive or exhaustive review of the issues.
To date, most learning on MNP interventions has been from programmes implemented at small scale with a high level of external technical assistance, financing, and process evaluations during the start-up phase. The extent to which the evidence generated from these evaluations has informed subsequent changes in programme design and/or implementation, however, is often not adequately captured in programme reports or publications. Similarly, the high level of external support during start-up phases has not necessarily translated into strong monitoring or ongoing evaluation as programmes transition into routine monitoring systems. Most platforms used for MNP delivery have limited human and financial resources to sustainably adopt new indicators, collect and analyse data, and report results. Successful integration of evidence-based programme improvement activities into routine systems requires technical support (at a minimum at the programme initiation stage), careful planning, and adequate budgeting. Capacity assessment and development (if needed) should be included during that phase to ensure systems can continue to inform programmes and evolve once external support is reduced.
Understanding the processes through which programmes achieve high acceptance of MNP and high adherence to recommendations regarding the use of MNP is critical to informing programme improvement, replication, and scale-up. Yet although programme theory frameworks, such as logic models and PIPs, are considered necessary, the randomized control trials dominating the MNP literature are rarely designed to pinpoint implementation constraints. As a result, the biological pathway to impact for MNP is well articulated but programme implementers have insufficient knowledge of the processes through which successful delivery of the intervention could be expected to happen. This observation echoes other recent calls to improve the capacity of nutrition researchers to measure intermediary steps between programme inputs and biological outcomes and assess implementation fidelity (Habicht & Pelto, 2012). Programme implementers need this information to identify the critical path to impact, prioritizing what information is necessary and sufficient to inform subsequent action.
Improving the efficiency and effectiveness of programme improvement activities including monitoring, process evaluation, and supportive supervision requires identification of a small set of indicators critical for detecting implementation challenges, integration with existing programmes and systems, strengthening technical capacity, and financing for its implementation. More implementation research on programmes at scale and greater efforts to document and share research results and programme experiences is also needed.
Areas of needed implementation research identified during the consultative process include the following: • Examine how to effectively link monitoring and process evaluation to decision-making processes; • Document lessons in how to sustain monitoring systems from pilot to larger scale; • Assess how to manage monitoring across multiple, integrated interventions; • Document how to carry out effective supportive supervision, especially in contexts with high turnover of MNP staff; and • Determine how to link MNP interventions to broader health systems, strengthening activities at all levels.

CONFLICTS OF INTEREST
The authors declare no conflicts of interest.

CONTRIBUTIONS
This paper was written by MV, AT, LMN with substantial contributions from AD, ABA, RG, CI, LI, GM, NMS, and NT. AT and LMN chaired the working group, and AD served as the secretariat. All authors were involved in developing the paper concept and have reviewed and approved the submitted manuscript.

SUPPORTING INFORMATION
Additional Supporting Information may be found online in the supporting information tab for this article.