Nutrition modeling tools: a qualitative study of influence on policy decision making and determining factors

Abstract Nutrition modeling tools (NMTs) generate evidence to inform policy and program decision making; however, the literature is generally limited to modeling methods and results, rather than use cases and their impacts. We aimed to document the policy influences of 12 NMTs and identify factors influencing them. We conducted semistructured interviews with 109 informants from 30 low‐ and middle‐income country case studies and used thematic analysis to understand the data. NMTs were mostly applied by international organizations to inform national government decision making. NMT applications contributed to enabling environments for nutrition and influenced program design and policy in most cases; however, this influence could be strengthened. Influence was shaped by processes for applying the NMTs; ownership of the analysis and data inputs, and capacity building in NMT methods, encouraged uptake. Targeting evidence generation at specific policy cycle stages promoted uptake; however, where advocacy capacity allowed, modeling was embedded ad hoc into emerging policy discussions and had broader influence. Meanwhile, external factors, such as political change and resource constraints of local partner organizations, challenged NMT implementation. Importantly, policy uptake was never the result of NMTs exclusively, indicating they should be nested persistently and strategically within the wider evidence and advocacy continuum, rather than being stand‐alone activities.


Introduction
Appreciation of the role of nutrition in human, social, and economic development has led to increased nutrition-focused policy, public investment, and donor contributions in low-and middleincome countries (LMICs). 1,2 Current resourcing remains insufficient, however, and advocacy to strengthen commitments long-term is needed, as is the concurrent prioritization of the most urgent and most effective nutrition issues and programs to receive limited resources. 2 There is a recognized need for the rapid generation of context-specific evidence to answer relevant questions for nutrition policy and program decision making and inform resource prioritization, guide regulatory and evaluation activities, and encourage further commitment and investment toward improving nutrition. 3 A number of nutrition modeling tools (NMTs) have been developed by researchers working in academic institutions and international organizations to generate the evidence needed to answer some of the questions relevant at different nutrition policy and program cycle stages. Most of these NMTs differ in terms of data inputs, analytical methods, and results that can be generated, but are common in that they apply mathematical modeling and are intended to be adopted and used, or accepted and supported, by decision makers or their advisors to inform nutrition policy and programs in LMICs. Several NMT developers are members of the Nutrition Modeling Consortium (NMC), which aims to increase the connection between NMTs and to encourage the uptake of results. 2 An overview of the Consortium and functions that can be performed by the different tools is provided on the NMC website (https://www.nyas.org/programs/ nutrition-modeling-consortium/).
Despite often widespread application of NMTs, the literature to date has been largely limited to model development, methods, and modeling results from individual analyses, rather than how the tools and evidence they generate are applied and any influence this has had on policy or program decisions. [6][7][8][9][10] Anecdotally, NMC members have shared that NMT results have not always had the desired use or program or policy influence in-country, and that there is a need to understand how to strengthen this. Simultaneously, there is increasing pressure on researchers and technical assistance providers to demonstrate the impact of their evidence-generation methods on policy processes and outcomes to justify approaches and resource allocation and encourage decision makers in other contexts to apply similar methods. 11 However, attributing policy change to a single piece of evidence is difficult and mostly unrealistic. 12,13 Individual analyses are unlikely to be comprehensive or convincing enough to be used as the only rationale for significant policy and related investment decisions since additional questions will inevitably remain. 11,12,14,15 Therefore, the influence of individual pieces of evidence, or the ability to measure their impact in a simple way, should not be exaggerated. 11 That said, evidence can be catalytic in kicking off new discussions or nudging forward slow or stalled policy processes, which could contribute to significant policy decisions. 12,13 NMT applications may not have directly resulted in changes to policy implementation, yet the evidence generated could have been instrumental in changing the framework in which nutrition problems and solutions are viewed and shaping aspects of the policy process, including generating consensus, creating an enabling environment for a cause, or contesting proposed actions or positions. 13,14,16,17 For the above reasons, rather than attempt to objectively measure and compare policy impact across NMTs or NMT applications, the aim of this study was to draw on user experiences from a large selection of NMT case studies to document reported decision-making applications of the tools and their results and identify internal and external factors to the tool application process that limit or enhance the extent to which the modeling is considered and used by decision makers. Furthermore, this study explores end-user perceptions of NMTs and their use in general and does not seek to measure and compare the potential and accomplishments of individual tools. NMTs relevant to this study and the program and policy questions they can answer are outlined in Table 1 and Table S1 (online only). 7,10,[18][19][20][21][22]

Methods
This qualitative study involved semistructured interviews with end users of 12 different NMTs across 22 LMICs (Table 2). Nutrition Modeling Consortium members listed all applications of their tools in which results were disseminated in the country of application within the past 7 years. This time period was set to encompass applications of all relevant NMTs while still being recent enough for respondents to recall details of the activity. A target of three country case studies per NMT was set to capture different analysis processes and geographical variation. Fewer case studies were targeted for tools that were new or still under development, and two of the 14 NMC tools were excluded since appropriate case studies could not be identified. Thirty case studies were then randomly selected from the list of NMT applications. The tools represented by the case studies were: Cost of the Double Burden; 23 Cost of Hunger; 24 Cost of Not Breastfeeding; 22 Cost of the Diet; 25 Fill the Nutrient Gap; 26 34 Interview participants were selected using purposive and snowball sampling 35 to represent three end-user groups for each case study, as defined by NMC members: 36 (1) "Brokers" who know about an NMT and may recommend, coordinate, or fund its application; (2) "Technical analysts" who conduct analysis using the tools; and (3) "Consumers" who would ideally use the modeling to inform advocacy and decision making (Table 3). Sampling continued until all end-user categories were represented and information saturation occurred, in terms of new data repeating what had been expressed in previous interviews. 35,37 In total, 109 key informant interviews were conducted with: technicaland policy-level government staff (n = 25) from national health, planning, and agriculture ministries; local and international program staff from United Nations (UN) agencies (n = 34); nongovernmental organizations (NGOs) (n = 31); and academic researchers (n = 19) (Table 4). Although there were no recorded refusals, participation was affected by connectivity and scheduling challenges.
The conceptual framework for the study was derived from a review of gray literature describing NMT applications and policy influence, NMC meeting reports, and scoping interviews with NMC members. Six key concepts (reasons for analysis, analysis process, stakeholder involvement, dissemination, use of results for advocacy or decision making, and factors influencing this) were used to develop the interview guide (Table S2,   Country-based representatives from donor organizations, and managers and nutrition staff from international organizations managing a tool application process.

Technical analysts
Those who conduct an analysis using a nutrition modeling tool.
Local technical nutritionists, epidemiologists, economists, or other analysts and international specialists, researchers, or consultants.

Consumers
Policymakers and program planners (or their advisors) who may ask initial analysis questions and would ideally use the modeling results to inform decision making or for advocacy.
Management and higher-level government staff, including SUN coordinators, government unit directors, and ministers, as well as locally based managers or directors from NGOs or other organizations.  7  7  5  19  Affiliation  Academia  4  12  3  19  Government  2  6  17  25  NGO  13  10  8  31  UN agency  20  8  6  34  Sex  Female  26  19  20  65  Male  13  17  14  44  Total  39  36  34  109 r Perceived influence of the modeling and factors that influenced this (Q14-Q24) Interviews were conducted by the first author between August 2018 and March 2020, mostly in English, otherwise in Spanish or Portuguese, in person or via phone/web conference. Participants were provided with an information sheet and asked to provide consent after discussing study aims, confidentiality provisions, and any questions. Inter-views were digitally recorded with permission, transcribed verbatim and, if necessary, translated into English. Detailed notes were taken when recordings were not possible. Transcripts and notes were checked and edited to aid understanding. NVivo 12 was used to organize, code, and analyze deidentified data from the interviews. 38 Approval for this study was obtained from the LSHTM Research Ethics Committee (ref. 16729).
The data were explored using reflexive thematic analysis (RTA), which is useful for the categorization of large datasets and to examine the experiences, knowledge, and opinions of diverse stakeholders. 39,40 In RTA, themes are created inductively and deductively throughout the analytical work as opposed to being found in the data alone, meaning the approach is flexible and allows the development, review, and sorting of codes and data across different stages of the analysis. 39,40 The analysis followed the six-step RTA process outlined by Braun and Clarke, with movement back and forth between steps. [39][40][41][42] (1) Interview transcripts and notes were reviewed to provide familiarity with the data. (2) Data were deductively coded using broad codes from the conceptual framework; 41,42 more detailed codes were then developed inductively and applied semantically to explore topics raised in the interviews. 41,42 (3) Themes, defined as patterns of shared meaning underpinned by a central concept or idea, were generated by grouping codes and coded data. 39,40 (4) Transcripts and notes were reread, and themes edited to ensure coding completeness and that the themes were useful and accurately represented data. (5) The scope and focus of each theme was defined and named. (6) Demonstrative quotes were selected and organized by theme to highlight findings across enduser groups.
Tools and case-study countries were not identified in the analysis in order to protect the identity of participants and avoid comparison of the features and methods of tools themselves or organizations applying them. Instead, case studies were grouped by their geographical regions, policy stage, and general tool characteristics (e.g., tools that translate nutrition issues to social cost).
Results are presented across the three following sections: (1) tool application processes, including reasons for selecting NMTs, processes for engaging and working with stakeholders, planning, carrying out and disseminating analysis; (2) analysis objectives and influence, covering analysis objectives, reaction to and use of results, and perceived influence of these; and (3) internal and external factors affecting the use of modeling results and their influence on program and policy decision making.

Processes for applying NMTs
The case studies were varied in terms of reasons for selecting the NMTs that were used, partners involved, who did the analysis, and how results were disseminated. While the countries using the tools were LMICs, most analyses were initiated by international NGOs, UN agencies, and European or North American universities. Occasionally, NMTs were applied in response to government requests to these organizations for assistance to answer key policy questions or use data that had been collected for another purpose. This generally happened when government partners were already aware of analyses that had previously been done by these organizations or their particular technical capacity. In very few cases, there was existing capacity to use a particular tool in country or knowledge about one or more NMTs, which prompted tool use. Government inputs at this stage were predominantly focused on whether to endorse a proposed NMT analysis, instead of deciding which tool could answer necessary policy questions.
Government partners oversaw, conducted, or validated NMT analyses in nearly all cases, especially when they aimed to influence national decision making. Table 5 summarizes six distinct processes for technical government partners or other local entities to engage with NMTs or their NGO, UN, or academic brokers. These included: (1) via preloaded online dashboards; (2) independently accessing and using NMTs; (3) being trained and supported to apply NMTs on an ongoing basis; (4) being trained to use an NMT for a specific analysis; (5) preparing inputs for analyses done by international specialists; and (6) reviewing analyses conducted by international specialists. With the exception of NMTs with preloaded dashboards, the engagement process was mostly determined by context and stakeholder preferences, with some tools applied differently across case studies.  NGO or UN brokers commonly described the NMT processes as being government "owned," a term which would generally imply that local stakeholders regard an analysis as theirs and are closely involved in its planning, implementation, and dissemination. 43 Explanations of how ownership was achieved, however, ranged from nominating a local analysis champion, such as a Scaling Up Nutrition (SUN) Focal Point, to appointing a government-led technical working group to review and validate analysis inputs and outputs, to government partners doing the analysis themselves. Nutrition policy mapping exercises were done in a few cases to identify relevant stakeholders and decision makers to involve in the analysis or dissemination and determine how and when they should participate.
We (SUN Coordinator and team) were involved from the beginning, engaging partners and facilitating the whole process. We worked closely with the analysis team.
Government Consumer 018, Southern Africa Local analysts were trained and supported to conduct the analysis in several cases, where practical, which was generally considered preferable to the  more common approach of international specialists doing analyses. Issues of sustainability, both in terms of reliance on outsiders and maintaining local capacity, were raised by many respondents. In some contexts, "divisions of labor" were identified to facilitate the analysis and also promote understanding of the methods and involvement of local partners. In some cases, the NMT process expanded beyond analysis and dissemination to actively translating evidence from the modeling into advocacy. Communications and advocacy specialists occasionally supported such activities, but elsewhere, expertise in these areas and resources were lacking. Some analysts were frustrated that advocacy based on NMT results and the development of materials to support this could take significantly more time and resources than analysis, yet were not prioritized or planned for.
The main product was a manuscript, but we also did a lot of work converting it (modeling) into very quick, simple presentations for high-level decision makers.

Government Consumer 024, Southern Africa
A meeting was organized through high-level officials, and the results were used to make the case and really remind the parliamentarians of what they had committed to. NGO Broker 028, West Africa

Analysis objectives and influence
Objectives and reported uses or influences of the modeling are summarized in Table 6, by policy or program cycle stage. Almost all NMT analyses were done to inform government decision making and some simultaneously targeted multiple entry points, for example, encouraging commitments to addressing malnutrition long-term while informing the specific program designs short-term. Tools translating malnutrition into social, economic, and human capital costs were almost always applied to encourage awareness and investment in nutrition and often timed to coincide with political transitions or national budget planning.
We have a new government administration. We wanted to use the modeling to show that nutrition should be top of their list for development priorities.

UN Broker 009, Latin America
Many consumers reported that the modeling, alongside other evidence and advocacy, influenced government prioritization of nutrition, for example, commitments to nutrition in national development plans, malnutrition task forces, or the introduction of nutrition teams in nonhealth ministries.

The approval for the national food security and nutrition council was based on this (modeling).
Government Consumer 023, Southern Africa Occasionally, consumers said modeling substantiated existing policy, which helped ensure continued funding. Elsewhere, the modeling was being applied as part of ongoing advocacy efforts for specific goals.
Even if the analysis only served to validate existing policy, this was valuable because the policies prioritizing nutrition from the Ministry of Health, Agriculture, etc. exist, but strategies are not always funded. We used the results to make a case to Treasury.
Government Consumer 014, East Africa The analysis was important because even though the country has a strong policy focus on malnutrition, with this economic situation we're promising more than we can actually afford. So, they (Treasury) are looking for things to cut and we need to prove why we need to keep them. Government Consumer 007, Latin America A few analyses were important for engagement and credited with intergovernmental agencies and NGOs gaining a seat at the table for discussions at subsequent stages of a policy cycle.

The (analysis) enabled (organization) to get a seat at the table on policymaking. This helped us insert recommendations supporting the Multisectoral Nutrition Action Plan.
UN Broker 073, Southeast Asia A few analyses were done for situation assessment, to complement health and nutrition surveys, and to better understand factors contributing to, or the extent of, the malnutrition burden. Some analyses were credited with addressing misconceptions, which led to a more favorable environment for appropriate policy responses.
In the common mind of government, people had malnourished children because they were "misbehaving" and not feeding them properly, but now we have facts    geographies. This was common in contexts with significant subnational diversity and limited resources and was often a precursor to further modeling to compare the potential effectiveness of different program options. Issues or groups highlighted by the modeling commonly gained momentum and were spoken about more by stakeholders. In a few examples, new policies or programs were introduced by government or others to target vulnerabilities focused on by the analysis. Analyses at the policy or program review and development stage mostly sought to guide government decisions on which nutrition-sensitive or -specific actions to implement, where, and targeted toward whom. Few analyses focused on nongovernmental programming alone, although some included models to inform decisions for both government and nongovernmental bodies. Modeling compared the potential impacts of actual or proposed health, agriculture, food security, or social protection policies or programs on outcomes, such as nutrient intake, lives saved, nutritious diet access, and prevalence of inadequacy. The modeling was occasionally mentioned as justification for decisions, including mandatory fortification legislation, regulation on processed food use in school feeding, or inclusion of nutritious foods in social protection programs.
We did analyses on under-five mortality, stunting, and wasting reduction. We proposed five intervention scenarios, some that were being used and others that were new. The Minister of Health used our models to develop projections for a 10-year action plan.
Academic Technical Analyst 071, West Africa The government approved the ruling on imported oil; now the oil coming to this country will be fortified with Vitamin A. Government Technical Analyst 067, East Africa Elsewhere, technical-level government advisors used the modeling to advocate for specific changes, such as fortification standards or social protection transfer amounts. In many instances, however, local nongovernmental partners used the modeling for advocacy. A few NMT analyses estimated cost-benefit using data from pilot or early phase program implementation to link observed benefits with implementation costs. Such modeling informed program adoption and expansion, such as the government scale-up of an NGO pilot incorporating agricultural, early childhood development, and school feeding actions.
The data that we are generating is going into the government planning for the scale-up. When they roll out, they'll have more data coming back in to continuously inform the program.
Academic Technical Analyst 109, Southern Africa Occasionally, NMTs contributed to program evaluations, predicting the extent to which selected indicators may have changed by retroactively comparing baseline models with those based on new data. This demonstrated likely program value and identified or justified necessary alterations. In some cases, design changes required approval or further resources and the modeling was considered important for making a case to decision makers.
There was a budget excess and the donors wanted to reach more people. We're using the modeling to make a case for improving program quality instead. We agreed to give both animal-source foods and vegetables; previously households got only one or the other, so now we are doubling the support in a way.
NGO Technical Analyst 037, South Asia  Table 7 summarizes 10 factors most commonly reported as influencing the extent to which modeling was accepted and used by decision makers. We distinguish between internal factors that were largely within the control of tool users and external factors.

Factors influencing the use of NMT results
Internal factors. Reputation and acceptance of methods. Reputation of tools and their methods was instrumental; some brokers and consumers supported proposals to use a tool because the methods had been published in peer-reviewed journals and or was considered to have academic acceptance. Examples of a tool's successful use in neighboring countries also encouraged acceptance and uptake. Existing relationships were influential and government policymakers valued the opinion of donor and UN counterparts. Therefore, if such counterparts had previous knowledge or an opinion about a particular tool, positive or negative, this could influence government decisions to approve an analysis process or accept NMT findings.
As such, it was important that brokers consulted widely to obtain support for NMT analyses.

We used the results from Mexico, Ecuador, and Chile to get support for the (NMT) study. This was important as the examples were from our region. Government Consumer 001, Latin America
Targeting. Conscious messaging and targeting of key stakeholders during the analysis process and delivery of outputs was commonly mentioned as a success factor. Brokers strategically identified technical partners, analysis "owners," and advocacy recipients based on intended use and influence of the modeling. As such, some analysis working groups were led by agriculture or education departments rather than traditional nutrition partners, such as health. Similarly, at dissemination, finance ministers, treasury, and regional government leaders, who could influence resource allocation, were often singled out with specific materials. Occasionally, this was recognized by government consumers who praised the presentation of nutrition issues in a language that was relevant to them, such as economic cost.
Partners need to match the focus of the modeling but also have the capacity to move things. The "leading organization" Timing. Many brokers planned analyses to coincide with national nutrition policy reviews or program design and attributed this to successful results uptake. Similarly, delayed analyses or a lack of alignment with a specific policy stage were said to have limited the influence of the NMT. However, some consumers of NMTs with multisectoral relevance felt it was more important to strategically apply modeling results when and where needed as policy cycles differed between government ministries. Perceived complexity. Stakeholder understanding of NMT methods and results was critical for adoption and use of the modeling recommendations. Many brokers and consumers considered the NMTs complex to understand and explain to others and thought this hampered government validation or use of their results. Some local technical partners did not feel confident presenting tool outputs as they felt they did not understand it sufficiently or would struggle if asked to answer questions. Modeled estimates of program impact, potential coverage, or diet quality were often misunderstood to be actual figures from observational surveys. Misunderstandings were further compounded when materials were translated into secondary  Ownership. Almost all respondents highlighted local ownership, in terms of stakeholders being involved throughout the analysis process and considering it theirs, as important for NMT results uptake, especially when analyses aimed to influence government policy. Brokers often mentioned ownership when describing their NMT analysis processes and what had been achieved, while government consumers tended to refer to ownership when reflecting on what is needed for successful evidence generation activities.
All of the institutions put in a lot of time and effort, they weren't just passive stakeholders. It was really important as the results were then from those people in the government, it wasn't information that came in from some international group.
UN Broker 003, Latin America Respondents overwhelmingly agreed that stakeholders were more likely to accept, promote, and apply evidence that they felt reflected their contri-butions but that agreeing on a shared definition of ownership was critical. Some government end users considered technical involvement in the analysis a prerequisite for ownership, while others were satisfied with a coordinating role.
If we really want to have national ownership you have to train them (government partners) to produce results themselves.
Government Consumer 094, Southeast Asia Building capacity to apply NMTs. Representatives from almost all case studies that included capacity building to apply NMTs felt this fostered ownership and understanding, which encouraged the uptake of results. This was the case even where partners were trained but did not do the analysis. In select case studies, NMTs had been institutionalized and were being taught at local universities to public health master's students. This was believed to contribute to sustainable capacity among graduates, many of whom worked in government, to both use the tool and to understand and apply results.
We always put emphasis on local capacity building, it's more sustainable than bringing in consultants and builds knowledge (on the NMT) so we can work closely with those who will have to move forward with the actions.
Government Consumer 099, Southern Africa Some respondents felt local analysts should always be trained in NMTs, while others thought the ability to apply evidence to advocacy and decision making was a more pressing capacity need for local nutrition partners. Many brokers and consumers were concerned that infrequent NMT use or high staff turnover would mean the time and resource investment for capacity building would be difficult to justify.
We would have to invest a lot of time and funds in building up the analytical capacity of country office staff, this would be useful in the short term but as there is a lot of turnover and you probably wouldn't do it (the analysis) more than once a year, is that really useful?
NGO Consumer 040, Southeast Asia Secondary data. Data sources were commonly linked to notions of ownership. Most analyses used secondary government data from demographic health, household consumption and expenditure, dietary, or micronutrient surveys. Local partners considered these data high quality because methods were validated, and they were representative at different geographical levels and time points. Some government analysts and consumers felt they owned analyses done with their own data and that secondary analyses validated their work. A few government brokers were wary of modeling done with external data.
When studies are done by an international agency but using our data, it validates what the government is doing, it gives credence to our work.
Government Consumer 054, Southeast Asia The government ownership was very strong because we saw that it was our data and our people doing it, what we call a "home-grown solution." This was the reason for the high acceptance. Government Consumer 014, East Africa Some respondents also shared that analyses based on secondary data facilitated NMT use and, ultimately, policy application, as could be done more rapidly and using less resources than analyses that required primary data collection. In some case studies, it would not have been possible to conduct the analysis if secondary data had not been available, as funds for primary data collection were not available.
After the initial analysis we received many requests from partners, other countries, etc. This is when we started exploring the possibility of doing the analysis with "big data," removing the time, cost, and capacity needed for comprehensive data collection.
NGO Broker 032, Southeast Asia External factors. Capacity for evidence translation and advocacy. Most respondents acknowledged that the modeling could not bring about policy change alone, and that results needed to be translated into advocacy messages and strategically applied to inform or influence decision making. Many flagged capacity needs for such activities within government or NGO nutrition teams. In a few cases, advocacy training was included in the NMT process and said to stimulate broad uptake of the modeling by different government sectors. Elsewhere, individuals with strong advocacy skills were credited for embedding the modeling into government decision making. Resources to support analysis. Government consumers as well as local, nongovernmental partners raised time, staff, and resource constraints as barriers to their meaningful participation in the analysis process and use of results. Even when modeling was done externally, local partners were expected to participate in technical meetings, share data, review results, assist dissemination, and translate evidence into advocacy. Sometimes, suitable government staff could not be identified, were not available, or did not have the management support to do these activities. Occasionally, consultants were hired by local NGO or UN offices or seconded to government offices to support the process. Government partners received funds directly to support NMT applications in rare cases only. The new government had different priorities. The level of interest in overweight and obesity shifted. We will have to be patient and wait to raise this again.

Discussion
NMTs have been used to generate evidence and inform advocacy and decision making at different policy and program cycle stages in diverse LMICs. The findings provide examples of the policy and program influences NMT applications have achieved, but also suggest that there are opportunities to strengthen the engagement process, capacity to use NMTs, and capacity to apply evidence to ensure that results are used effectively. NMT applications were mostly driven by international organizations as opposed to being demanded by government decision makers. In the case of more recently developed tools, this may reflect a necessary approach to introduce and build acceptance of novel methods or approaches, similar to the experience of introducing HIV and vaccine modeling tools. [44][45][46] However, this could also be indicative of broader power imbalances between governments and research institutions in LMICs and donors from high-income countries or international organizations who can influence research or analysis priorities. [47][48][49][50][51] Many government end users shared a preference for building domestic capacity to use NMTs and local analysts were trained in some cases. More frequently, national partners reviewed model inputs and outputs, and managed stakeholder engagement. Most analyses had an objective of informing government decision making, often targeting more than one sector or policy cycle stage. Positive influences of the modeling were reported for most cases; from putting nutrition issues on the agenda, to informing specific program components, such as target group selection, transfers, and behavior change messages, to contributing to larger policy changes, including fortification and school feeding legislation.
Almost all respondents acknowledged that changes to policy implementation were not the result of evidence generated by NMTs alone. Instead, strategic advocacy based on the modeling, which often took longer and required more resources than the NMT analysis, was key, as well as complementary evidence generation, such as program costing. Furthermore, respondents credited processes that built understanding and ownership of the modeling with its uptake and use by consumers, as well as conscious planning to make analyses and advocacy relevant, targeted, and appropriate for their intended audiences. However, local partners often required greater capacity and resources to support NMT processes and translate evidence into advocacy and decision making, which limited the reach of the modeling.
As defined by Gillespie et al., "enabling environments" for nutrition occur when "political and policy processes that build and sustain momentum for the effective implementation of actions that reduce undernutrition" exist. 52 Such processes require evidence to encourage best practices, but also leadership, advocacy, technical support, and implementation capabilities, platforms, and resources. 17 Consequently, given the strength of other forces shaping policy processes, including institutional factors, context, and individual and collective policymaker values, beliefs and culture, evidence generated by NMTs is unlikely to be the sole, or even the most influential, catalyst for substantial policy change. 11,12,14,15 Indeed, most respondents acknowledged external and complementary advocacy, political processes, and evidence that contributed to the reported NMT policy influences, similar to findings from targeted studies of nutrition policy processes, which also identified the ability to achieve cohesion within policy community as an influential factor. 16,53,54 This echoes broader discussions in health policy literature that recognizes that evidence is only one of multiple elements influencing policy processes and that, depending on the context, its influence may be outweighed by other factors, such as policymaker beliefs and values. 14,51,55,56 It is also important to acknowledge the role that policymaker values and context may have in shaping the type of evidence that is generated, which evidence is used, and how it is used. 14,51,55,56 For the most part, uses of the NMT results were concerned with informing policy decisions, helping to change thinking, or positioning of organizations or topics, in addition to program design. Our findings suggest that NMTs are an important part of a broader ecosystem of evidence generation, capacity building, and advocacy that together build enabling environments for nutritionbenefitting policy. 14,16,52,57 Appreciating how NMTs can fit within this ecosystem will ensure they are applied with objectives to identify, complement, and contribute to existing advocacy, in response to endogenous demand, rather than being an end in themselves.
We found 10 factors that influenced how NMTs were used to inform policy and program decisions. Unsurprisingly, similar factors have been highlighted in strategic reviews of evidence use in public health policy, namely, local analysis and advocacy capacity, government ownership or collaboration with decision makers, reputation, targeting and institutional resources, and support. [58][59][60][61] The importance of targeting evidence generation to align with key policy cycle stages is highlighted throughout health policy literature; however, serendipitous opportunities for influence are also acknowledged. 14,15,[62][63][64][65][66] Lin appreciates that being in the "right place, at the right time, in the right company" for policy change can occur through strategic planning or by accident. 65 Parkhurst echoes comments from some respondents in pointing out that analysts may not consider the "transfer of knowledge to influence policy" as their role or as within their skillset. 61 Therefore, political actors or NMT consumers, rather than technical analysts, should carry responsibility for adopting and embedding NMT modeling within targeted advocacy activities. 63,65,67 At the same time, these political actors or consumers should seek and capitalize on opportunities for serendipitous applications of the modeling results to decision making. 63,65,67 In both instances, consumer end users, such as government or NGO nutrition focal points, need to understand the policy cycle and be capable of applying evidence to inform and advocate for policy. 53,68,69 Similar to some respondents in this study, Michaud-Létourneau et al. suggest that building capacity for evidence use may be more pertinent than capacity for evidence generation. 68 Oliver and Cairney, however, explain that the complexity of the tasks associated with applying evidence to influence policy may extend beyond capacity that can be easily built; the emotional, practical, and cognitive efforts required to advocate for policy change can be substantial. 58 It may be necessary to recognize the diverse skillsets or technical profiles needed for a successful NMT application and ensure there are resources to involve advocacy, communications, and policy experts, as well as analysts. It is also important to mention that focusing on the capacity needs of individuals risks overlooking the broader institutional structures required to generate and connect evidence to policy. 61 As such, efforts to build or seek capacity for evidence use and advocacy should consider the rules, resources, and principles through which institutions, such as government and NGO partners, operate. 61 The perceived complexity of the NMT methods, the tools themselves, or their outputs limited results uptake in some settings. Roberton et al. discuss the trade-offs between the accuracy, and hence perceived complexity, of modeling methods and the usability of the LiST health modeling tool and emphasize the importance of analyst confidence in using health modeling tools and their outputs. 10 In addition to training and support to use the tools, as discussed further below, the usability of NMT software was raised by some respondents in this study. Anecdotally, tool developers have shared that efforts to make the analysis more accessible to a wider group of less technical users through simplified graphical user interfaces, prepopulated inputs, and time-saving input and export functions were appreciated. However, as in other studies examining facilitators and barriers to evidence use in policy, respondents put more emphasis on the need to provide high-quality, simple, and relevant communication of NMT methods and findings to nonanalyst consumers for the modeling results to reach and be used for decision making, rather than a need to simplify or improve the tools themselves. 59 Meaningful policy impact is often the result of trusting relationships, which can take time and effort to create and maintain, and cannot be rushed or produced during ad hoc workshops. 11 This has implications for the nature and duration of collaborative relationships and necessary project budgets to maintain these. 70 Some respondents reported positive experiences in coaching local partners, who were known and trusted by wider stakeholders, to present the modeling themselves, reducing the dependence on international analysts for the interpretation and ongoing use of the modeling.
Competing priorities and insufficient institutional capacity for analysis have been listed as barriers to the local use of other health modeling tools or their results. 10 translation, and application of evidence are identified early, and that analyses only proceed if these needs can be met. 53,68,69,71 This is particularly important given the emphasis respondents in this study placed on involving national institutions and local partners in NMT applications and fostering government ownership. Furthermore, even though definitions of ownership are also ambiguous in the development and public health policy literature, there is general agreement that involving government partners and other stakeholders meaningfully in analysis processes encourages the uptake of evidence for policy decisions. [72][73][74][75][76] To provide greater clarity and avoid disappointment, Marjanovic et al. recommend that stakeholders be invited to jointly define ownership at the beginning of evidence generation processes and that this definition guides the process of working together, as well as sourcing and distributing resources. 77 It is worth noting that the MAPS tool has employed a "codesign" process consisting of in-person and virtual workshops in multiple countries. 78 This is a collaborative process that involves stakeholders and end users throughout the tool's development to ensure that their needs are met and that the final product is useful, valuable, and usable, in addition to promoting ownership. 78 The existence of a suite of NMTs that can inform policy decisions at different levels of maturity and specificity, for a range of institutions, presents a great resource for LMIC nutrition workforces as well as partners across health, education, social protection, agriculture, and other sectors. In spite of this, and as previously recognized, NMT applications were seldom initiated or led by national governments and were applied in response to government requests in some cases only. 2,10 The perceptions, values, and interests of nongovernmental stakeholders can influence what is considered trustworthy or quality evidence and which policy or program options are deemed appropriate, feasible, or legitimate. 14,61 In some of the featured case studies, NMTs were applied among competing ideas or programming options. We can assume that NMT analyses conducted by particular implementing stakeholders or funded by donors with interests in steering government policy discussions in a particular direction may be less impartial than analyses led by governments themselves. 73,79 Therefore, government ownership and leadership of evidence gener-ation, such as NMT application, should be encouraged to avoid the privileging of external objectives over national interests, in addition to reported benefits for capacity building and evidence uptake. 73,79 The NMC aims to disseminate information to help end users in LMICs understand tool capabilities and their relevance to specific policy processes. 2 There is an opportunity to address potential power, resource, and expertise imbalances and to change to demand-driven, country-led models in how NMTs are introduced and applied. 80 This would require financial and technical support for countries to define their own policy questions and be able to consider the tools available to answer them. 2,9 Unlike most evidence generation discussed in policy literature, many of the NMTs could be applied with or by decision makers in LMICs on an ongoing basis, pending new policy questions or data. This appears to align with the long-term approach through partnerships with relevant actors to understand the policymaking process recommended by Cairney et al., over "one-off disseminations of state-of-the-art knowledge." 11 Similarly, Pelletier et al. highlight that the evidence base to support nutrition advocacy should be developed in an "ongoing manner," rather than only at the beginning of such a process. 70 While many brokers mentioned the importance of long-term partnerships with decision makers, one-off NMT analyses with limited follow-up were common. In some cases, this was due to the relatively short-term need for the information some NMTs are designed to generate, while in others, there was a lack of medium-or long-term funding to support the ongoing, collaborative relationships desired by all parties. For NMTs designed to be applied continuously or often, capacity could be institutionalized so that decision makers are supported when needs arise. Tool institutionalization was rare in the case studies, for reasons noted above, yet there were promising examples, such as where NMTs were taught at local universities, leading to demanddriven, locally managed analyses and a shift in roles for international organizations from "the generation of knowledge to agents of action." 81 In some other cases, tool applications were hosted or supported by local government research bodies or international organizations with an in-country presence. Such organizations were able to apply NMT results to advocacy and decision making continuously. Inves- tigation is needed regarding which tools or policy processes require long-term, in-country capacity or hosting, and, for those selected, what resources and circumstances are needed for successful institutionalization and the long-term sustainability of this.
An alternative would be to make tools as accessible as possible remotely and encourage independent use. However, in their current state, few NMTs can be independently applied without intensive training. While new, preloaded, online tools allow instant access to analysis outputs, they are only designed to answer single analysis questions and are limited in their ability to consider custom data or variations to policy questions. 22 The ability for self-teaching via web-based resources was not mentioned in the analysis, yet is available, to some extent, for a few of the featured tools. Tool developers could support users to run their own analyses by developing selfguided training materials, videos, or webinars. 10 Similarly, tools able to be applied without first needing resources for trainings or for hiring consultants to do the analysis or primary data collection would be more accessible to independent users. The feasibility and suitability of independent access and online support for the use of NMTs requires discussion, as does the support needed for end users to identify the appropriate tool(s) for a given policy question.
To our knowledge, this qualitative study is the first to document multiple NMT applications and factors determining their influence on policy or program decision making. As such, the findings provide considerations for planning and conducting NMT analyses, as well as tool development. However, these findings should be considered in light of some limitations. Only case studies that NMT developers knew about were eligible for selection, meaning independent applications of tools available in the public domain may be underrepresented. While quota sampling was used to ensure different end-user groups were represented, respondents were identified via snowball sampling, which may have limited the range and experience of stakeholders interviewed. There were more case studies from Africa (19) than Asia (7) or Latin America (4), and for three tools, only one case study was identified, so our results may not adequately reflect the experiences of all tools or regions. Furthermore, there was an uneven geographical distribution of enduser types across regions, based on the participants identified during the snowball sampling. This could mean that responses are somewhat skewed toward areas of higher participation per category, such as Southern Africa and Southeast Asia for consumers and Latin America and Southern Africa for brokers.
Only one researcher (F.K.) conducted the interviews and analysis for this study. While she conducted the interviews with a structured approach, it is possible that some insights were missed or overstated. Moreover, as the researcher used RTA, the development of themes was influenced by her background and knowledge of the subject and may have differed if the analysis were done by others. 42 Lastly, individual tools and case studies were not identified in the analysis to avoid comparison of tool methods or implementing organizations and to preserve anonymity, as the focus of the paper was on the process of applying NMTs rather than the tools themselves. This limited the ability to describe key tool elements in detail and any relevance they had for policy influence.
We found that NMTs have had wide-ranging influences across policy and program cycles and contexts, contributing to an enabling environment for nutrition, but that the extent of these is highly dependent on the tool application process, accompanying advocacy, and the expertise, capabilities, and resources of local teams. Understanding how NMTs have been applied and what was achieved as a result could assist potential end users to appreciate the policy questions these tools can address, as well as make a case for allocating resources for such analyses by demonstrating their policy value. We also found that potential remains for even stronger policy influence. The factors explored in this study can inform how NMTs are applied in the future, including planning the analysis process, materials to support NMT applications, modifications to existing tools or new tool development, and resourcing for local collaborator participation and evidence translation in all of these activities.
importantly, we would like to thank the interview participants who generously gave their time to take part in this study. Support to M.W. was from the United States Agency for International Development (USAID) through its flagship multisectoral nutrition project USAID Advancing Nutrition, under the terms of contract 7200AA18C00070 awarded to JSI Research & Training Institute, Inc. Additional support to M.W. and K.S. was from USAID under the terms of Cooperative Agreement No. AID-OAA-A-12-00005 through the Food and Nutrition Technical Assistance III Project (FANTA), managed by FHI 360. The contents of this article do not necessarily reflect the views of USAID or the U.S. Government. [Correction added on May 04, after first online publication: The acknowledgments section was updated.]