Evaluating agricultural weather and climate services in Africa: Evidence, methods, and a learning agenda

Weather and climate services (WCS) are expected to improve the capacity of Africa's agricultural sector to manage the risks of climate variability and change. Despite this, a lack of evidence prevents a realistic analysis of whether such services are delivering on their potential. This paper reviews 66 studies that have evaluated outcomes and/or impacts of agricultural WCS in Africa, highlighting areas that have received relatively more attention as well as persistent gaps. While the evaluation of WCS outcomes is relatively straightforward, estimates of the number of people who access and use these services are uneven (covering a small number of communities in 23 of 54 African countries) and highly variable (with access estimates ranging from ~2 to 86%, depending on the service and the population). Meanwhile, 22 documents estimate the impact of WCS with respect to yields and/or income. Developed with a variety of methods, these estimates are also wide ranging and illustrate how impact is conditioned on a number of characteristics of the service, the user, and the context in which both operate. The paper uses lessons developed through this review to develop a “learning agenda,” or evidence‐building roadmap, to establish priorities that can guide work to improve the design, delivery, and impact of agricultural WCS in Africa. Priority learning areas include activities that can strengthen the evidence of access, use, and impacts of WCS, along with those that can advance the use and usability of evidence so as to improve the design and targeting of WCS services.


| INTRODUCTION
Agriculture plays a critical role in most African economies and among the livelihood strategies of a majority of Africans (Collier & Dercon, 2014). Yet African agriculture is marked by low productivity, low levels of investment, and high levels of weather and climate-related risk (Sonwa et al., 2016). Weather and climate services (WCS), which involve the production, translation, transfer, and use of scientific information for decision-making, are expected to improve the capacity of Africa's agricultural actors to manage the risks associated with climate variability and change-and, in so doing, to transform investment in this important sector. As such, agricultural WCS stand to play a critical role in Africa's efforts to meet the Sustainable Development Goals, including those that pertain to poverty (SDG1), hunger (SDG2), health (SDG3), gender equality (SDG5), responsible consumption and production (SDG12), and climate action (SDG13) (Campbell et al., 2014).
Given the degree to which weather and climate shape agricultural production, a great deal of research has explored the relationship between weather, climate, and African agriculture (Calzadilla, Zhu, Rehdanz, Tol, & Ringler, 2013;Knox, Daccache, & Wheeler, 2012;Kotir, 2011;Schlenker & Lobell, 2010;Ziervogel et al., 2014). Research has also focused on: the kinds of weather and climate information that can inform agricultural decision making (Hansen, Mason, Sun, & Tall, 2011;Sheffield et al., 2014;Stern & Cooper, 2017); the quality of existing weather and climate information Landman, DeWitt, Lee, Beraki, & Lotter, 2012); and the ways in which African farmers can use such information to improve their livelihoods (Cooper et al., 2008;O'Brien, 2000). Despite this, evidence regarding the effectiveness of agricultural WCS in Africa lags these other fields.
This evidence gap can be traced in part to the unique history of the WCS communities, originally dominated by scientists with more experience evaluating the quality of weather and climate information than in understanding the impact of its use (McNie, 2013). Even as the number and diversity of actors interested in this information has grown-including through the Global Framework for Climate Services, which was formally implemented by the World Meteorological Organization in 2012-a number of challenges complicate the evaluation of WCS, thwarting the efforts of skilled evaluators and tempting information providers to defer evaluation, or to rely on more easily tracked but less meaningful metrics including web traffic, workshop participants, and peer-reviewed papers (Meadow et al., 2015) (Box 1).
This lack of objective evidence has prevented the community from developing a more complete understanding of the role that climate services can and do play in African agricultural development. How and to what extent do farmers access and use weather and climate information to inform their decisions? What impact do WCS have on farmers' livelihoods and on agricultural development goals? How do particular aspects of the design and implementation of WCS influence their effectiveness? Were they available, convincing answers to these questions could be used to improve the implementation of existing services; support adequate investment of public funds in National Meteorological Services; inform the appropriate balance of investment in the production, translation, transfer and use of climate information (World Meteorological Organization, World Bank, & United States Agency for International Development, 2015); and shape the role that WCS play in national adaptation and climate finance plans, among other things.
To guide efforts to generate the evidence that can meet these needs, our paper develops a "learning agenda" for the evaluation of agricultural WCS in Africa. We begin by defining terms and describing the state of agricultural WCS in Africa (Section 2). Section 3 details the methodology used to guide our review. Section 4 reports on (Collier & Dercon, 2014) that state of evidence regarding the access, use, and impact of WCS in Africa's agricultural sector and (Sonwa et al., 2016) the methodological approaches that have been used to develop this evidence. After considering these threads separately, the paper brings them together in Section 5, where evidence and methodological gaps are synthesized, before developing a "learning agenda" in Section 6. The learning agenda is designed as a roadmap to prioritize areas where additional work holds the most potential to advance our understanding of how WCS can and do contribute to improved agricultural outcomes in Africa.

| Agricultural WCS in Africa
A number of factors have contributed to a growing interest in the development and use of weather and climate information for agricultural decision making in Africa. In 1994, a study by Cane, Eshel, and Buckland (1994) showed that maize yields in Zimbabwe were more strongly correlated with Pacific sea surface temperatures, a feature of the El Niño Southern Oscillation, than with seasonal total rainfall in Zimbabwe itself. Three years later, the public visibility of a strong El Niño event (1997/1998) prompted a surge of field research on the potential use and value of seasonal forecasting for Africa, with a particular focus on smallholder agriculture. Coincidentally, Regional Climate Outlook Forums (RCOFs) were initiated in Southern, East and West Africa the same year, though planning had been initiated before the El Niño event was anticipated (Gerlak et al., 2018).
As interest in WCS has grown, so too has the capacity of African actors to provide them. Most national meteorological services in Africa now provide seasonal rainfall forecasts, monthly climate outlooks, agrometeorological bulletins, and extreme weather alerts (Kadi, Njau, Mwikya, & Kamga, 2011a, 2011b; 15 African countries regularly share agrometeorological bulletins through the World Agrometeorological Information Service (World Agrometeorological Information Service, 2019). Of the 17 sub-Saharan meteorological services that responded to an email survey in 2010, most reported adding value to RCOF forecasts by including additional information (e.g., start and duration of the rainfall season, rainfall frequency or distribution), though only two reported making historical observations freely available. Most also reported proactively serving farmers and other agricultural stakeholders, with the nature of the partnerships and the communication channels used to do so varying by country (Hansen et al., 2011).
African WCS have also benefited from support of the Global Framework for Climate Services, an outcome of the third World Climate Conference. For instance, support from the GFCS prioritized African efforts to develop national climate service policy frameworks and action plans; this has included Benin (Meteo-Benin, 2017), Chad (Ministere du Developpement aeronautique et de la Meteorologie Nationale, 2016), Cote d'Ivoire (Ministre des Transportes, Direction de la Meteorologie

BOX 1 TERMINOLOGY
While terms can be used differently across disciplines, this paper uses the following definitions (adapted from World Meteorological Organization, World Bank, & United States Agency for International Development (2015)): Evaluation is a structured process designed to generate information about a program, with the purpose of informing judgments about and/or improving the program.
Evaluations can be ex ante or ex post: Ex ante evaluation (often termed appraisal) analyzes the anticipated impacts of the program, using models and/or stated preference methods to forecast value and/or potential impact. In full ex ante appraisal, this can be extended to consider the possible options and select the preferred design.
Ex post evaluation is conducted after a project is complete, documenting the lasting benefits associated with actual use Evaluations can focus on outputs, outcomes, and/or impacts: Output: the result of a project (i.e., the existence of information tools, products, or services) Outcome: the degree to which an intervention leads to a change in knowledge or behavior (i.e., reported changes in access to or use of climate services) Impact: the degree to which outcomes lead to changes in welfare (i.e., increased agricultural outputs, increased efficiency, reduced losses, etc.) A variety of methods can be used to collect and analyze evaluative evidence: Household surveys quantitatively and/or qualitatively sampling individual units of a population Facilitated focus groups allow evaluators to get a sense of a range of opinions and experiences Interviews provide an opportunity to gather in-depth evidence one-on-one with individuals Participatory processes structure evidence gathering through group activities Contingent valuation is a survey-based technique used to ascertain willingness to pay for services Descriptive statistics describe or summarize data regarding access, use and/or impact Inferential statistics allow for generalizations about the populations from which samples are drawn Benefit transfer methods transfer estimates developed in one context to another context, as a substitute for developing entirely new estimates WCS provide scientific information for decision making Weather services provide weather forecasts and warnings about hazardous conditions; they may include storm warnings, daily, 3-day, and 10-day (dekadal) forecasts, and so on.
Climate services involve the provision of climate information in a way that assists decision making by individuals or organizations; they may include seasonal outlooks, drought forecasts, agroclimatic bulletins, and so on. Nationale, 2016), Malawi (MetMalawi, 2014), Mali (Republique du Mali, 2016), Senegal (Republique du Senegal, 2016), and South Africa (Republic of South Africa, 2016), among other countries.
Within this context, a number of development organizations have come to see climate services as a practical way to advance climate adaptation and resilient development goals in Africa. Significant recent initiatives include: ClimDev-Africa (African Development Bank, Africa Union Commission, and UN Economic Commission for Africa); the World Bank's Africa Hydromet Program; DfID's Weather and Climate Information Services for Africa program; CGIAR research program on Climate Change, Agriculture and Food Security (CCAFS); and sizeable projects implemented with support from the Green Climate Fund and bilateral donors (DfID, NORAD, SIDA, USAID, German Federal Ministry of Education and Research).
While there is no comprehensive inventory of agricultural WCS in Africa, a 2017 regional coordination event organized by GFCS (World Meteorological Organization, 2017) was a step toward capturing and sharing knowledge.

| Challenges of WCS evaluation
The evaluation of WCS should not be confused with the evaluation of weather and climate information; while the latter assesses the quality of the information itself (e.g., data quality control, forecast verification), the former is primarily concerned with (Collier & Dercon, 2014) documenting the extent to which potential users are able to access and use services (Sonwa et al., 2016) estimating the actual or potential impact and/or value services; and (Campbell et al., 2014) identifying those elements of design and implementation that lead to better outcomes with respect to Collier and Dercon (2014) and Sonwa et al. (2016).
Several characteristics of WCS impose challenges to evaluation (Tall, Coulibaly, & Diop, 2018). First, the nonrival, nonexclusionary nature of WCS means that information can easily be passed along social and family networks. At the same time, however, the information transferred through informal networks may be incomplete or distorted. This makes it difficult to distinguish between those who receive the service and those who do not, complicating efforts to identify a control sample that does not have access to the information, as required for a randomized control trial.
Second, because of the stochastic nature of the climate, the use, impact and even the mechanism of impact, can vary considerably from year to year. The number of years required to sample the range of variability, and hence provide reliable estimates of use and impact, can be expected to exceed a typical project cycle. Furthermore, climate conditions during project baseline and end-line surveys may confound cumulative indicators of impact, making it difficult to distinguish between benefits of the service, and the influence of climatic conditions in the baseline and evaluation years.
Third, the impact of climate information comes through changes in management decisions, which are also influenced by other agricultural development interventions, and by farmers' varying goals, skills and constraints. Information has no intrinsic economic value. The fact that weather and climate information is one of many interacting factors that influences decisions and determines livelihood impacts makes it difficult to isolate the relative contribution of WCS. It also means that causal pathways between access to climate information and livelihood impact can vary among farmers. While not unique to WCS, this is a particular concern when the evaluation calls for input from farmers who may not be comfortable discussing the nature of their productivity or economic performance. Even in cases in which farmers are willing to report, they may be unable to correctly estimate harvests or production costs; they might also prefer to over-or underestimate harvests, if they perceive that those estimates might lead to some potential gain (e.g., food aid).

| METHODS
This paper reviewed documents (the vast majority of which were in English, though some were in French; e.g., Ngana, Maina Ababa, Gapia, & Kossi, 2013) that provide evidence of access, use, and impact of agricultural WCS in Africa. Documents were identified with online searches using Web of Science and Google Scholar (see Table 1), and by requests made within the authors' professional networks, including as part of two sessions at the fifth International Conference on Climate Services, held in Cape Town, South Africa in 2017, and at the Adaptation Futures Conference held in Cape Town in 2018.
Papers were included in the study if they generated qualitative or quantitative evidence that addressed one or more of the following criteria: 1. documenting the extent to which potential users are able to access and use services; 2. estimating the actual or potential impact and/or value of services; and 3. identifying those elements of design and implementation that lead to better outcomes with respect to (1) and (2) To ensure the review was focused on issues of access, use and impact, studies that verified climate and/or production forecasts (Bezuidenhout & Schulze, 2006) without exploring the impact of actual or potential use of those forecasts, were not included. Studies that explored potential users and uses of information, whether qualitatively or quantitatively, were also not included, though they are considered in depth in a separate review paper (Carr, Goble, Rosko, Vaughan, & Hansen, 2017). Studies that focused only on the use of traditional, rather than scientific, weather or climate indicators were also not included.
Both peer-reviewed and gray-literature studies were included regardless of whether documents were originally intended as program evaluations or not.
The evaluations described in each paper were analyzed with respect to region; country; number of individuals surveyed; type of weather or climate information provided; estimates of access; estimates and descriptions of use; estimates of impact; implementation and design factors identified or surveyed; and evaluation methods. This analysis allowed the authors to describe (Collier & Dercon, 2014) the current state of evidence and (Sonwa et al., 2016) the methods used to generate that evidence, detailed and interpreted below.

| RESULTS
The review resulted in 66 studies, which were conducted in 23 African countries over a span of 40 years. Studies were concentrated in West (27/66), East (22/66), and Southern (22/66) Africa (note some studies include more than one location); the review found just one study in Central Africa and one in North Africa that met our criteria. Even in regions where evaluations were relatively more common, certain countries (e.g., Burkina Faso, Kenya, Tanzania, Ethiopia, Zimbabwe) turned up frequently, while other countries (e.g., Angola, Eritrea, Cote d'Ivoire, Gambia) were not represented at all. The vast majority of studies evaluate services that were built on forecast information at weather, subseasonal, or seasonal climate time scales, or were ambiguous with regards to the kind of information that was provided. Roughly 80% of the documents included in the study were published in scientific journals; only one was published as a book chapter, and 15 were project outputs or case studies. All documents are available online. Information regarding the geographical scope of these studies is presented in Figure 1. A full list of documents is found in Appendix A.
This section first presents the evidence base generated by these studies before considering the methodologies used to generate that evidence.

| Evidence
Evaluation efforts have generated evidence regarding the access, use, impact, and design of climate services; while evidence remains somewhat limited, particularly in certain regions, the evidence that does exist seems to indicate that WCS are more accessible in certain regions than in others; that when available, services are used more frequently by farmers than pastoralists; and that the impacts associated with WCS depend on a number of factors related to design, targeting, and implementation.  Note. Many papers turned up in more than one search.

| Variable access to information
The evidence indicates that Africans' access to WCS varies based on region, livelihood strategy, demographic characteristics, and information type. In East Africa, for instance, studies in Kenya, Ethiopia, Tanzania, and Uganda estimate the number of certain populations that are able to access WCS in the range of 15-82%, with lower estimates for pastoralist versus farming communities and some indication that men are more able to access climate information than women (Daly, West, & Yanda, 2016;Lybbert, Barrett, McPeak, & Luseno, 2007;Ngugi, Mureithi, & Kamande, 2011;Oyekale, 2015). In Southern Africa, estimates in Malawi, Mozambique, Namibia and Zimbabwe range from 27 to 86%, with an indication that radio is the primary source by which farmers access weather and climate information (Coulibaly, Kundhlande, Tall, Kaur, & Hansen, 2015;Mudombi & Nhamo, 2014;Mulwa, Marenya, Bahadur, & Kassie, 2017;O'Brien, 2000;Zamasiya, Nyikahadzoi, & Mukamuri, 2017;Zuma-Netshiukhwi, Stigter, & Walker, 2013). Evidence also suggests that weather information is more accessible than seasonal forecasts in this region (Lazo, 2015).
There is some indication that WCS may be more accessible in East and Southern Africa than in West Africa, though evidence is mixed (Limantol, Keith, Azabre, & Lennartz, 2016;Oyekale, 2015;Zongo et al., 2016). One West African study comprising Mali, Burkina Faso, Niger, and Nigeria found that 70% of 566 surveyed households were aware of and able to access climate information (Tarhule & Lamb, 2003), though more recent studies have presented somewhat lower numbers. In Burkina Faso, for instance, Zongo et al. (2016) found 22% of sampled farmers (n = 629) had access to climate information, while Rasmussen, et al. (2014) found a minority of Burkinabe herders (n = 61) had access to weather forecasts (30%), flood information (6-13%), and seasonal climate forecasts (SCFs) (7%) (Rasmussen et al., 2014;Zongo et al., 2016). Oyekale (2015) sampled 701 farmers in five West African countries, finding that slightly more than half were able to access climate information.
While just one study considers access in Central Africa, it finds only 2% of farmers in the Central African Republic have access to any kind of meteorological information (Ngana et al., 2013). A summary of access estimates is found in Table 2. More detailed information regarding access estimates is found in Appendices B-D.

| Differing levels of use
Evidence regarding the use of agricultural WCS varies based on livelihood strategy, among other things (see Table 3). In fact, many studies find evidence that a majority of farmers use WCS when they are accessible. Averaging across six documents that provide sufficient information to calculate the rate of use of SCFs by farmers who report accessing such forecasts, the rate F I G U R E 1 The geographical scope of the WCS evaluation studies included in the review. WCS, weather and climate services of use is found to be 74% (Amegnaglo et al., 2017;Coulibaly, Kundhlande, et al., 2015;O'Brien, 2000;. Aggregating across the range of information products, management response, locations and social groups in the 14 studies that provide sufficient information to calculate aggregate use rates for any type of WCS found that the majority of farmers who access WCS use the information to adjust a range of management decisions, as illustrated in Appendix E. These studies document farmers' application of weather and climate information to a range of decisions, including those regarding the choice of fields, crops, and/or crop varieties; the timing of agricultural tasks; the application of inputs, and the negotiation of annual loans (Egeru, 2016;Lo & Dieng, 2015;Mudombi & Nhamo, 2014;Phillips, Deane, Unganai, & Chimeli, 2002;Rao, Hansen, Njiru, Githungo, & Oyoo, 2015;Tarhule & Lamb, 2003).
T A B L E 2 Regional summary of estimates of access to agricultural WCS found in the literature  33+ interviews (Broad & Agrawala, 2000) Working with Malian farmers, Carr et al. (2014) and Carr and Onzere (2017) found that farmers' use of climate information was dependent on the roles and responsibilities associated with particular identities, primarily seniority and gender. Contrary evidence was found in Malawi, where a roughly equal number of male and female farmers (n = 320) report having access to climate information, but not using it-relying instead on indigenous knowledge and personal experience, which they perceive as more reliable and more relevant to local decision making (Coulibaly, Kundhlande, et al., 2015). The current review found no evidence that farmers use climate information to make costly investments (e.g., irrigation or agroforestry), a finding that is supported by work in Ethiopia and South Africa (Bryan, Deressa, Gbetibouo, & Ringler, 2009).
While evidence indicates that farmers use WCS in a variety of circumstances, pastoralists' use of WCS appears more limited. In documenting the use of forecasts among herders of southern Ethiopia, for instance, a 2016 study found that most (96%, n = 200) rely on traditional, rather than scientific, forecast information (Egeru, 2016). An earlier study found that a majority of Kenyan and Ethiopian pastoralists (n = 323) did not use SCFs-a finding the authors attributed to the relatively large number of mid-season decisions that allowed herders to cope with unfavorable conditions as they developed, rather than in advance (Luseno, Mcpeak, & Barrett, 2002). This is corroborated by studies that show that fewer types of information are useful to pastoralists, who face a very different suite of decisions than farmers. A 2014 study found that Burkinabe pastoralists (n = 61) were inclined to make changes to herd management based on information regarding the availability of grazing (75%); onset date (6-53%); and rainfall during the first 2 weeks of the rainy season (17%)-though they did not adjust herding practices in response to seasonal forecasts (Rasmussen et al., 2014). In Uganda, pastoralists (n = 198) reported using information on onset and drought to make decisions regarding shifting livestock to new grazing areas; seeking military escorts to those areas; selling firewood and charcoal; and purchasing veterinary drugs (Egeru, 2016).
More detail regarding access and use estimates is found in Appendices B-E.

| Context-dependent impacts
Evidence regarding the impact and potential impact of agricultural WCS on yields and/or incomes is generally positivethough also relatively varied, depending on the context, climate, and crop, as well as the type and accuracy of the information in question (see Table 4). In Burkina Faso, for instance, evidence indicates that on-farm gains associated with the use of seasonal forecasts increased monthly cereal production (Zongo et al., 2016). Similarly, Ouédraogo et al. (2015) report that Burkinabé cowpea farmers (n = 170) with climate information showed higher yields than those without information (an average of 847 kg/ha compared to 685 kg/ha); information recipients were shown to have made decisions that resulted in savings in seed and pesticides, resulting in gross margin gains of 66% above the control group. The same study found that sesame farmers provided with climate information had slightly lower yields (550 kg/ha compared to 605 kg for the control group) and lower margins, as the information prompted an increased investment in fertilizer that did not lead to significant returns (Ouédraogo et al., 2015).
Also in West Africa, Roudier et al. (2014) modeled (ex ante) the introduction of seasonal and 10-day forecasts in Senegal: their modeling findings indicate farmers who used both types of information were likely to experience yield gains in roughly one-third of the cases; this study also estimated that impacts vary according to the nature of rainy season, the accuracy of the forecast, and the type of response.
Reporting on the results of a multi-year project in Southern Africa,  found that Zimbabwean farmers (n = 578) who used seasonal forecasts showed a small, insignificant difference in yields in the first year, and a larger and marginally significant difference in the second year.
In 2004, Thornton et al used a bio-economic modeling approach to show that commercial ranchers who consistently used seasonal forecasts over 25 years were likely to experience considerable benefits (~$85,000, adjusted to 2018 USD), though at the cost of increased variability in year-to-year income (Thornton et al., 2004). A modeling study of the potential impact of SCFs in Lesotho found that while the use of forecast information had the potential to improve outcomes for marginal households, forecasts with poor skill were more likely to be associated with negative impacts ; a companion study found that the timescale of adoption for seasonal forecasts is likely to be very long (Bharwani et al., 2005).
In East Africa, Anuga and Gordon looked at the relationship between agricultural outcomes and the employment of "climate resilient" strategies in Ethiopia; they found that receiving training in the use of weather information had the greatest influence on yield (17% increase) (Anuga & Gordon, 2016). Barrett and Ndegwa (2016), working in Kenya, found that farming households with access to local advisories and seasonal forecasts had consistently higher income levels. Also in Kenya, an ex ante modeling study conducted by Hansen et al. (2009) found that that seasonal forecasts based on a general circulation model led to gross margin increases of 9-24%, averaged across years, while perfect knowledge of daily weather was worth an estimated 24-69%. At a regional level, Rodrigues et al. (2016) modeled economy-wide impacts of national seasonal forecast systems in Kenya, Malawi, Mozambique, Tanzania, and Zambia. The study estimated that perfect information adopted by all farmers would generate regional GDP gains averaging $113 million USD per year ($3 USD per hectare) relative to a no forecast baseline. The study estimated benefits are higher for poorer households as they are more likely to be engaged in farming (Rodrigues et al., 2016).

| Perception of value
While the impact estimates described above are developed using surveys or modeled analysis, other studies solicit the "value" of WCS from potential users directly. These studies ask or elicit what users would be willing to pay for WCS information or services in the future, and thus the benefit farmers would expect to receive from the use of such services. Adjusted to 2017 rates (USD), these estimates have ranged from $1.19 (Zongo et al., 2016) to $15.36 (Ouédraogo et al., 2015) for improved seasonal forecasts (see Table 5).
Individual studies elaborate on this range: Ouédraogo et al. (2015) for instance, found that seasonal information ($15.36) is much more highly valued than decadal information ($3.55) or even contextualized agrometeorological advisories ($5.77) in Burkina Faso. Rao et al. (2015) found men and women valued different services (e.g., training, advisories, etc.) differently, and Zongo et al. (2016) showed variations in willingness to pay for seasonal information depending on agroecological zone. In addition to the willingness-to-pay estimates, Amegnaglo et al. (2017) used a ranking system to identify which information is perceived to be more valuable to farmers.

| Design and targeting
As earlier sections make clear, many studies have generated evidence regarding the degree that elements of design, implementation, and targeting affect access, use, and/or impact; evidence regarding several of these factors are discussed below and presented in Table 6.

User characteristics
A number of studies have focused on the role that user characteristics have played in conditioning the access, use, and impact of WCS. These studies have primarily focused on livelihood strategy and identity, including for instance, gender, education, and socioeconomic status (Carr & Owusu-Daaku, 2016;Gebrehiwot & van der Veen, 2013;Ngugi et al., 2011;Oyekale, 2015). Several studies have also explored understanding of climate forecasts, showing that users are able to recognize and adapt to the uncertainty implicit in such forecasts (Luseno et al., 2003;Lybbert et al., 2007).

Context
With respect to broader context, several authors have shown that information is more impactful under certain conditions (i.e., drier than normal, wetter-than-normal) and in certain agroecozones Makaudze, 2014). Relatively few studies have considered how broad issues related to supply, institutions, and/or context influence access and use of weather and climate information in African agriculture-though Ngugi et al. (2011) and  are notable exceptions, exploring the influence of the 1997-1998 El Niño on forecast use and the various factors that may motivate providers to develop conservative forecasts (i.e., for the middle tercile), respectively.
T A B L E 6 Frequency with which the relative effectiveness of implementation and/or design factors have been explored in the literature

| Evaluation methodologies
The evidence presented above was generated using a variety of methods; here we consider these methods more directly, paying attention to how current approaches have been able to deliver useful information regarding agricultural WCS and to the extent to which they are suited to address the unique challenges of WCS evaluation.

| Outcome evaluation
Studies regarding the access and use of WCS fall under the heading of outcome evaluation, since they take stock of changes in the behaviors and practices that result from an intervention. Table 7 summarizes the frequency of use of particular methods to evaluate access and use of WCS in Africa.
To date, most of what is known about access to WCS in Africa's agricultural sector has been gathered through household surveys and analyzed using descriptive statistics. This includes studies that use panel survey methods to collect data over several years and others that have sampled progressively (e.g., surveying the population before, during, and after the agricultural season) to document changes over time, or differences between farmers' expectations regarding access and reality once the season had begun (Egeru, 2016;Mudombi & Nhamo, 2014).
Surveys are well suited to capturing the number of individuals who access particular information products. Since access reflects both availability and demand for particular information products, its evaluation may seek evidence of either or both. Surveys can be expanded to answer questions such as: What is the reach of particular communication channels? What is the demand for a particular information product, or relative demand among different information products? What are the most effective communication channels? How do farmer characteristics (e.g., gender, age, farming system) influence access to particular information products and use of particular communication channels?
Studies that explore the use of WCS employ similar, though a wider variety of, methods to those that engage issues of access: Data on use is gathered through focus groups, workshops, interviews, and/or household surveys, and results are reported using statistics (Bryan et al., 2009(Bryan et al., , 2013Deressa et al., 2009;Klopper & Bartman, 2003) and/or qualitative methods  Patt & Dessai, 2005;Tarhule & Lamb, 2003;Zongo et al., 2016). This wider variety of methods reflects the fact that studies of use require the establishment of some kind of counterfactual, that is, what the decision would have been without the information. Indeed, eliciting how individuals use WCS depends on the ability of those individuals to attribute changes in particular management decisions to the information they contain. Since many conscious and sub-conscious factors can influence decisions, this may be a strong assumption. Obtaining management plans from individuals before and after they have been exposed to predictive information (e.g., seasonal forecasts) increases confidence by providing a reasonable counterfactual. The nonexcludability of information makes it difficult to compare management decisions between samples of farmers with and without access. Since management can vary as a function of forecast conditions-which are stochastic-many seasons may be required to provide a complete understanding of use of predictive information.

| Impact evaluation
Impact evaluations are designed to generate evidence regarding the ultimate impacts of an intervention, whether those impacts are direct or indirect, intended or unintended. While this type of evidence is critical for understanding the role that agricultural WCS can play in building the resilience of Africa's agricultural sector, there is far less evidence regarding impact than access and use: 17 out of the 66 documents reviewed for this study used this approach and many of these involve modeled estimates. While there is a broader range of methods used than in evaluations of access and use, each of these methods has significant limitations, and provides rather indirect evidence of the actual benefit of use of climate information.
Methods to evaluate the impact of WCS can be classified into two distinct categories: ex-post empirical studies of the benefits of WCS-informed decisions (i.e., investigating existing WCS); and ex-ante methods that model or estimate how potential uses of information could improve production, livelihoods or other impacts of interest. A summary of the frequency with which different methods have been used to evaluate the impact of agricultural WCS in Africa is found in Table 8.

Ex post studies
Eight studies have used ex post analysis to evaluate the impact of WCS on African agriculture (Anuga & Gordon, 2016;Lo & Dieng, 2015;Ouédraogo et al., 2015;Phillips et al., 2002;Stats4SD, 2017); those that have are generally based on household surveys and/or interviews, and focused on yields and/or marginal income (Anuga & Gordon, 2016;Ouédraogo et al., 2015). While these methods are time and resource intensive, they have been used in a variety of contexts, allowing for the development of a relatively large literature that explores the difficulties of survey-based ex post evaluation, including the attendant strengths and weaknesses of experimental and quasi-experimental design (Baker, 2000;Bamberger, Rugh, Church, & Fort, 2004).
Another method that has been used to generate evidence regarding the impact of WCS involves test plots. In this case, WCS are used to make decisions regarding a specific plot of land throughout the season, after which yields from the test plot are compared to those of plots where more traditional practices were employed (Lo & Dieng, 2015;. If well designed, test plots have the advantages of providing a counterfactual, capturing farmer decision-making, and potentially overcoming challenges of farmer recall and the elicitation of sensitive economic information. The test plot studies reviewed as part of this work were limited to the management of individual crops, and therefore missed potentially important resource allocation decisions made in response to WCS. They also focused on yield-but since many uses of WCS involve saving the costs of inputs, gross margin (i.e., market price of harvest minus costs of production, per unit area) may be a useful metric. It is important to note as well that test plots are only useful when they compare farmers' management based on WCS to farmers' management without WCS; trials that compare climate-based expert recommendations with farmers' normal practice, the difference between the experts' and the farmers' decision criteria confound the influence of WCS. Because of the stochasticity challenge, it is generally not feasible to run a test plot for enough years to provide stable estimates of the value of the information.

Ex ante studies
Twelve studies have attempted to characterize the potential impact of agricultural WCS in Africa using ex ante methods. These include appraisal studies (undertaken as part of design or in advance of WCS implementation) but also the use of ex ante methods, such as models, to estimate the potential impact of existing WCS. These studies have employed a range of approaches-including experimental economics, models, and surveys-as a means to estimate, rather than analyze ex post, the possible benefits of planned or existing services.  (2015), Rao et al. (2015), and Zongo et al. (2016) Sometimes described as participatory methods or "serious games," experimental economics approaches provide farmers with an opportunity to simulate how they might use WCS, were they to be provided, and to report their perception of potential results. While recent work has shown that these methods can be very useful in helping to estimate the impact of WCS, several challenges have been noted: In order for these methods to work, for instance, they must be run by skill facilitators who can ensure that all participants feel they can share and explore-and issues of power, gender, and hierarchy may distort the results if facilitators are not aware. In addition, participatory methods are time consuming; they may lead to fatigue among participants, and potentially expose them to uncomfortable or challenging situations Suarez, Mendler de Suarez, Koelle, & Boykoff, 2014).
Several modeling approaches have also been used to estimate the potential value of agricultural WCS in Africa. For instance, crop simulation and economic models have been useful in helping evaluators to estimate the value of WCS. Biophysical models may be used on their own to estimate yield changes, or linked directly with economic models (e.g., bioeconomic models) so as to characterize the impacts of different decisions informed by WCS. Simple field-and farm-level bio-economic models have been used to estimate potential impacts of weather and climate information in Kenya, Niger, and Senegal; these models have also allowed researchers to explore the range of possible outcomes associated with different contexts and/or the design and implementation of the services themselves (Hansen et al., 2009;Roudier et al., 2011;Sultan et al., 2010;Thornton et al., 2004).
Agent-based models have also been used to reveal how agents interact with other agents and the environment-allowing for the exploration of the impact of WCS in complex systems, as well as the how interactions producing emergent effects may differ from effects of individual agents. These analyses have helped to characterize which types of actors are likely to be impacted, and in what capacity, by the use of climate information (Bharwani et al., 2005;Ziervogel et al., 2005). A limitation of farm-level and agent-based modeling approaches is that they ignore the potential impact that the collective action of many farmers can have on market prices or on other sectors of the economy. One study in Africa undertook such an economic analysis, using computable general equilibrium modeling to estimate the potential economy-wide impacts of largescale farmer adoption of SCFs across several countries (Rodrigues et al., 2016).
Where long-term records are available, bioeconomic modeling approaches can sample many years of weather observations and climate information, and can provide insight that may address the stochasticity challenge. The main weakness of modelbased impact estimates, however, is that their realism can be limited by the ability of the models to capture farmer decisions and resulting economic impacts, particularly when weakly grounded in empirical data. In most instances, globally and in Africa, these modeling approaches have been applied in a stylized manner to hypothetical uses of information; realistic evaluation was often not the objective (Meza, Hansen, & Osgood, 2008). However, this is not an intrinsic limitation of the approach-and though bio-economic modeling tools tend to have demanding data requirements, it is feasible for analyses to be well grounded in empirical data, as is often the case for ex-ante evaluation of agricultural production technologies.
Finally, contingent valuation is a survey-based econometric technique that elicits the amount that potential users would be willing to pay for specific services, and thus implicitly their benefit (to these users). The contingent valuation, or "willingnessto-pay," approach, estimates the maximum price that a user will pay for the information or service directly, whereas the other ex ante approaches estimate value through the expected economic benefits of the use of the new information. Nonetheless, all these techniques seek to derive the (economic) value of information (Hilton, 1981).
Contingent valuation assumes the potential user can adequately estimate the expected economic gain, averaged across years, from using the new information; as such, it avoids the need for an explicit counterfactual. It is simpler for the analyst, but has significant limitations, especially for new or planned information products. Farmers cannot assess, and typically underestimate, the value of information that they do not have a lot of experience with. For a stylized seasonal forecast system, Pope et al. (2017) estimated that users would have to experience at least 30 forecasts in order to provide a realistic estimate of the value of the information (Pope, Buontempo, & Economou, 2017). However, training or accelerated experience through participatory processes can help to reduce this bias.

| Strengthening evidence of access and use
Evidence regarding the access and use of agricultural WCS in Africa is more plentiful and more homogenous than evidence regarding the impact of those services, making it relatively more straightforward to compare estimates of access and use across regions and/or populations. Our analysis has pointed to several evidence key gaps, however.
It is clear, for instance, that estimates regarding access to weather and climate information vary considerably-and while there has been some effort to understand the factors that cause this disparity, a more complete mapping of who has access to what kind of information, as well as the factors that enable or constrain access, would help to inform the investment and design of WCS. As Table 2 makes clear, evidence regarding access to agricultural WCS is particularly lacking in Central and North Africa-though no region boasts a preponderance of evidence, particularly given the diversity of actors that contribute to Africa's agricultural sector.
The review also highlights the uneven nature of evidence regarding the use of agricultural WCS. As seen in Figure 1, our analysis includes data from just 23 of Africa's 54 countries-and even within these countries, certain populations are relatively more studied than others (e.g., farmers vs. pastoralists). Moreover, with a few exceptions (Daly et al., 2016;Glantz, 1977;Hulme et al., 1992), the literature rarely explores the extent to which WCS are used by government and nongovernment agencies-and it makes almost no mention of how/whether private organizations (e.g., input suppliers, wholesalers, etc.) use such information.
With regards to methodology, two gaps stand out as well. The first is linked to the fact that many of the studies included in our analysis were performed without baseline analysis, making it difficult to identify changes in access and use over time, or as a result of a particular intervention. Fortunately, the kind of baseline survey that can help to build this sort of evidence is becoming more common in Africa and elsewhere (Coulibaly, Birachi, Kagabo, & Mutua, 2017;Coulibaly, Kundhlande, et al., 2015;Coulibaly, Mango, et al., 2015;Daly et al., 2016) and will likely improve the evaluation of agricultural WCS in the future. Methods to attribute changes in access and use of climate services to changes in the service and/or context will also need to be improved.
Finally, while the literature has begun to explore the barriers and enablers to the use of climate information, new qualitative and quantitative approaches are needed to continue to flesh out the role that these factors play in influencing access and use. This includes expanding consideration of the role of communication strategies (e.g., messaging, visualization, etc.) and dissemination mechanisms (e.g., radio, SMS, etc., see Jost et al. 2016), and exploring the study of new factors. For instance, the business models that sustain WCS have rarely been evaluated, despite the important role that they play in conditioning access, use, and impact on both short and long timeframes .

| Strengthening evidence of the impact of WCS
The landscape of evidence regarding the impact of WCS is both scarcer and more complicated than that regarding access and use, displaying a wide array of ex post and ex ante methodologies that rely on a range of different assumptions to help characterize the contribution of the services in question. These methodologies respond to the various challenges associated with the evaluation of WCS, though they also generate evidence that is somewhat more indirect than the evidence regarding access and use.
Several evidence gaps call particular attention-especially regarding the extent to which different groups benefit differently from WCS. It is also important to note that the studies included in this review have defined impact almost entirely with regards to yields and/or incomes, despite the fact that WCS can be expected to have a host of impacts (e.g., improved decision processes, time saved, more efficient allocation of resources, changes in social organization, etc.), and that these impacts can be expected to accrue at different levels. To date, no studies have explored the impact of agricultural WCS on African societies (e.g., does the promotion of scientific information isolate young, educated cohorts from traditional knowledge?) or the environment (e.g., do WCS facilitate more intensive farming practices? Could they lead to increased greenhouse gas emissions?). Exploring this broader range of impacts will be important to strengthening the evidence base.
Improving the evaluation of the impact of WCS will also require more attention to generating evidence in the face of a stochastic climate. Given the interaction between impact and stochastic climate variability, future studies should develop a better sense of the relationship between confidence/stability of impact estimates and the number of years sampled. It will also be important to explore the available options for capturing variability and increasing confidence during limited number of years of any particular project.
In improving methodology, the key issue will be to develop approaches that allow evaluators to compare the consistency of methods. This will require applying different methods in the same contexts in order to help estimate biases and identify methods that may be better suited to particular cases. This approach can also expose weaknesses in methods-allowing methods to be improved and better tailored to the climate service context or helping inform recommendations regarding the triangulation of methods in certain cases.
Combining methods provides promising opportunities to triangulate estimates of the impact of WCS, take advantage of complementary strengths, and overcome some of the most problematic limitations. For example, if bioeconomic modeling were combined with participatory approaches and/or survey-based data, the empirical results could be used to validate and, if needed, drive modeled farmer management response to climate information, while the models could then sample more of the range of variability of observed weather and climate information (Jochec, Mjelde, Lee, & Conner, 2001). A related issue will be to validate models to make it easier to assess the robustness of their results.

| Improving usability of evaluative evidence of WCS
Improving evidence regarding access, use, and impact is essential toward improving our understanding of the relative contribution of WCS. Another important component here is to improve the usability of such information, so that national governments, development agencies, program managers, and private companies can make informed decisions regarding their own climate service investments. Improving the usability of this information will require focusing on three key issues.
First, it will be important to expand evidence regarding a wider variety of users and uses. For instance, there is more evidence regarding access and use by farmers than by pastoralists, much more for small-scale farmers than commercial farmers, and for staple rather than cash crops. There is very little information regarding government and/or nonprofit organizations and none regarding the private sector. Addressing these gaps should be a priority, especially with regards to disaggregating populations by the identities that shape their roles and responsibilities and exploring access and use for a wider range of actors (e.g., input suppliers, insurance companies, wholesalers, extension services, etc.) (see for instance Table 3).
It is also important for evaluators to take into account the fact that many WCS in Africa's agriculture sector are operating suboptimally; evidence of the relative contribution of these services limits the understanding of what constitutes good practice. There is currently no guidance on how this existing information could be used to improve outcomes and impacts; developing this guidance will help advance the field. Given the variability in the design of climate information that has been evaluated, future work can help identify how the quality of services can best be factored into the body of evaluation results.
This will help to improve knowledge of how to generate and use impact information to improve the design and/or implementation of agricultural WCS. Documenting how services are able to deliver outcomes and impacts, and good practice, will be critical to improving design and delivery, providing insights. With the exception of a 2015 study in Kenya (Rao et al., 2015), there is little robust analysis regarding how different strategies (e.g., user engagement, coproduction, dissemination, capacity building, use) lead to different outcomes or impacts. Developing a broader suite of methods to generate this kind of information-and use this in design-has huge potential to improve future WCS.
A final issue will be to standardize the evidence generated by evaluations of WCS so as to allow for comparison of interventions. This includes expanding the use of economic approaches to appraise (ex ante) the socioeconomic benefits of WCS, to help better design new services, to justify prior and existing investments, and to help develop the case to continue (or increase) allocation of resources for WCS into the future. This will also require additional methodological development for costs as well as benefits (Lazo, Raucher, Teisberg, Wagner, & Weiher, 2009;World Meteorological Organization et al., 2015). Ensuring that methods capture the full range of costs and benefits require new techniques and will require guidance and support, but further development in this area has a key role to play in the scale-up of WCS in Africa.

| CONCLUSION: A LEARNING AGENDA FOR THE EVALUATION OF AGRICULTURAL WCS IN AFRICA
This review has characterized the state of evidence and methods regarding the evaluation of agricultural WCS in Africa. It has identified where our knowledge of access, use and impacts of WCS is relatively strong, but also where there are persistent gaps. Building on this analysis, we have developed a "learning agenda," or evidence-building roadmap, to establish priorities that can guide future work to generate evidence that can improve the design, delivery, and impact of agricultural WCS in Africa. Priority learning areas include activities that can strengthen the evidence of access, use, and impacts of WCS, along with those that can advance the use and usability of evidence so as to improve the design and targeting of WCS services; they are summarized below: 1. Strengthening evidence regarding the access and use of WCS