Using big data for evaluating development outcomes: A systematic map

Abstract Background Policy makers need access to reliable data to monitor and evaluate the progress of development outcomes and targets such as sustainable development outcomes (SDGs). However, significant data and evidence gaps remain. Lack of resources, limited capacity within governments and logistical difficulties in collecting data are some of the reasons for the data gaps. Big data—that is digitally generated, passively produced and automatically collected—offers a great potential for answering some of the data needs. Satellite and sensors, mobile phone call detail records, online transactions and search data, and social media are some of the examples of big data. Integrating big data with the traditional household surveys and administrative data can complement data availability, quality, granularity, accuracy and frequency, and help measure development outcomes temporally and spatially in a number of new ways.The study maps different sources of big data onto development outcomes (based on SDGs) to identify current evidence base, use and the gaps. The map provides a visual overview of existing and ongoing studies. This study also discusses the risks, biases and ethical challenges in using big data for measuring and evaluating development outcomes. The study is a valuable resource for evaluators, researchers, funders, policymakers and practitioners in their effort to contributing to evidence informed policy making and in achieving the SDGs. Objectives Identify and appraise rigorous impact evaluations (IEs), systematic reviews and the studies that have innovatively used big data to measure any development outcomes with special reference to difficult contexts Search Methods A number of general and specialised data bases and reporsitories of organisations were searched using keywords related to big data by an information specialist. Selection Criteria The studies were selected on basis of whether they used big data sources to measure or evaluate development outcomes. Data Collection and Analysis Data collection was conducted using a data extraction tool and all extracted data was entered into excel and then analysed using Stata. The data analysis involved looking at trends and descriptive statistics only. Main Results The search yielded over 17,000 records, which we then screened down to 437 studies which became the foundation of our systematic map. We found that overall, there is a sizable and rapidly growing number of measurement studies using big data but a much smaller number of IEs. We also see that the bulk of the big data sources are machine‐generated (mostly satellites) represented in the light blue. We find that satellite data was used in over 70% of the measurement studies and in over 80% of the IEs. Authors' Conclusions This map gives us a sense that there is a lot of work being done to develop appropriate measures using big data which could subsequently be used in IEs. Information on costs, ethics, transparency is lacking in the studies and more work is needed in this area to understand the efficacies related to the use of big data. There are a number of outcomes which are not being studied using big data, either due to the lack to applicability such as education or due to lack of awareness about the new methods and data sources. The map points to a number of gaps as well as opportunities where future researchers can conduct research.

1 | PLAIN LANGUAGE SUMMARY 1.1 | Big data offer unrealised potential for impact evaluations (IEs) Traditional data collection can be costly; target populations may be inaccessible, phenomena cannot always be directly observed and interviewing people may be unethical, dangerous or impossible. In addition, budget constraints can limit data collection.
"Big data" does not require data collection in the field, and can provide insight into economic, social and political behaviour.

| What is this evidence and gap map about?
Big data consist of data such as online searches, social media, citizen reporting or crowdsourced data, process-mediated data such as mobile phone call record details (CRD), commercial transactions data and machine-generated data from satellites, sensors or drones.
Big data can measure outcomes that could not previously be measured using household surveys. The potential of big data to answer causal attribution, however, is still not widely understood, especially in low-and middle-income countries.
What is the aim of this evidence and gap map (EGM)?
This EGM of studies using big data reviews methodological, ethical and practical constraints relating to the use of big data.

| What studies are included?
The map includes IEs that use big data to evaluate development outcomes, systematic reviews (SRs) of big data IEs and other measurement studies that innovatively use big data to measure and validate any development outcomes.
Sources include social networks, internet searches, mobile data, crowd sourced data, data from public agencies, data produced by business, CRD and satellite data.

| What are the main findings of this map?
There is considerable potential for measuring various development indicators using big data. The measurement studies serve as a proof-of-concept for evaluators wanting innovative ways to measure development outcomes.
There is potential for more IEs on development interventions.
The map shows that the number of IEs that use big data to measure outcomes or control variables is growing fast, and there is scope for greater use.
Satellite data are used the most. The use of satellite data for IEs and measurement studies has been facilitated by the availability of preprocessed satellite data, new ML techniques and increased computational capacity to process the satellite images into meaningful measures of development outcomes. Despite a number of highprofile measurement studies, CRD data has not been used to rigorously evaluate any development outcome. Similarly, other human-sourced data and process-mediated data have been used only sparingly in IEs.
There are potential sectors and themes where SRs will be useful.
The map highlights a few potential thematic areas where SRs will be informative, most notably those of (i) all IEs that have used satellite data; and (ii) those with reference to the data sources used in rigorously evaluating forest cover.
The number of measurement studies indicates potential for more IEs in fragile contexts.
Ethical concerns and transparency issues are substantial. Ethical issues related to informed consent, data privacy, data security and unintended exclusion are severe for some of the sources of big data.
Few studies report on ethical issues related to using big data.
Some capacity constraints are acute. Computational capacity is constrained and technical expertise on large-scale big data analysis is siloed.

| What do the findings of the map mean?
This map shows that big data can contribute to the evidence base in development sectors where evaluations are often infeasible due to data issues.
One of the key "absolute gaps" that the map has identified is that there are fewer IEs than measurement studies. Given the fastgrowing availability of big data and improving computation capacity, there is great potential for the use of big data in future IEs. However, analytical, ethical and logistical challenges may hinder the use of big data in evaluations.
This report calls for standards to be set for the reporting of data quality issues, data representativeness and data transparency, and an Institutional Review Board (IRB) review specifically designed for ethical issues related to big data.
More interaction is needed between big data analysts, remote sensing scientists and evaluators.

| How up-to-date is this EGM?
The authors searched for studies published up to December 2019. With the increasing complexity of development programmes, there is a need to collect a vast array of output, outcomes and contextual variables to robustly assess impact. However, significant data collection challenges remain. Data challenges for IEs include limitations on sample size and power due to budget constraints, inaccessible or difficult-to-reach sections of target populations, measurement errors due to recall bias, inadequate frequency and level of aggregation, inadequate information on controls and covariates, data collection lag times and difficulties in measuring long-term impact ‡ (Wassenich, 2007). Further, in some contexts, like conflicts and humanitarian emergency situations, data collection is often impossible. The data gaps and challenges are particularly significant for the populations and countries where the need for evidence-informed policy decisions are perhaps the greatest (Gaarder & Annan, 2013). Another key shortcoming of survey data are inadequate aggregation at subnational administrative units such as districts, counties or villages, inhibiting evaluation of programmes with spatial attributes.
Integrating big data with traditional household surveys and administrative data can complement data availability, quality, granularity, accuracy and frequency, as well as help measure development outcomes temporally and spatially in a number of new ways (Be-nYishay et al., 2018;Lokanathan et al., 2017;Salganik, 2017;UN Global Pulse, 2016;York & Bamberger, 2020). For example, satellite images and mobile CRD have been used in mapping poverty , disaster response  and food security . Web searches and social media were used in predicting unemployment and crime instances (Gerber, 2014;Xu et al., 2013).
While big data is increasingly used for tracking indicators and monitoring development progress on SDGs UN Global Pulse, 2012;Vaitla, 2014), available data are less often utilised to address causal questions about the effects of specific policies and programmes. Big data can contribute to answering some of the causal questions around which interventions work. Big data prediction models can generate proxy estimates for key development outcomes such as wealth, human development, infrastructure quality, forest cover and more, which can be used in experimental (Jayachandran et al., 2016; and quasi-experimental studies . Satellite images such as night light, crop intensity, water availability, land use, proximity to services and physical attributes such as elevation or slope can be used in IEs as a direct measure of outcomes or as covariates. Furthermore, big data can be used for measuring and evaluating the long-term impacts of policies and programmes, conducting ex-post evaluations and estimating spatial heterogeneity. For example, satellite data are available at least as far back as 1993 for all places (high-resolution pictures are available for the entire globe at a granularity as low as 1 × 1 metre), allowing measurement of long-term impacts. This can help fill the gaps in evidence that cannot be addressed by traditional data sources.

| Why it is important to develop the systematic map
The potential of big data to answer causal attribution, however, is still not widely understood or used, especially in LMICs (York & Bamberger, 2020). In this context, a systematic collection of various sources of big data and ways of measuring and evaluating development outcomes will be a great value addition to the development community's contribution to evidence-informed policymaking.
In this paper we look at IEs, SRs and measurement studies § that use big data to evaluate development outcomes with a special focus on fragile contexts. The study highlights the new sources of data; how these new data sources can be used for measuring development outcomes innovatively; and how these new measures can be used in IEs. We map different sources of big data onto development outcomes based on SDGs to identify the current evidence base and its gaps.

| OBJECTIVES
The overarching aim of this report is to inform policymakers and evaluators of existing evaluations based on big data and to provide a database of big data-based IEs and studies that could inform future IEs. Specifically, the objectives of the research are to: • Identify rigorous IEs, SRs and the studies that have innovatively used big data to measure any development outcomes, with special reference to fragile contexts; • Summarise current understanding of potential uses, pros and cons, reliability, biases, risks and ethical issues in using big data for measurement and evaluation of development outcomes and • Generate interest and awareness among key stakeholders (evaluators, researchers, donors, practitioners, implementers and policymakers) of the potential as well as challenges of using big data.
This systematic map addresses the following questions: • How have different types of big data and methods been used for measuring and evaluating development outcomes?
• How dispersed or concentrated is the use of big data across development goals and geographies?
• What are the potential biases, measurement reliability issues, pros and cons, risks and ethical issues in using big data for measuring and evaluating development outcomes?
• What are some of the unexplored but promising applications of big data for IEs? 4 | METHODS

| Evidence and gap map: Definition and purpose
We follow 3ie's methodology and process for evidence gap maps . To create this map, we used systematic methods to identify any completed and ongoing IEs, SRs and big data measurement studies relevant to our research objectives. We conducted systematic searches and data extraction as described in Ap- There are links to study summaries in the 3ie repositories (wherever applicable) and confidence ratings for the SRs.

| Framework development and scope
For the purpose of this research, we define big data sources as digitally generated, passively produced and automatically collected data, as defined in UN Global Pulse (2013). The sources of big data include satellite images, sensors and drones, mobile phone CRDs, commercial transactions data, online searches, social media, citizen reporting or crowdsourced data. See Table 2  • Chemical or radio-nuclear issues • Disease outbreaks or epidemics.
Using big data in evaluation poses a number of analytical challenges on issues including data quality, transparency, generalisability, and privacy and ethical challenges such as consent for using data and anonymisation of the data. This report also explores how the included studies dealt with these challenges. is less likelihood of social desirability bias (Salganik, 2017). The other key characteristic of big data that matters for evaluation is that it is near real-time and can be available across multiple frequencies (e.g., hourly, daily) over a long period. Table 2 provides the types of big data, subclassifications, definitions and sources.

| Types of study design
Using these definitions, we include the following broad classification of big data ‡ ‡ : • Human-sourced information from social networks that is provided voluntarily by users; • Process-mediated data from traditional business systems and websites that includes digitally recorded business activities; • Machine-generated data from automated systems includes information from sensors and machines that measure and record events and situations in the physical world.

| Dependency
Each unit of analysis was a report, peer reviewed journal publication, working paper. We included one study per unit of analysis. In cases where we had multiple reports we included the latest one with preference given to peer review versions and/or based on ease of access to a particular version. When multiple studies were covered by a single report, we included the report and then mapped the data sources and outcomes accordingly.

| Methods for mapping
The studies were mapped using 3ie's evidence gap map platform, which is organised into rows and columns. Various big data sources are placed in the rows and the development themes are placed in the columns. Any intersecting cell represents the development outcome measured or evaluated using the particular type of big data. Different colour bubbles represent the type of study: grey bubbles denote IEs, blue bubbles denote measurement studies, green bubbles denote high-quality SRs and red bubbles denote low-quality SRs. Hovering over the bubbles will T A B L E 1 Selection criteria for studies

Category Description
Population This map includes all population from all countries but we provide breakdowns for rural areas, urban areas, conflicted-affected persons and ethnic minorities. We also provide breakdowns for LMICs and fragile contexts separately Sources of big data Big data may originate from any of the following sources: • • Randomised controlled trial • Regression discontinuity design • Controlled before-and-after study using appropriate methods to control for selection bias and confounding, such as propensity score matching or other matching methods • Instrumental variable estimation or other methods using an instrumental variable such as the Heckman two-step approach • Difference-in-differences • A fixed-effects or random-effects model with an interaction term between time and intervention for baseline and follow-up observations • Natural experiments • Other quasi-experimental studies inducing synthetic control studies • Survey, laboratory or lab-in-the-field type experiments • Cross-sectional or panel studies with an intervention and comparison group using methods to control for selection bias and confounding as described above

Measurement studies
We included the studies that innovatively used big data to measure and validate any development outcomes. These studies use big data to measure components that would have been difficult to measure using survey data.

SRs
We include only the reviews that specifically looked at studies that used big data to measure development outcomes and explicitly described the search, data collection and synthesis methods according to a standard SR protocol, such as the 3ie SR protocol.
Abbreviations: CRD, call record detail; IE, impact evaluation; SR, systematic review.  The figure also shows that applying big data to IEs is a new phenomenon. The first IE using big data was published in 2009. While almost all the IEs were published after 2013, more than threequarters of the IEs were published in the last five years. We expect that the measurement studies will be proofs of concept, leading IEs to adopt to these approaches to innovatively measure development outcomes in evaluations. The map also points to the gap between the growth of measurement studies and use of big data in IEs.  Table H1 for a list of top 20 countries with the maximum number of studies and Table H2 for the geographical distribution of studies across the regions in Appendix G.
We find that most of the studies are concentrated in middle income countries. There are 232 studies (53%) in the middle income group, fol-

| Distribution of studies across data sources
As discussed in Section 2.1.2, big data can be generated by human interaction on social media, process-mediated data recorded by governments and the business and machine-generated data that is recorded by the automated systems. Figure 6 shows that machine-generated data are used the most. Of the total number of studies, close to 84% (n = 380) of the studies used some form of machine-generated data, while 12% (n = 53) of the studies used human-generated sources and 17% of the studies (n = 77) used process-mediated data.

Data from satellites and fixed sensors
Satellite data are the most used source of big data as it accounts for 71% of the measurement studies (n = 210) and close to 81% of the IEs (n = 39). Data from fixed sensors (such as weather and pollution sensors, traffic sensors and electricity metres that provide highfrequency, localised measurements) could also be readily used in IEs.
This is the second most used data source, with 15% of the IEs using these sources. This shows that the data from satellites and in situ sensors that help measure spatial outcomes are used most in IEs.
Other big data sources have been seldom used for IEs despite measurement studies showing proof-of-concept.

Mobile phone CRD
A good number of measurement studies have used CRDs (17% n = 65) for measuring population movement, migration, disease spread and even to understand the literacy level of the subscribers.
Surprisingly, we found no IEs that used this source of big data despite the availability of a good number of proof-of-concept papers in measuring key development outcomes.

Complementarity between data sources
There are about 57 studies on the map that have combined at least two sources of data and about seven of them have combined three or more sources ( Note: Percentage of subcategory total in parentheses. Columns do not add up due to multiple entries.
complementarity. See Table H3 in Appendix G for a list of studies using multiple sources of big data.

| Distribution of studies across development themes
Section 4.2 identifies 10 broad development themes based on SDGs. Figure 7 and Table 5  While most of the SRs looked at cross-sectoral themes, health is the most-studied sector (n = 5), followed by urban development (n = 3) and environment sustainability and economic development (n = 2).
See Appendix F for critical appraisal of the SRs. shows that about 70% of the measurement studies (n = 267) and 65% of the IEs (n = 31) have administrative units as their unit of observation.

| Units of observation
There is a clear distinction between different sources of big data, as shown in Figure 8, Panel 2. The unit of analysis for satellite data-based studies is predominantly administrative units (n = 259, 83%), while CRDbased studies are usually based on population units (n = 53, 82%). This difference shows that satellite data are more applicable when the outcome of interest has some spatial dimension such as local economic development, agricultural land productivity, forest cover or urban development.

| Studies using mixed methods
Mixed-methods IEs that combine qualitative and quantitative analyses help assess the quality implementation and reliability of data and understand the mechanism of programme impact (Bamberger, 2012). Big data IEs can be combined with qualitative methods. However, only three IEs and five measurement studies reported using mixed methods. were studies of ethnic minorities and 2 studied refugees. 15 studies were conducted in the context of epidemics or disease outbreaks. There was one measurement study in the context of a chemical/ radio-nuclear disaster. IEs follow the same pattern, except for one no-   Figure 11 shows that very few studies meet any of the following methodological quality markers.

| Risk of bias in included reviews
• Is the construct validity explained (ie is there a discussion on how the big data-based indicator measures what the study claims to measure)?
• Are data and codes publicly available for replication?
• Are data collection methods discussed?
• Are there data quality issues in the dataset used and how are they addressed?
• Is the data representative of the population of interest?
• Are challenges in the analysis and reporting process discussed? There is, however, considerable difference between IEs and measurement studies in terms of reporting on data quality issues and transparency. Table H4 in Appendix G shows that IEs report a lot better on all these parameters. Of the total 48 IEs on the map, 46 of them report at least one aspect of transparency. Almost all the IEs report on data collection methods, 90% (n = 43) report on construct validity, 60% (n = 29) discuss representativeness of data, 54% (n = 26) discuss generalisability and 45% (n = 22) discuss various data quality issues. However, only 23% (n = 11) make data and codes available and 13% (n = 6) discuss key data analysis and reporting challenges. Figure 11 shows that very few studies meet any of the following methodological quality markers.
• Is the construct validity explained (ie is there a discussion on how the big data-based indicator measures what the study claims to measure)?
• Are data and codes publicly available for replication?
• Are data collection methods discussed?
• Are there data quality issues in the dataset used and how are they addressed?
• Is the data representative of the population of interest?
• Are challenges in the analysis and reporting process discussed?
• Are the results generalisable? For example, are the research findings generalisable to other situations such as other platforms (data sources) or communities, or over time?
Only 95 studies (22%) have reported on at least one of the above transparency criteria. For example, only 20% (n = 91) of the total studies reported on data collection methods, 6% (n = 25) on data quality issues, 8% (n = 36) on data representativeness, 14% (n = 64) on construct validity and 7% (n = 30) on generalisability. Only 4% (n = 19) of the studies have data and codes publicly available or available upon request.
There is, however, considerable difference between IEs and measurement studies in terms of reporting on data quality issues and transparency. Table H4 in Appendix G shows that IEs report a lot better on all these parameters. Of the total 48 IEs on the map, 46 of them report at least one aspect of transparency. Almost all the IEs report on data collection methods, 90% (n = 43) report on construct validity, 60% (n = 29) discuss representativeness of data, 54% (n = 26) discuss generalisability and 45% (n = 22) discuss various data quality issues. However, only 23% (n = 11) make data and codes available and 13% (n = 6) discuss key data analysis and reporting challenges.

| Summary of main results
The use of big data in measuring development outcomes has been on the rise over the past 5 years. This rising trend is powered by the availability of (and our capacity to process) big data. In this section, we discuss the key findings, some of the notable gaps and the potential for future SRs.
6.1.1 | There is a considerable potential for measuring various development indicators using big data We identify a significant and growing evidence base of measurement studies that use some form of big data to measure a development outcome. Some outcomes are more amenable to the use of big data than others; environmental sustainability, economic development and livelihoods, health and well-being and urban development are where the majority of studies are concentrated. Education, sanitation, governance and human rights seem to be less responsive to big data use.
Multiple entries for most development theme indicate the potential of big data in contributing to measuring development indicators. Identifying measurement studies will be a valuable addition to development evaluators who look for innovative ways to measure a development outcome that was difficult to measure at all required spatial and temporal scales using conventional data collection methods.
6.1.2 | There is potential for more IEs using big data on development interventions  (Sabet & Brown, 2018). The IEs also concentrate on using satellite data.

| Satellite data are used most
The map shows that 71% of the measurement studies and 81% of the IEs used satellite data. This is also one of the sources that has been used since the early 2000s. The prominence of satellite data are primarily due to the fact that satellite images offer unique possibilities for measuring and evaluating development outcomes. Given the vast number of satellites covering almost every location on earth, it is possible to collect data at a high granularity (spatial resolution) and  [Hansen et al., 2013], and several others). This preprocessed data or the image data could then be processed and converted into meaningful outcomes to measure economic activity at local level, urban development, forest cover, land productivity, distribution of the population, and so forth. These indicators can also be used for controlling for covariates.

| Spatial dimension matters
One of the key findings of the map is that most of the big data studies are applied in the context where the phenomenon studied has a spatial dimension, meaning the outcome and other covariates are measured on a spatial scale. Close to 70% of the studies on the map report using administrative units as their unit of measurement. This is particularly true for satellite and sensor data-based studies, as 82% have administrative units as their unit of measurement (such as local economic development, agricultural land productivity, forest cover or urban development). This is referred to as geospatial IE (BenYishay et al., 2017). However, there is considerable difference across data sources as CRD data are used to measure changes at the population level.
6.1.5 | CRD data has great potential for measuring and evaluating development outcomes but is not yet used in IEs CRD data are one of the most widely used sources in measurement studies. This is used for measuring population movement, migration, disease spread, and so forth. Despite a number of high-profile measurement studies, our systematic search did not find even one IE that used CRD data for rigorously evaluating a development outcome. This is a notable gap and a potential area for future exploration. It should be noted that CRD data are also fraught with multiple methodological challenges (such as nonrepresentativeness, lack of completeness, etc.) and ethical challenges (such as consent, unintended exclusion, etc.). Further, CRD data has been difficult to obtain as it is proprietary and hence it is difficult to maintain data transparency.
6.1.6 | Other big data sources such as human-sourced and process-mediated data have good proof-of-concepts Human-sourced data (such as social networks, internet searches, mobile data content citizen reporting or crowdsourced data) and processmediated data (such as data produced by public agencies and by businesses) have a good number of measurement studies as proof-of-concept for using these sources to measure various development outcomes, but not many IEs use these sources. This also shows the possibility of potentially using these sources in future IEs. Similar caveats on methodological and ethical challenges discussed above in relation to CRD data will apply.
6.1.7 | East Africa is well-represented, but not the rest of Africa adequate to the task. The ability to "zoom in" on particular zones of interest, and to produce estimates for small areas, is an oft-cited advantage of many types of big data (e.g., satellite and building footprint data, mobile phone CRD and signalling data and app-based location data) and one with particular relevance to evaluative contexts and SDG-related urbanisation, climate change and infrastructure. This holds particular promise for settings where census data renders small area estimation methods unsuitable. Big data has also been shown to be particularly advantageous for the analysis of disaster-induced displacement and disease outbreaks. In each of these cases, the advantage of big data is that it can support rapid appraisal and introduction or adjustment of policies/interventions on the basis of near real-time information.
In this section, we highlight a few examples from the map to show the steps involved in collecting, processing and using satellite and CRD data for measuring development outcomes. We draw on recent projects from 3ie and Flowminder to illustrate the processes.

| Using satellite data in IEs
In a 3ie funded evaluation conducted by the Institute for Financial Using satellite data to assess the impact of EC process requires data on the timing of the intervention, the geographical scope of the intervention (ie the individual mines in this case), the outcome of interest (such as air pollution, land cover and water quality for the corresponding intervention) and control areas for the years before and after the intervention.
The following were the key steps involved in the big data IE.
Step 1: the researchers used web scraping techniques to collect information on the mines from their EC application for the years from Step 2: Satellite data on various key environmental outcomes such as air pollution, land cover and water quality were collected from different sources. Step 3: The researchers then linked the geocoded mine sites to the corresponding cells of environmental outcomes. Of the total 934 mines in their database, they could link 889 of 1 km cells of EVI data and 882 of 250 metre cells with corresponding geocoded mine sites.
They could also link 538 site monitors to the mines.
Step 4: The new EC reform required the mines of between 5 and 25 hectares in area to hold a public hearing that had not been considered big enough to hold public hearings during the previous regime (ie only the mines of above 25 hectares in area were required to hold the hearings). This study exploited the discontinuity around the 25 hectare mark and compared the mines marginally above 25 hectares with the ones marginally below 25 hectares. The final sample included 134 mines, of which 68 were treatment mines (<25 hectares) and 66 were control (>25 hectares). Using data before and after the EC applications, they estimated a difference-in-difference model.
This study, utilising web scraping to collect data on project characteristics and various sources of satellite data for measuring the outcomes of interest, is an excellent example of innovative data collection methods in a sector where the evidence base is very small (Rathinam et al., 2019b).

| Using CRD analytics to inform disaster management
In this section, we briefly outline the process for undertaking CRD analytics to measure, characterise and predict population displacement and returnee/resettlement patterns in post-disaster settings. While applications of CRD data analytics to date have lacked an evaluative component, their potential in this regard is evident. We draw on a recent project at given the need to safeguard subscribers' personal data. † † † † Prior to analysis, MNO data underwent a long series of cleaning and preprocessing steps as part of quality assurance and to support the generation of standardised metrics. A first stage of analysis was undertaken to structure the data in a usable format and to detect data anomalies. Once data was cleaned, quality assured and converted into an analysable format, a number of preliminary processing steps were undertaken, including: • Clustering of cell tower locations • Assessment of each subscriber's phone usage behaviours (number of events, frequency and regularity) • Determination of a predisaster "home" location.

| MNO data: Preprocessing steps
Here are some commonly occurring issues in MNO data. Once identified, corrections and/or accommodations can be performed prior to and/or during the preliminary processing and analysis phases.
1. Standard data quality issues applicable to MNO data: • Item missing data (incomplete data records ie missing fields) • Invalid entries for fields ***There are several ways to collect the required satellite data. For a few key variables such as night lights, air pollution, land cover and water quality, elevation, slope, distance from certain services or infrastructure, geocoded data for various granularity and frequency is readily available in several databases such as Aiddata DataQuary, SEDAC, and so forth. This study has utilised data from differences sources that provide readily useable data. Alternatively, the researchers use ML techniques to analyse satellite images and predict development outcomes . Another useful source of areal images come from custom built drones that can provide very high resolution data for the exact spatial and temporal frequency . † † † † Flowminder's original "data partnership" model developed lasting collaborations with individual MNOs. The priority was countries where substatial potential gains were available from novel, digital data given the existing data landscape. Lengthy negotiations with MNOs followed, often spanning many years and consuming extensive organisational resources, with no guarantee of a successful outcome. When this model "worked," it led to strong and sustained partnerships with MNOs. In pursuit of impact at scale, Flowminder Foundation supplemented its original "data partnership" model with a toolkit-based approach designed to break down silos between data and methods, in effect negating the need for Flowminder to access MNO data by transferring methods expertise to MNOs themselves via the "Flowkit" suite of software.
• Duplicate records • Interrupted data series' (e.g., no data for a particular time period) • Inconsistent values (either in format, or definition) for keys that are used to join multiple datasets together • Inconsistent entries for the "same" value (e.g., different spellings of the same place name).
2. Issues specific to MNO datasets, CRD: • Inconsistencies or errors in method used for "hashing" (a form of pseudonymisation) subscribers' IDs • Inconsistent "hashing" of sender and recipient IDs for communication events (e.g., standard SMS or phone calls).
3. Issues specific to MNO datasets: Cell location and coverage maps: • Cell locations occur outside national borders 4. Common data anomalies, indicative of a network issue or a sudden change in subscribers' behaviour due to an event or "shock": • Individual cell towers have significantly more/less traffic than normal • Overall network traffic is significantly higher/lower than expected • Traffic from a particular region is significantly higher/lower than expected.
The processing steps undertaken to discern at individual level disaster-induced displacements from pseudonymised, time series CRD data are presented below in Figure 13.
The analysis disclosed striking commonalities in internally displaced person (IDP) return/resettlement rates, with the fraction of IDPs who remain displaced exhibiting a common rate of decay across all three post-disaster settings studied.
In a further step, the team developed new mobility and social network metrics to permit analysis of the relationships between contextual and individual variables and displacement duration, distance and trajectories, controlling for the severity of impacts and humanitarian response. The results suggest that the dispersal of an individual's social contacts and travel history predisaster are highly predictive of their post-disaster displacement trajectories. Individuals with localised travel patterns and social contacts were more likely to be displaced in the vicinity of their usual residence compared with those with more dispersed travel patterns and social contacts. A majority of IDPs remained within a 10 km radius of their usual place of residence. Across the three disasters, 60-70% of long-distance displacements (in excess of 100 km) involved travel to a familiar location and/or proximate to one or more contacts discernible in the predisaster CRD data. This pattern holds controlling for the severity of impacts at local area level and is consistent across all three disasters.
Results were validated via comparisons with reports retrospectively quantifying population displacements produced by the International Organisation for Migration, as well as with reference to data on the intensity of each disaster's impact on affected areas. The results indicate that CRD data analysis can be used to predict the estimated number and spatial distribution of IDPs at different time points based on initial estimates of the number of persons displaced in the immediate wake of a disaster, as well as to predict recovery/ resettlement timelines. This has important implications for postdisaster humanitarian response and resettlement efforts. The same methods can support disaster resilience assessments and planning and provide a means to compare recovery and resettlement rates across different disaster events.
6.3 | Areas of major gaps in the evidence 6.3.1 | Satellite data also presents misclassification problems Researchers, however, point to several technical challenges in using satellite images that may provide misleading conclusions. For example, Jain (2020) argues for the need for ground validation of satellite data as sometime the images could be misclassified (e.g., flood irrigation may be classified as flooded area); often this misclassification is systematic (ie forest cover is almost always misclassified as agriculture, which will bias the study results). This can be rectified with a field visit. Further, there could be differences in the data coming from different satellites and from the same satellite constellation but using different sensors (e.g., Normalised Differ-

| Data quality and transparency is paramount
The map also points to the need to set standards for better reporting, as about 87% of the measurement studies did not report on data quality issues, representativeness, construct validity and generalisability. This would lead to questioning the internal and external validity of the findings. There is also a need to set standards for data transparency, taking into consideration the challenges in sharing proprietary data, data storing and the capacity of the Dataverse (see Box 1 for more details on data transparency).

| Ethical concerns are substantial
Ethical challenges such as consent, data privacy, data security and unintended exclusion are well documented in the literature York & Bamberger, 2020).
A brief analysis of the studies on the map shows that very few studies report on any of these ethical challenges. However, the challenges are different for different sources of big data. For example, satellite data that involves little human interaction may not need an IRB review but most other big data source that use human-generated data without explicit consent for secondary use should be reviewed by IRBs. We also recommend more mixedmethod big data evaluations to mitigate the potential disconnect between development stakeholders and big data researchers.
Any mixed-method research needs to be reviewed by IRBs.
The map shows that most IEs have done well on reporting data quality issues but not on ethical issues. Since big data involves ethical issues (such as consent for secondary data use and unintended exclusion) that are new to conventional ethical standards, there is a need to update the current ethical standards practice to include big data use as well.
6.3.4 | Big data may be growing in use and popularity, but the need for independent auxiliary data for "ground-truthing" remain Many sources of big data are partial in terms of coverage and prone to biases that are difficult to measure, control and correct for in the absence of secondary data. Despite growing awareness and acknowledgement of its limitations, the household sample survey remains the dominant source of development policymaking. Big data often require survey data as "ground-truth" data to validate the findings. Demographic Health Surveys and Living Standards Measurement Studies are the two main surveys used in ground-truthing.
There is considerable scope for merging the income and expenditure surveys, and food surveys conducted in several developing countries with big data to assess food shortages, poverty hotspots, and so forth.

| Some capacity constraints are acute
Development organisations need to build staff capacity in order to use big data as a strategic asset (Perera . They need to build multidisciplinary teams consisting of data experts and subject matter professionals, and also compete with the private sector to recruit the staff. Other major costs involve scaling up the technical infrastructure to enable data storage and processing on a large scale and data accessibility costs. The latter can be more difficult to predict considering that big data sources that are currently public may involve licensing in the future. Besides, ensuring the sustainability of data can be a cause of concern. As suggested by Hammer et al. (2017), as most of the | 21 of 57 big data is produced as a by-product by the private sector, continuity of data provision cannot be guaranteed in this age of evolving technology and market conditions. These concerns will call the wisdom of committing resources upfront to build capacity into question.

| Need for better coordination between data scientists and evaluators
Big data analysts and evaluators use different framework and analytical tools. In particular, the big data measurement studies look for hidden patterns in the data with little support from theory and aim at prediction rather than causality (York & Bamberger, 2020).
Further, the expertise needed to analyse big data remains largely localised and siloed. Outside of a small and highly specialised group of data scientists, there is uncertainty about how best to carry out large-scale big data analysis. The degree of technical specialisation combined with strict access restrictions to many types of big data has hindered big data applications in development evaluation.
Hence, there is a need to promote interaction between development evaluators and data scientists for better cross-learning and adoption of big data in measuring and evaluating development outcomes.

| The cost of collecting, analysing, storing and reporting big data is largely unknown
There is very little publicly available information on the cost of collecting, analysing and reporting big data.  reported that the phone survey for ground-truthing the CRD data costed USD 12,000 and took 4 weeks to administer. This is, however, only the variable cost of data collection in this study. There are multiple hidden costs such as staff costs and the cost of the necessary computing infrastructure (including storage); in addition, the opportunity cost of time involved in developing partnerships with data providers in some cases is not known. BenYishay et al. (2017) report that the cost of geospatial IE is around USD 150,000.
One of the 3ie funded studies using satellite data are reported to have spent USD 3300 on data collection, which is about 1% of the total study budget, but have spent USD 103,864 on data analysis and reporting (about 32% of the total cost ‡ ‡ ‡ ‡ ). Similarly, another 3ie funded study that used in situ fixed sensors reported spending USD 6152 on data collection or acquisition (11%) and USD 54,444 on staff costs for analysis and reporting (55%). However, studies that have combined satellite data with household survey have reported higher costs of data collection (USD 171,582; 43%) and analysis and reporting (USD 109,495;27%). This is, however, roughly comparable to the data collection cost of an average 3ie funded multi-year, multi-round survey IE, which costs about USD 176,000 (Puri & Rathinam, 2019).

| Evaluation of potential methodological issues
While big data can help resolve many data related challenges, there are considerable methodological, analytical, logistical and ethical challenges in the way of using it in measuring development outcomes (Letouzé, 2016;Lokanathan et al., 2017;Olteanu et al., 2019;Salganik, 2017). This section briefly discusses some of the prominent methodological challenges. As big data is varied in type, quality and composition, we also discuss if the challenges are specific to any particular type of big data.
We have grouped all the big data challenges that may affect measuring development outcomes credibly and lead to questionable internal and external validity of the studies.

Nonrepresentativeness of data and selection bias
Big data may unintentionally exclude certain sections of the population or marginalised communities, thereby making the sample unrepresentative of the population being analysed. Large samples do not solve this systematic bias. This, however, is not a challenge when using satellite data that has universal coverage but, with humangenerated and CRD data, nonrepresentativeness is a serious challenge. Human-generated data such as Twitter, Facebook or web searches, as well as mobile phone use that generate CRD data, are not representative as the usage is limited by income, education, infrastructure, and so forth. However, clarity on what is the sample frame (ie who is included and who is excluded) will help interpret big data results appropriately. Nonrepresentative sample is still useful for within-sample comparisons, but may lead to erroneous out-ofsample generalisations (Olteanu et al., 2019;Salganik, 2017).

Construct validity
Construct validity is whether the proposed measure actually measures what it claims to be measuring. This becomes important when the construct is unobservable and has to be operationalised via some observed attributes (Olteanu et al., 2019). For example, does night light data truly reflect local GDP and other development outcomes such as health and education? What is it in the CRD data that reflects people's income or employment status? In many cases, the big databased measures may not be straightforward, and it is good practice to clearly state construct validity and provide necessary support to back the claim in the papers. Development measures based on social media are particularly challenging due to different communication styles, special usage of terms and differences in language proficiency.

Data quality issues.
• Comparability of data over time: Since most of these data are collected routinely as a part of business, the nature and quality of data may change with the technology and business requirements.
This may happen because the underlying technology has changed or because the people who use it have changed. For example, satellite data are not readily comparable across the years as there is a vast quality difference (Jain, 2020); good flu trends based on online searches peaked comparing to officially reported data when ‡ ‡ ‡ ‡ It should be noted that the cost discussed here includes only the variable cost of data collection and the staff time, but may not include the cost of fixed infrastructure and equipment. the underlying Google algorithm started prompting people to query more and broke the relationship between Google searches and flu prevalence (Archie et al., 2018).
• Lack of completeness: Most big data is a by-product of peoples' everyday action and/or result of system logs of the government and businesses. It may not contain all the necessary information, such as demographic characteristics. However, combining multiple sources of data, especially big data and administrative data, can help resolve this problem (Salganik, 2017).

Generalisability
Generalisability or external validity refers to the applicability of the findings of a study to population or context other than it was produced. In the context of big data, generalisability would mean the applicability of the model to a setting different from the setting of the data that the model was trained on. For example, a model trained on satellite data from a specific geographical region may not be generalised . It is good practice for studies to report on the representativeness of training data.

Data transparency
Transparency in this context refers to publishing all relevant materials, including the data and code, used in a study in the public domain for independent verification. Sharing of raw data in the public domain is often crucial for establishing confidence and reliability in the results.
There are two challenges here: first, some of the major sources of big data (such as CRD) are proprietary and sharing may not be permitted beyond the closed group of researchers; and second, the data has to be de-identified before it can be shared and it is crucial to check whether there are variables or a combination of variables that can be used to reidentify research subjects.
We assessed whether the studies included in the systematic map asked the following questions: • Is the data representative of the population of interest?
• Is the construct validity explained (ie is there a discussion on how the big data-based indicator measures what the study claims to measure)?
• Are there data quality issues in the dataset used and how are they addressed?
• Are the results generalisable? For example, are the research findings generalisable to other situations, such as other platforms (data source) or communities, or over time?
• Are data and codes publicly available for replication?

| Evaluating reporting on privacy and ethical considerations
There are concerns over data access, privacy, consent and ethics in using big data. Although these are foundational issues for both small and big data studies, the challenges posed by big data have greater repercussions.
When using big data sources such as mobile data, most mobile operators have "inform and consent" policies that mandate disclosure of all relevant information to potential participants who can then evaluate this information and give explicit permission. However, these policies often contain legal language that is generally not discernible, and it is not clear if explicit consent is obtained to repurpose the data. This kind of informed consent may be completely absent in research leveraging social media data due to the impracticality of obtaining consent from millions of users.
Mobile phone user data and social media data are some of the most used sources of big data that can inform researchers about individuals' behaviour. Even if the data are de-identified, concerns still remain over the consent and ethics of sharing such data with researchers. It is thus imperative to have an ethics approval process in place that lays down the conditions under which such research can take place. There is a need for clear ethical standards for big data research and studies should be monitored by the IRBs.
Another ethical criterion when using big data can be concerned with the assessment of risks, the most common being privacy breaches leading to identity theft or other cybersecurity risks. The possibility of the reidentification of any individual user from poorly anonymised datasets adds to the concerns over anonymity of subjects. When combined with other sources, such datasets can be used to gain detailed insights about people without their knowledge. Such precise inferences may create the capacity for discrimination or mass manipulation. Sometimes data obtained for one purpose in social data research is used for secondary analyses, but the associated risks may not be well understood. For example, Facebook data in the past has been used for ad targeting, as well as for tailoring propaganda (Horowitz et al., 2018).
Big data may also inadvertently exclude certain sections of the population. For example, this bias can be observed in the case of "Street bump," a mobile app that notifies the Boston City Hall whenever the user hits a bump on the road (Carrera et al., 2013). The data includes information only from the app users who often use both their cars and the app; this might inadvertently exclude poorer parts of the city that app users may not frequent. Policy based on such big data sources may have unintended consequences for the people who are excluded.
We assessed the studies on the following: 6.3.10 | Reporting on privacy and ethical challenges Figure 12 shows that most studies (81%) do not report on ethical challenges and privacy issues. Of the few that do discuss such challenges, the most frequently discussed issue is consent for data use. This map covers large thematic areas and outcomes corresponding to SDGs. Given the wide scope of the outcomes, the evidence is sparse and bunched around a few themes. The thematic gaps here may not be read as actual gaps, but these areas may be not readily relevant to using big data. This map rather shows what evidence or proof-ofconcepts are available to measure and evaluate development outcomes using big data.
This map followed a systematic process of searching, screening and coding of studies based on a predefined set of criteria in the study protocol developed with inputs from key stakeholders. However, despite best efforts in searching and screening the studies, given the wide scope of big data sources and their application across all developmental themes and the pace at which the literature is growing, it is possible that some relevant studies (especially measurement studies) could have been missed out.
It was beyond the scope of the study to provide a critical quality appraisal of the IEs or the measurement studies, given the large number of studies included on the map; nor did the report look at the details of ML methods used in the included studies.
Given the wide scope of development applications, it was not possible to code the studies for all subclassifications. Though the submaps (especially for economic development and livelihoods, health and well-being and urban development) provided coding at level 2 indicators, it was not possible to provide granular analysis of development themes corresponding to SDG indicators at level 3.
Future systematic maps should aim to produce more granular classifications on the use of big data at the indicator level.
Some studies have used ML techniques for treatment effect heterogeneity in RCTs (Chernozhukov et al., 2019). However, it was beyond the scope of this report to include the role of big data analytical methods in conventional IE designs such as RCT and other quasi-experimental designs. This is a nascent but growing body literature and could be considered for inclusion in future maps.
Several studies suggest that the key advantages of big data sources (especially satellite data) are their long-term availability which will help evaluate the long-term impact of development interventions. The possibility of collecting a vast array of information on several contextual factors using big data can help evaluate complex interventions (Bamberger, 2016). However, this map did not code the studies for long-term impact or for complex interventions.
Future maps may code and analyse the role of big data in measuring long-term impact and in evaluating complex interventions.

| Stakeholder engagement throughout the systemaic map process
The stakeholders in this systematic map included an advisory group comprised of sector experts, FCDO staff, CEDIL staff. All stakeholders were engaged in reviewing drafts and final reports associated with the map. The protocol for the map was developed with inputs from the advisory board, and FCDO and CEDIL staff. The advisory board played a key role in steering the search strategy and building the list of key words.

| AUTHORS' CONCLUSIONS
Big data has great potential to help address questions of relevance to international development, including for evaluating the effects of interventions. This systematic map compiles IEs, SRs and measurement studies that incorporate big data to highlight how this innovative, new data source is being used to evaluate development outcomes and (more importantly) where there is more potential to use big data in the future evaluations. We found 437 studies, of which 48 are IEs, 381 are measurement studies and 8 are SRs.
Roughly half the studies are from Asia and another 30% are from Africa; about 70% are from LMICs. Of the 48 IEs, 8 are RCTs and the remaining are quasi-experimental studies.
Our results highlight considerable potential for using big data for measuring various development outcomes across SDG themes, but big data is more relevant to environmental sustainability, economic development and livelihoods, health and well-being and urban development.
This map also highlights that big data can contribute to the evidence base in development sectors where evaluations are not generally feasible due to a lack of data, particularly due to fragile contexts.
One of the key "absolute gaps" the map has identified is that the number of IEs is lower in comparison to measurement studies. Given the fast-growing availability of big data and improving computation capacity, there is great potential for using big data in future IEs. This may not, however, be straightforward as there are several analytical, ethical and logistical challenges that may hinder the use of big data in evaluations. The development community that helps set standards and best practices and development stakeholders (including donors who facilitate rigorous evaluations and learning) have a strong role to play in facilitating this process. The report highlights the need for setting standards for better reporting on data quality issues, representativeness, construct validity and generalisability, as well as the need for data transparency and sharing. The report also calls for facilitating better interaction between big data analysts, remote sensing scientists and evaluators.
One of the key findings of the report is that satellite and sensor data are the most used data sources for both measurements studies and IEs. There are several sources of preprocessed satellite data that could be used directly in evaluations without the evaluators having to process them using complex ML models themselves. Satellite data seems to be particularly useful in the context where the development interventions and the outcomes studied have spatial dimension economic activity at the local level, urban development, forest cover, land productivity and distribution of the population, or where the outcome and other covariates are measured on a spatial scale (ie villages, counties, districts, plots or protected areas). CRD data, on the other hand, despite being used widely in measurement studies, is not yet used in IEs. The data deficiency in international development is partly due to fragile contexts such as diseases spread, violence, natural calamities and difficult terrain. This map highlights the potential of big data in fragile contexts: one-quarter of the studies were conducted in such a context.
For evaluators and researchers, the report calls for better reporting on data quality, ethics and transparency. There is also an absolute gap in using mixed methods jointly with big data and costeffectiveness. For the donors, this report calls for more efforts on setting up best practices and ethical standards and in facilitating more interaction among remote sensing scientists, big data analysts and development evaluators.
7.1 | Implications for research, practice and/or policy • Reliable data are paramount to evaluating development outcomes and future resource allocation.
• This systematic map compiles the IEs, SRs and measurement studies to highlight how innovative, new data sources are being used in evaluating development outcomes, and more importantly where there is more potential to use big data in the future evaluations.
• This map shows that big data can contribute to evidence base in development sectors where evaluations are not generally feasible due data deficiency.
• Given the fast growing availability of big data and improving computation capacity, there is a great potential for using big data in the future IEs.
• There are several sources of preprocessed satellite data that could be used in evaluations directly without the evaluators having to process them using complex ML models themselves • There is also an absolute gap in using mixed methods jointly with big data and cost effectiveness. This should be prioritised by donors and researchers as a mix of quantitative big data analysis and qualitative field level analysis will help strengthen the validity of the results.
• More efforts, on the donors' end, is required to set up best practices and ethical standards, and facilitating more interaction among remote sensing scientist, big data analysts and development evaluators.

ACKNOWLEDGEMENTS
The authors would like to thank Sriganesh Lokanathan, Ariel Be-nYishay, Neeta Goel, Marshall Burke, Marie Gaarder for their valuable inputs. The authors will also like to thank John Eyers, members of the CEDIL quality assurance team for their feedback. FCDO and CEDIL provided funding for this systematic map and report.

DECLARATIONS OF INTEREST
The authors declare no conflict of interest.

PLANS FOR UPDATING THE SYSTEMATIC MAP
The systematic map shows that both IEs and measurement studies have dramatically increased in the past 5 years and are continuing to grow in number. Given the potential for faster growth in the availability and computational capacity, it is very likely that the number of studies will grow faster than we have witnessed over the past 5 years. Hence, we recommend that this map be updated within the next 2 years.
The fact that more than 80% of the included studies are peerreviewed shows the growing number of journals interested in big data application in international development. It will be useful to include a more exhaustive grey literature search to identify the full extent of the literature.
This map shows the potential for big data to measure and evaluate various development themes. However, most of these studies are supported by universities and specialist organisations and conducted by researchers associated with these organisations. Widely disseminating the findings of the map among development researchers, evaluators, practitioners and donors will help promote the adoption of big data measures in future IEs. Allcott, H., & Rogers, T. (2014). The short-run and long-run effects of behavioral interventions: Experimental evidence from energy Feyera, S. (2018). Community perception of land use/land cover change and its impacts on biodiversity and ecosystem services in northwestern Ethiopia. Journal of Sustainable Development in Africa, 20, 108-126. Finger, F., Genolet, T., Lorenzo, M., de Magny, G. C., Manga, N. M., Rinaldo, A., & Bertuzzo, E. (2016). Mobile phone data highlights the role of mass gatherings in the spreading of cholera outbreaks.

DATABASES SEARCHED
We developed a systematic search strategy in consultation with an information specialist after finalising the protocol.
• Eldis: www.eldis.org We also searched grey literature via Google Scholar and checked references of any SR that we find in our searches and met our inclusion criteria.
Given the relative recent state of the evidence base in this field and the fact that most programmes/initiatives started to flourish in the late 2000s we conducted the searches from 2005 onwards.
APPENDIX G: SR APPRAISAL TOOL AND SUMMARY G.1. Summary of the SRs Yan et al. (2017). Utility and potential of rapid epidemic intelligence from internet-based sources The study aimed to summarise internet-based methods that use freely accessible and unstructured data for epidemic surveillance, exploring their timeliness and accuracy outcomes. The study is based on 84 articles published between 2006 and 2016 relating to internet-based public health surveillance methods. These studies employ search queries, social media posts and approached derived from existing internet-based systems for early epidemic alerts and real-time monitoring. The primary methodology used for this review is the preferred reporting items for SR and meta-analyses. Using this method, the authors can assess the benefits and challenges of a healthcare intervention through an evidence-based minimum set of items. The study does not clearly demonstrate the inclusion criteria for the study designs, making it difficult for the readers to interpret the findings. Fung et al. (2016). Ebola virus disease and social media: A systematic review The study is an SR of the existing research pertinent to the Ebola virus and social media, especially to identify the research questions and methods used to collect and analyse social media. The study searched six databases for research articles relevant to Ebola and social media.
Twelve articles were included in the main analysis: seven from Twitter and one including Weibo, one from Facebook, three from YouTube and one from Instagram and Flickr. All the studies were cross-sectional. The study uses a standardised form to extract the data. A key challenge of the review is that the methods used by the review authors to analyse the findings of the included studies are not clearly defined. | 51 of 57 articles were selected. The authors did not explicitly state the methods used to analyse the quality of included studies and also did not report results for each of the studies in the review.
De Souza et al. (2019). Data mining and machine learning to promote smart cities: A systematic review from 2000 to 2018 This study aimed to present an SR regarding data mining (DM) and machine learning (ML) approaches adopted in the promotion of smart cities. The study seeks to provide, from a literature review in journals belonging to the Web of Science and Scopus databases, the different DM and ML techniques used, as well as to present the sectors most engaged in the promotion of smart cities. A total of 39 studies were included on the map, which were further analysed to assess the most commonly used DM and ML techniques to promote smart cities.
While the report describes individual results of the studies clearly, it does not combine the results of all the studies. -Smith et al. (2015). Using social media for actionable disease surveillance and outbreak management: A systematic literature review

Charles
The studies in this review demonstrate how social media may be a valuable tool in improving the ability of public health professionals to detect disease outbreaks faster than by using traditional methods and to enhance outbreak response. A social media application was defined for this review as "an internet-based application where people can communicate and share resources and information, and where users can activate and set their own profiles, have the ability to develop and update them constantly and have the opportunity to make such profiles totally or partially public and linked with other profiles in the network." A total of 60 articles were selected for this SR, which addressed the two research questions: can social media be integrated into disease surveillance and can it be used to improve health outcomes? Krenn et al. (2011). Use of global positioning systems (GPSs) to study physical activity and the environment: a systematic review The aims of this SR were to determine the capability of GPS to collect high-quality data on the location of activities in research on the relationship between physical activity and the environment. Studies were eligible for inclusion if they were undertaken on humans, used GPS to measure the locations where physical activity occurred and included analysis of the relationship between the characteristics of the environment and any form of physical activity behaviour (including leisuretime physical activity, sport or active travel). The capability of GPS was expressed in terms of data quality, which in turn was defined as the proportion of GPS data lost in each study. The authors do not mention the proper risk of bias assessment of the included study. Bennett and Smith (2017). Advances in using multitemporal night-time lights satellite imagery to detect and estimate and monitor socioeconomic dynamics