PROTOCOL: Adult skills development and training in high‐income countries: A Campbell evidence and gap map

Human capital is a chief driving force of economic development. The Organization for Economic Co‐operation and Development (OECD) estimated that approximately 9% of current jobs within OECD member states are threatened with automation and digitalization—all significant successes and advances in artificial intelligence, robotics, and computer science (Arntz et al., 2016). With such global changes and forecasts, in the labor market, there are ever‐evolving demands on employees and employers to gain or update new and/or different skill sets and competencies. Governments and policy‐makers at various levels have shifted towards investments and surveillance of various types of job training and skills acquisitions (OECD, 2009). Active labor market programs help to bring individuals into employment, keep workers employed, optimize productivity and earnings through various means (Brown & Koettl, 2012). Further to the threat of automation and digitalization, international organizations are monitoring and estimating labor market trends considering COVID‐19 and its impact on the various economies worldwide. As of September 22nd, 2020, there have been over 31 million cases and over 960,000 deaths associated with COVID‐19 (John Hopkins University, 2020) with numbers still increasing daily. The consequences of this pandemic will include a significant disruption and impact on global unemployment and underemployment rates, implications for the quality of work available with reduced income, and working poverty, and worsening social inequalities and inequities as less access to social protection may also impact the quality of work done by employees (International Labor Organization, 2020). Furthermore, these critical implications can contribute to a global economic recession (UN Conference for Trade and Development, 2020). COVID‐19 has led to increased demand for some occupations and shortage in specific skills as well as social displacement of workers, for example, some staff have been redeployed to new posts and newly trained healthcare professionals were introduced earlier than expected into needed posts (such as screening for COVID‐19). Evidence on effects of labor market programs to reduce unemployment from previous economic crises may be applicable to the COVID‐19 recovery plan. A key challenge will be to reduce unemployment by skills development and training, whether through upskilling to remain acquainted with and ensure skills match present labor market needs, or reskilling. The UN suggests that skills development programs may be a part of a policy response to COVID‐19. Skills development and training is important by virtue of the outcomes they can produce at both the system, employer, and individual levels.


| The problem, condition, or issue
Human capital is a chief driving force of economic development. The Organization for Economic Co-operation and Development (OECD) estimated that approximately 9% of current jobs within OECD member states are threatened with automation and digitalization-all significant successes and advances in artificial intelligence, robotics, and computer science (Arntz et al., 2016). With such global changes and forecasts, in the labor market, there are ever-evolving demands on employees and employers to gain or update new and/or different skill sets and competencies. Governments and policy-makers at various levels have shifted towards investments and surveillance of various types of job training and skills acquisitions (OECD, 2009).
Active labor market programs help to bring individuals into employment, keep workers employed, optimize productivity and earnings through various means (Brown & Koettl, 2012).
Further to the threat of automation and digitalization, international organizations are monitoring and estimating labor market trends considering COVID-19 and its impact on the various economies worldwide. As of September 22nd, 2020, there have been over 31 million cases and over 960,000 deaths associated with COVID-19 (John Hopkins University, 2020) with numbers still increasing daily.
The consequences of this pandemic will include a significant disruption and impact on global unemployment and underemployment rates, implications for the quality of work available with reduced income, and working poverty, and worsening social inequalities and inequities as less access to social protection may also impact the quality of work done by employees (International Labor Organization, 2020). Furthermore, these critical implications can contribute to a global economic recession (UN Conference for Trade and Development, 2020). COVID-19 has led to increased demand for some occupations and shortage in specific skills as well as social displacement of workers, for example, some staff have been redeployed to new posts and newly trained healthcare professionals were introduced earlier than expected into needed posts (such as screening for COVID -19). Evidence on effects of labor market programs to reduce unemployment from previous economic crises may be applicable to the COVID-19 recovery plan. A key challenge will be to reduce unemployment by skills development and training, whether through upskilling to remain acquainted with and ensure skills match present labor market needs, or reskilling. The UN suggests that skills development programs may be a part of a policy response to COVID-19. Skills development and training is important by virtue of the outcomes they can produce at both the system, employer, and individual levels.  (in-person, digital, practitioner-led training, by public or private providers;Orlik, 2018). These interventions are often provided in various combinations and may be geared toward paid employment or self-employment. They can lead to an impact at the participant level (e.g., skills, employment and retention, earnings), and at the level of the employer (e.g., retention, hire quality) and the system (e.g., diversity of workforce/equity, community well-being).

| The intervention
Although many programs have a positive impact on participants, employers and labor market systems, some negative impacts such as reduced wages and displacement have occurred as a result of imbalance in labor demand and supply (Brown & Koettl, 2012;Crépon et al., 2013). See the conceptual framework of interventions and outcomes we developed in consultation with stakeholders who have expertise in skills development and policy ( Figure 1).

| Why it is important to develop the evidence and gap map (EGM)
An evidence and gap map EGM is a decision making and research prioritization tool used to identify areas of evidence as well as gaps in research to inform social policy, program, and research priorities (Snilstveit et al., 2013). The roles of artificial intelligence and other disruptive technologies in economies have gained prominence among governmental and legislative policy agendas (Jansen et al., 2020). This EGM is aligned with the priorities of the Future Skills Centre (FSC), a pan-Canadian initiative, and who will be the map's primary user. FSC's mandate is to connect ideas and innovations generated across Canada so that employees and employers may thrive in the labor market, and to ensure that economies at the local, regional, and national levels flourish.
Working with their partners to inform and support local approaches to skills development and employment training in ever-changing economies, FSC will use this rapid and evidence gap map to direct their future research agenda, which is time-sensitive due to the social and economic consequences of COVID-19.

| Existing EGMs and/or relevant systematic reviews
Similar EGMs exists with a focus on higher levels of education and skills (Winters, 2018), funding for adult skills development (Gloster et al., 2016) and youth skill development in low-middle income countries (Rankin et al., 2015). Two systematic reviews have also been conducted one on vocational and business training and women's labor market outcomes in low-and middle-income countries (Chinen et al., 2017) and the other on youth employment programs and labor market outcomes in high-and low-middle income countries (Kluve et al., 2019).
Since there are existing EGMs for skills development in low-and middle-income countries and the setting for skills F I G U R E 1 Conceptual framework of interventions and outcomes development is different in low-resource settings, we decided to focus on high-income country evidence for this EGM.

| OBJECTIVES
The objective of this EGM is to identify primary studies and systematic reviews on the effects of adult skills development and training on outcomes for the jobseeker/employee, employer and labor market systems in high income countries.
2. Identify the available evidence.
3. Appraise the quality of the evidence. 4. Extract, code, and summarize the data that relate to the objectives.

Visualization and presentation of the findings in a user-friendly
manner.
We will use the Campbell Collaboration mapping tool developed by the EPPI-Centre (Digital Solution Foundry & EPPI-Centre, 2020) to generate and display identified studies using the framework described below.

| Framework development and scope of the EGM
Our intervention and outcomes framework was developed and adapted from the National Endowment for Science, Technology and the Arts (NESTA) and the FSC's Common Outcomes Framework (Future Skills Centre, 2019). Our 15 intervention categories are based on NESTA's nine components for an inclusive and responsive learning ecosystem (Orlik, 2018), which are highlighted as recommendations on how to plan policy for skills provision. The FSC Common Outcomes Framework is a set of measures that focus on employment-related outcomes approaches both within Canada and internationally. We held a stakeholder workshop in February 2020 with academics, economists, and community-based organizations in order to refine our interventions-outcomes framework. Workshop participants suggested a recategorization of the framework categories and filter options, and these will be used in our map. The framework will follow a standard intervention and outcomes matrix where the rows are interventions and the columns are outcomes. This conceptual framework includes skills development interventions which recognize and seek to overcome systemic barriers to equitable employment, such as lack of sufficient wrap-around support, need for culturally responsive interventions and need for commitment and activation measures from employers to ultimately support a diverse workforce and promote community well-being.

| Eligibility criteria
We will include studies which assess the impact of interventions or programs with a component of skills development and/or training among individuals post high school age. For example, a jobs search assistance program without a skills development component will be excluded.
Interventions may take place in any setting (e.g., community centre, workplace, educational institutions, etc.) in high income countries. We provide more details on our dimensions below.

| Dimensions
The EGM framework for an EGM informs the inclusion and exclusion criteria. We chose intervention and outcomes as our key dimensions.

| Types of study designs
We will include primary studies and systematic reviews evaluating the effectiveness of interventions.
For primary studies, we will use the Maryland Scientific Methods Scale as it is adapted by the What works Centre for Local Economic Growth (Madaleno & Waights, 2014) which ranks impact evaluations by the robustness of the methodology of the studies and the quality of implementation. The scale provides a score from 1 to 5, where 1 is the least robust (e.g., cross-sectional comparison of treated groups with untreated groups) and 5 is the most robust (e.g., randomized controlled trials). We included primary studies if the methods scored between levels 3 and 5. The included levels are defined as follows: 1. Level 3: outcomes are compared in intervention group before and after the intervention, and the comparison group is used to provide a counterfactual.
2. Level 4: the intervention and control groups vary in their exposure to the random allocation of the intervention. WELCH ET AL. | 3 of 10 3. Level 5: research designs that involve explicit randomization into intervention and control groups.
For systematic reviews, we will include studies that explicitly describe the methods used to identify studies (i.e., a search strategy), strategies for study selection (e.g., eligibility criteria and selection process) and methods of synthesis or analysis (Moher et al., 2015).
This study design, as defined by PRISMA, uses a transparent and an a priori methodology in order to ensure rigor. We will not exclude eligible systematic reviews with level 1 or 2 studies.
We will list articles and books under studies awaiting classification if they are not available electronically due to the rapid process of this map.

| Treatment of qualitative research
We do not plan to include qualitative research due to the rapid nature of the map.

| Types of settings
We will limit primary studies to high income country settings as defined by the World Bank Classifications (The World Bank, 2020).
We will not exclude systematic reviews that do not report the settings.

| Status of studies
We will include all relevant on-going and completed studies. We will not search for unpublished articles.

| Population
Our map will focus on adults who are post high school age in high income countries. We recognize that individuals may drop out from school and the age at which individuals complete high school may vary between countries and provinces/states. For example, the typical age for completing high school is 18-19 years in Europe and 17-18 years in North America. We will also include studies with a subset of eligible participants as some studies may include participants with overlapping age range, for example.
Intervention: the intervention categories of this map, as outlined below, are adapted from the NESTA (Orlik, 2018)

| Outcomes
We adapted the Common Outcomes Framework from the FSC (Future Skills Centre, 2019). The outcomes we will map are found below in Table 2.

| Search methods and sources
We will develop and pilot a search strategy (with a selection of suggested studies that met our inclusion criteria) with an information scientist (Douglas Salzwedel). This search will comprise social and economic databases including gray literature sources such as dissertations and conference abstracts databases. We will use the search strategy for on-going and completed studies in: ABI/INFORM  Table 3 for a full PsycInfo search strategy. We held a stakeholder meeting with FSC to invite suggestions of relevant studies and sources. We will not contact other organizations or individuals for information about unpublished or ongoing studies.

| Screening and selection of studies
We adhered to the Cochrane Rapid Review Guidance (Garritty et al., 2020) as this present map is a rapid EGM. We will use two reviewers for dual screening of at least 20% of abstracts, with conflict resolution. We will use single screening for the remaining abstracts and have a second screener to screen all excluded titles and resolve any conflicts. Title and abstracts will be screened based on intervention, study design (i.e., levels on the Maryland Scientific Methods Scale), population, year of publication, language, and not based on outcome. We will exclude studies published before 2000, level 1 or 2 studies and studies that are not in either English or French. We will use Rayyan for screening at this stage and utilize the program's text-mining function to calculate likelihood of inclusion (Ouzzani et al., 2016). For full-text screening, we will have dual screening for approximately 20% of studies, and have conflicts resolved by discussion or a third reviewer. The remaining studies will be screened by one reviewer, and a second reviewer will screen excluded studies. We will use EPPI-Reviewer to screen at the fulltext stage. We will also screen the reference lists of relevant studies People in work need access to relevant, high quality training opportunities that fit their circumstances and preferred approaches to learning. Interventions listed in this category cover training programs focused on literacy, numeracy and additional language training Examples: job-embedded basic skills programs, postsecondary access programs focused on literacy and/or numeracy, English as a second language training, actualisation linguistique en français Training delivery: social-emotional skills People in work need access to relevant, high quality training opportunities that fit their circumstances and preferred approaches to learning. Interventions listed in this category cover training programs focused on social-emotional skills Examples: 21st century skills, self-awareness, self-management, social awareness, relationship skills, responsible decision-making, creativity, problem-solving, interviewing skills and systematic reviews identified. We will not contact authors of studies or reviews for missing information.

| Data extraction, coding, and management
Included studies will be coded by a single reviewer and we will check for consistency and agreement on~20% of studies. Our coding categories for data extraction are based on our framework.
We will also collect details important for our stakeholders such as region (i.e., Canada, United States, Europe, East Asia and Pacific, Latin America and the Caribbean, other), setting (i.e., urban or rural), population focus for intervention (e.g., francophone, newcomers, low-income, disability or Indigenous populations), age group (i.e., 15-24, 25-54, 55-64, 65+ years), auspices (e.g., employment services, digital delivery), and sociodemographics (e.g., education, marital status). These will be included as filters in the map. We have piloted and tested the extraction form on a sample of studies and generated a draft map to discuss with our stakeholders. Refer to Table 4 for coding categories.

| Quality appraisal
We will appraise the methodological quality of systematic reviews with AMSTAR-2 (Shea et al., 2017) in duplicate for 20% of eligible studies. κ statistics will also be used to check agreement for each item. If the agreement is over 80%, we will proceed with single data extraction with verification by a second reviewer for the remainder of studies. The methodological quality of primary studies is not usually assessed in EGMs (Snilstveit et al., 2017).

| Report structure
Our EGM report will include the standard sections: executive summary, background, methods, results, and conclusion. We will indicate any changes we made between the protocol and the final report. The results section will present the number of studies retrieved from our database search and provide an overview of the types of studies by intervention, outcomes, auspices and other filters used. The conclusion will provide a detailed account of policy implications for researchers and decision-makers and highlight key areas for research commissioning.
We will include the following tables and figures: 1. Figure: Conceptual framework 2. Figure: PRISMA flow chart 3. Table: number of studies by study design 4. Table: number of studies by interventions and outcomes 5. Table: number of studies by population 6. Other tables and figures will be included based on the coded information for other filters.

| Dependency
We will treat multiple reports (e.g., secondary analyses or protocols) of a single study as one. Systematic reviews will likely include primary studies that are included in the map, and there may be more than one systematic review which may include the same primary study. Primary studies which meet our eligibility criteria will be included in the map regardless of whether they are included in a systematic review.

| The evidence gap map
The key dimensions will be interventions (presented in rows) and outcomes (presented in columns). We will use bubbles of varying sizes to present included studies. Different colors will be used for different types of studies. Final decisions about the filters will be made based on the number of included studies and coded information. The online interactive map will be hosted by FSC and Campbell.

| STAKEHOLDER ENGAGEMENT
Our central team (V. W., K. G., L. M., and C. M.) will meet bi-weekly to discuss the direction and scope of the EGM. We held a stakeholder workshop on February 26th, 2020 with our advisory group to discuss our proposed intervention and outcome framework, and eligible study designs. This group of stakeholders represented perspectives from community-based organizations as well as academics and economists who work in the skills development and training sector in Canada. The advisory group will convene again in May to discuss a draft map with preliminary results.

ROLES AND RESPONSIBILITIES
Content: L. M. M., K. and G.-M.
EGM methods: C. M. and V. W.

SOURCES OF SUPPORT
This EGM is supported by the Future Skills Centre of Canada.

DECLARATIONS OF INTEREST
Luciana Manhaes Marins and Kelly Gallagher-Mackay work at Future Skills Centre in Canada and intend to use this map to discuss priority action and research areas with stakeholders.

PRELIMINARY TIMEFRAME
Approximate date for submission of the EGM: June 2020

PLANS FOR UPDATING THE EGM
This EGM will be updated every 2 years.