The many uses of regulatory impact assessment: A meta-analysis of EU and UK cases


Claire A. Dunlop and Claudio M. Radaelli, Department of Politics, University of Exeter, Amory Building, Rennes Drive, Exeter EX4 4RJ, UK. Email:;


Research on regulation has crossed paths with the literature on policy instruments, showing that regulatory policy instruments contain cognitive and normative beliefs about policy. Thus, their usage stacks the deck in favor of one type of actor or one type of regulatory solution. In this article, we challenge the assumption that there is a predetermined relationship between ideas, regulatory policy instruments, and outcomes. We argue that different combinations of conditions lead to different outcomes, depending on how actors use the instrument. Empirically, we analyze 31 EU and UK case studies of regulatory impact assessment (RIA) – a regulatory policy instrument that has been pivotal in the so-called better regulation movement. We distinguish four main usages of RIA, that is, political, instrumental, communicative, and perfunctory. We find that in our sample instrumental usage is not so rare and that the contrast between communicative and political usages is less stark than is commonly thought. In terms of policy recommendations, our analysis suggests that there may be different paths to desirable outcomes. Policymakers should therefore explore different combinations of conditions leading to the usages they deem desirable rather than arguing for a fixed menu of variables.

1. Introduction

Contemporary capitalism is an era of policy flux where regulatory, deregulatory, and re-regulatory shifts are occurring simultaneously. New regulatory domains have emerged, such as risk, the regulation of private security companies, financial (re)-regulation, and corporate governance, as well as ambitious attempts to forge international regulatory cooperation in areas such as climate change and intellectual property rights. Deregulation was widely adopted with the aims of increasing foreign direct investment and reducing regulatory burdens on growth. Yet deregulatory failures have triggered a re-calibration of policy strategies. The emphasis has shifted from deregulation to regulatory quality – the question is not the total level of regulation, but the efficiency, accountability, consistency, and transparency of regulation. In turn, research on regulatory quality has shown that it cannot be achieved by simply clamping down on the total number of rules. It requires a proper institutional design of regulatory oversight institutions.

In response, European governments have experimented for almost a decade with regulatory reform agendas often dubbed “better regulation policies” or “smart regulation” (OECD 2002; Wiener 2006; Commission 2010). These agendas include simplification, reduction of administrative burdens, consultation, access to regulatory policy formulation, notice and comment procedures, and regulatory impact assessment (RIA – often called impact assessment, IA, in Europe). Within this reform agenda, RIA is a key policy instrument (OECD 2009; readers of this journal may refer to Cecot et al. 2008 and Peci and Sobral, 2011 for the characteristics of this instrument). Existing empirical research demonstrates the malleability of RIAs: the appraisal process is molded and shaped by policy actors to serve a variety of different purposes (Renda 2006; Cecot et al. 2008; Turnpenny et al. 2009; Radaelli 2010a,b; for specific sectors see Torriti 2010). This article builds on these findings by addressing two specific questions. How is RIA shaped during the process of its implementation? What are the combinations of conditions that lead to different usages?

We introduce the concepts and the analytical framework in Section 2. There are several theoretical angles that are commonly used to analyze RIA – including institutional analysis, diffusion, knowledge utilization, and economic theory. In this article, we draw on the literature on policy instruments and focus on the implementation stage, looking at how constellations of actors shape the usages of RIA. Section 3 presents the research questions, expectations, data, and methods. Specifically, we identify four types of usages of RIA: political, instrumental, communicative, and perfunctory; present 31 case studies for meta-analysis; and explain sample construction and measurement. Qualitative comparative analysis (QCA) is used to explore the different combinations of conditions that lead to different usages of RIAs. Our use of QCA has a deductive element in that the choice of conditions or variables is theoretically informed. However, our interest is not in testing individual theories. Since QCA allows researchers to explore a limited number of cases in a configurational way, engaging in a “dialogue between cases and relevant theories” (Rihoux & Ragin 2008, p. 6), we are interested in examining how combinations of elements of the appraisal process inform RIA usage. Section 4 presents the univariate examination of the 31 cases, whilst in Section 5 we move to QCA to explore the different configurational paths that can lead to different types of RIA. Section 6 briefly concludes and reflects on the conceptual, methodological, and normative implications of the findings.

2. Concepts and framework

The standard mode of analysis of RIA focuses on how “good” the assessment is, using objective and subjective indicators to check whether the RIAs carried out by a given department or agency stand up to either formal guidance documents or more theoretical benchmarks, often drawn from applied cost–benefit analysis (Hahn & Litan 2005; Renda 2006; Cecot et al. 2008). Recently, however, political scientists have introduced different, theory-based perspectives that seek to explain how RIAs actually work (Turnpenny et al. 2009 provide a systematic overview of existing studies), such as diffusion theory (De Francesco 2012), knowledge utilization (Schrefler 2010), the political control of bureaucracies (Radaelli 2010a), institutional analysis (Sager & Rissi 2011), new institutional economics (Dunlop 2010), and decision theory (Nilsson et al. 2008).

In this article, we also seek to explore how RIAs work but, theoretically, we are concerned with the relationship between ideas and policy instruments (Schneider & Ingram 1993; Braun & Busch 1999; Salamon 2002; Kassim & Le Galès 2010). Policy instruments – including policy appraisal tools – can be analyzed by looking at how their usage is steered by the “theories” they contain (Salamon 2002; Hood & Margetts 2006; Lascoumes & Le Galès 2007; Turnpenny et al. 2009). Instruments are carriers of ideas, theories, and world views – such as the belief that monetization is a convenient way to measure the value of the environment and life (Kysar 2010). The key proposition, therefore, is that instruments embody normative and cognitive ideas about public policy. By carrying ideas, often implicitly, instruments bias the policy process. They tilt the scale or stack the deck in favor of certain actors and bring some ideational components of public policy to bear on policy outcomes, whilst silencing others. This corresponds with the findings of authors inspired by delegation theories. They explain administrative requirements, such as evidentiary standards and public disclosure obligations (of which RIA is an obvious incarnation), in terms of how they reduce information asymmetry and ultimately favor the principal (McCubbins et al. 1987). We can therefore predict that its usage will lead to certain outcomes in terms of power and policy.

There is a problem with this set of propositions, however. When we move from the adoption to implementation of RIA, and examine its usages, we typically find that the ideas, beliefs, and theories embodied in this policy instrument are ambiguous and malleable. In consequence, we have to draw on implementation analysis (for a review see O'Toole 2000), acknowledge ideational ambiguity up front, and examine the many usages of policy instruments. This is indispensable if we are to make theoretical progress on how instruments for policy appraisal are used on the ground. One important strand of implementation analysis has indeed argued that constellations of actors engage in evolution rather than execution of policy (Majone & Wildavsky 1978), bring communicative interaction to bear on their interactions (Grin & van de Graaf 1996), and reconvene to frame the policy issues (Rein 2006). Perhaps they even erode, cheat, and ultimately reshape the regulatory space (Richardson 1996; Thatcher & Coen 2008). Transaction-cost theories of politics come to the same conclusion: actors' constellations adapt incomplete contracts when implementing them. Such adaptation leads authors such as Dixit to treat “policymaking as a process that goes on in ‘real time’ and constantly combines some features of rulemaking and some of individual acts” (1996, p. 29).

Consequently, we relax the chain-of-command assumption that an instrument carries an unambiguous set of ideas and because of this steers usage in one direction or another. Instead, we consider ideational ambiguity at the implementation stage. The consequences are clear. Under conditions of ideational ambiguity, policy instruments are shaped by the constellations of actors at the implementation stage. Specifically, we identify conditions for different types of contingent framing in RIA use. We also explore the consequences of ideational ambiguity for policy instrumentation. Thus, we contribute to the ideational strand on ambiguity and strategic manipulation of ideas and knowledge (Jabko 2006; Rein 2006; Daviter 2007; Zahariadis 2008; Schrefler 2010).

In terms of policy recommendations emerging from this type of analysis, we take issue with the normative approach suggested by international organizations, audit institutions, and think thanks (Renda 2006; OECD 2008; NAO 2006, 2007, 2009). This approach recommends a single recipe for a successful implementation of RIA, revolving around “strong” central units, appraisal processes that start “early” enough to carry out economic analysis, intense consultation with the stakeholders, and a separation of the “technical” from the “political.” We do not say that all this is necessarily wrong. Rather, we stress equifinality: there may be different combinations of the key elements of the appraisal process that lead to the same usage. This ties in with our methodological choice for qualitative comparative analysis (QCA), as will be demonstrated.

3. Research questions, methods, and data

Having entered our framework based on policy instruments and implementation, we turn to our prior expectations and the research questions, before we move to methods and data. Our framework lends itself quite naturally to two main research questions. R1: how is RIA shaped and used at the stage of implementation? R2: what are the combinations of conditions that lead to one type of usage instead of another? Finally, the third research question is on the conceptual–theoretical implications of the findings. R3: what are the wider implications of our analysis in terms of the interplay of (ambiguous) ideas, framing, and policy shaping?

To address these questions, we need to develop ex-ante expectations about a finite number of ways in which RIA can be shaped at the stage of implementation. Unfortunately, there is no systematic literature that predicts how RIA will be molded by policy actors. However, two recent studies provide insights on this aspect. In his comparative analysis of European and North-American countries, Radaelli (2010b) argues that institutional variables (such as the position and role of the executive and presence of a minister who can call the shots for change, but also softer institutional variables such as “market for ideas and advice” in government) predict whether a country will steer RIA toward one of the following usages: control of the bureaucracy, instrumental learning, administrative reform, and perfunctory usages. This is a macro-approach that gives us an idea of how RIA may be shaped and reframed, but it does not really explain the level of implementation and why, even within the same country, some RIAs are less perfunctory in one sector than in another. Meuwese (2008), instead, works on four case studies of RIA in the European Union (EU) – thus, one jurisdiction, but with four cases. She finds five distinct meanings or ways of “framing” appraisal in the EU: (i) to speak the truth to power; (ii) to use RIA to highlight trade-offs in lawmaking; (iii) to provide a forum for the input of a wide range of stakeholders; (iv) to give reasons for legislative decisions (a notion close to the US Administrative Procedure Act requirement); and (v) to structure stakeholder deliberation and discourse.

We cannot import wholesale Meuwese's categorization – and even less so Radaelli's. Meuwese's framework is designed to capture the nature of organizational relations within the Commission, on the one hand, and between the Commission, the European Parliament, and the Council, on the other. It does not tell us when to expect one type of RIA or another. Radaelli's categories lead to the expectation that there is a median RIA with certain characteristics in a given country, hence it not suitable to examine variation within a single country. However, both studies shed light on a finite number of possible usages.

We can therefore combine the insights of these two studies and relate them to broader theoretical issues introduced in Section 2. The result is four possible ways of framing and shaping RIAs:

  • 1Political usage. Both delegation theorists (McCubbins et al. 1987) and RIA specialists (Radaelli 2010a) argue that RIA is shaped by the principal to control the agent. However, this conception of political is grounded in US administrative law. Since our evidence is European (specifically UK and EU, see below) we need to broaden out this concept. First, political usages should also capture the desire of the Member States to tame the regulatory activity of the European Commission. This way “political control of the bureaucracy” can be extended to the EU, where there is no unitary executive like in the US. Second, political usages cover instances in which the process of appraisal is not based on the ideals of evidence-based policymaking, but is used by affected social, economic, and political interests to provoke or handle conflict, for example with social partners or the opposition in parliament. We should therefore expect attempts to delegitimize the process or to bring explicit political conflict within the economic analysis of proposed regulation. Our interest here is in the political usage of RIA rather than the nature of the outcomes of this usage.
  • 2Instrumental usage. The appraisal process can be used to enhance substantive understandings of the cause and effect mechanisms that underpin the policy issue. This second type of usage arises out of the long search for rationality in the policy sciences (Carley 1980; Moran 2003). Recently, the movement for evidence-based policy has swept across Europe, and studies of RIA have highlighted the increased importance of analyzing and cataloguing the costs and benefits of regulatory proposals where truth speaks to power (Meuwese 2008).
  • 3Communicative usage. Best practice prescriptions of RIA usage stipulate that RIA be published as part of the formal consultation process to provide consultees with information on the impact of the policy proposal (OECD 2009). Thus in a way all RIAs should have a communicative element. However, we were looking for the use of RIA to shape interactions with stakeholders in a manner beyond formal consultation. Here RIA becomes a venue where policy actors communicate and conduct regulatory conversations. Essentially we combine Meuwese's concept of impact assessment processes as forums for dialogic encounters between regulators and stakeholders and as a mechanism that allows deliberation and structures stakeholder discourse. Again, our interest is in the use of RIA as a communicative arena rather than in the outcomes of these dialogues. Organizations such as the European Commission have sought to respond to their legitimacy deficit by embracing notions of open governance, participation, and even deliberation. The White Paper on Governance of 2001 is the best example of this aim (Commission 2001). The 2002 European Commission's better regulation agenda bears several traces of these notions of participatory–deliberative governance (Allio 2009). As mentioned, Meuwese found a forum-type of usage in her case studies, whilst Radaelli (2010b) does not consider this aspect. Scott (2010) argues convincingly that this type has to be considered because of its important properties for the reflexivity of law and regulatory conversations (Black 2002).
  • 4Perfunctory usage. Finally, although Meuwese (2008) does not consider this type, there is evidence (Radaelli 2010b) that governments adopt RIA but then constellations of actors water down, “mute”, or simply do not implement the instrument. Such behavior can be explained by organizational theory, which points toward decoupling between rationalistic tools and pragmatic policy formulation in loosely coupled organizations (Weick 1976).

Our first expectation is that we will find different types of RIA even within a single case; these four common usage types are not mutually exclusive (as illustrated by Radaelli 2010b). Current research suggests that instrumental and communicative usages are rare (Jacob et al. 2008; Nilsson et al. 2008; Hertin et al. 2009; Turnpenny et al. 2009). In contrast, political and/or perfunctory usages are widespread (Wiener 2006; Jacob et al. 2008; Nilsson et al. 2008; Hertin et al. 2009; Russel & Turnpenny 2010; Turnpenny et al. 2009; Scott 2010; NAO 2006, 2007, 2009). Thus, our second expectation is to find a distribution skewed toward political and perfunctory RIAs.

Another expectation is that the way in which an RIA is shaped depends on how actor constellations, so to speak, “sit down” and negotiate ideational ambiguity, resources, and usages of the instrument they handle. It is difficult to specify conditions ex-ante but both the evaluations of the United Kingdom's NAO (2006, 2007, 2009) and the academic literature cited in the previous paragraph suggest the following specific expectations: (i) RIAs that explicitly analyze trade-offs and distributive effects increase the probability of political intervention; (ii) availability of time, human resources, and expertise are preconditions for bureaucratic learning; and (iii) balanced consultation and transparency facilitate communicative regulation, but stakeholders that expand conflict within the process of appraisal and outside (via direct lobbying and appeals to public opinion), tilt RIA toward political usages. Finally, we have to model expectations about the role of the central regulatory oversight unit, such as the Better Regulation Executive and the Secretariat General of the Commission – we do not consider the EU Impact Assessment Board because it did not exist when the case studies we use in our analysis were produced. This role varies. On the one hand, the oversight body can provide assistance to the officers that work out the RIA in their department, on the other they can exercise political control to align the final proposal with the preferences of the principal (Radaelli 2010a).

To answer our research questions and check on expectations, we employ meta-analysis of existing RIA case studies, both with univariate and QCA techniques. Meta-analysis of case studies is a typical way to extract cumulative findings from the “gold mine” of individual cases, as shown by Jensen and Rodgers (2001), with different options in terms of how to handle the data. We meta-analyzed the case studies by scoring them on the basis of a list of variables and then coded them in a 0–1 format to make them amenable to QCA (see Table 2). Thus, to conclude that a RIA is of one type and/or another, we check for a coherent pattern across the variables identified in the case study. We classified the RIA as belonging to more than one category when indicators pointed toward different usages. This is a standard procedure when we move from highly abstract modes of use to individual cases: each case can contain empirical elements that refer to more than one usage (see Radaelli 2010b on the multi-purpose nature of RIA).

Table 2.  Codebook: list of variables, indicators, values
Variable nameIndicatorsValues
Missing value = 99
IDID number of the case study1–31
YearYear in which the RIA was carried outYear
LeaddepartmentThe department responsible for the RIADepartment/DG
PolsectorSubstantive policy areaPolicy sector
IssvisIssue was already visible in the media prior to RIAHigh = 1
Not high = 0
PolnovNew, emerging sector or traditional policy sectorNovel = 1
Not novel = 0
TimingEvidence about the RIA starting earlyEarly (at start) = 1
Not early = 0
IsscomplComplexity-tractability of the problem: can policymakers rely on existing measures of the problems, models or at least widely shared heuristics?Complex = 1
Not complex = 0
AffectActors affected by the RIAIndustry or industry + other(s) = 1
Other(s) = 0
DistribThe issue at the core of IA has distributional effectsHigh = 1
Not high = 0
BureauBureaucratic actors involved – from different departments, senior officers, or also junior officers, inter-service teamsName of actors
CentraluRole of central oversight unit. Does the central unit provide scrutiny and checks on the analysis produced by the regulators or provide a helping hand to the regulator, for example assisting in the preparation of impact assessment [1]? Is its role irrelevant or totally absent in this case [0]?Quality assurance or providing help = 1
Small / no role = 0
OutexpInformation on use of external expertise and consultants in the RIA processYes = 1
No = 0
ElectgovEvidence about the involvement of political actorsYes = 1
No = 0
ParlroleEvidence about the involvement of the Parliament in using the RIAYes = 1
No = 0
ConsultwhoActors consultedIndustry or industry + other(s) = 1
Other(s) = 0
ConsultnumberNumber of actors consulted1–N
InformconsInvolvement of social actors on policy formulation outside the RIAYes = 1
No = 0
BalancOverall balance of consultationBalanced = 1
Skewed = 0
TranspTransparency, including publication of RIA and documentsHigh = 1
Not high = 0
DialresponRIA shows how the issues raised in consultation were addressed by policy officersYes = 1
No = 0
TradeofAnalysis of trade-offsYes = 1
No = 0
CounterriaAppraisals produced to counter the official RIA or to influence it during the appraisal processYes = 1
No = 0
De-legprocEvidence of attempt to delegitimize the RIA processYes = 1
No = 0
OptrangeNumber of options considered1–N
DonowtEvidence that the do-nothing option and the status quo were considered and properly appraisedYes = 1
No = 0
UncertAnalysis of uncertainty (confidence intervals, sensitivity analysis)Yes = 1
No = 0
Interg-diffConsideration of different dimensions and integration of different perspectivesYes = 1
No = 0
TimedimShort-term or long-term analysisYes long-term = 1
No, short-term = 0
ResouEvidence of resource constraintsYes severe = 1
No = 0
InstrumentalEvidence that the RIA process is used to enhance substantive understandings of the cause and effects mechanisms that underpin the issueYes = 1
No = 0
PolusageEvidence that the RIA is being used to exercise control of the bureaucracy and / or is used by affected stakeholders to provoke or handle conflictYes = 1
No = 0
SummativeSummative statement on the type of usages of RIA; key quotations from the case studiesSentence

Turning to sample construction, we considered existing case studies of RIA. By “RIA case study” we mean a scholarly or governmental report on a specific process of appraisal with a narrative/analytical structure. We included only studies based on the case study authors' primary interviews and the analysis of available documentation, including published RIAs and relevant material surrounding them. For this reason, we did not include the large amount of data made available by scorecards of RIAs in Europe (Renda 2006; Cecot et al. 2008). There is quite a bit of information in these scorecards. Yet we cannot simply aggregate scorecard analysis with case studies, since the two sources of information differ in a radical way. To be clear, in this study an individual “case” refers to a single qualitative case study, not an individual RIA. As well as allowing us to analyze and collate detailed qualitative accounts, the meta-analytic approach also reduces the distance between our sample size and the universe of cases. While there are around 100 EU RIAs and 180 UK RIAs conducted each year,1 there are only 49 case studies. Here, we examine 31 of them.2

We found case studies in projects funded by the EU, such as a comprehensive evaluation of the Commissions' RIA (The Evaluation Partnership (TEP) 2007), EVIA (Evaluating Impact Assessment), and MATISSE (Methods and Tools for Integrated Sustainability Assessment). Reportedly, the European Court of Auditors carried out cases studies in the preparation of their 2010 report (ECA 2010), but they did not release them because they are considered internal preparatory material.

In this article, we focus on case studies based on the UK and the EU, since these are the most developed and most comparable systems of RIA. To the best of our knowledge, there are also a few scattered case studies on Denmark, the Netherlands, Sweden, Poland, and one on Italy, but they refer to systems of impact assessment that vary markedly (e.g. up until recently, Denmark and the Netherlands used impact assessment only to measure a special category of costs, that is, costs originating from administrative obligations). Only in the UK and the EU is the scope of RIA broad enough to cover a large number of possible negative and positive impacts on a wide range of stakeholders. In other European countries, the scope is narrowed to cost assessment or even narrower, that is, the measurement of administrative burdens arising out of proposed regulation.

This set of criteria led us to reject 18 case studies, leaving us to the 31 cases listed in Table 1. They were scored by the authors according to the codebook described in Table 2. We decided not to examine cases for which there was a high number of missing values – this is especially pertinent in some MATISSE cases. Our sample of case studies contains all the cases we could find (minus those with too many missing values, as explained above). It is not, however, a sample of RIA: in relation to the total production of RIAs in the UK and the EU, 31 is a very small number especially when compared to our aforementioned universe of actual impact assessments. But this is precisely the point: we are concerned with a sample of case studies, not with a sample of impact assessments.

Table 1.  Sample of case studies
CaseIDYearLead dept (jurisdiction)Project
Biomass action plan – communication on bioenergy policy12005TREN (EU)EVIA
Working time – amendment to a directive regulating employee working time22004EMPL (EU)EVIA
Batteries – directive on the disposal of batteries32004ENV (EU)EVIA
Sugar – reducing subsidies to sugar producers to liberate markets42005AGRI (EU)EVIA
Operating and financial review of directors – policy to extend company reporting requirements52005DTI (UK)EVIA
Railway interoperability – implementation of EU policy to promote competition and improve efficiency in the rail sector62006Dept of Trans (UK)EVIA
Solidarity – migration and the integrated management of the EU's borders72005JLS (EU)TEP
Gender equality – recasting of six directives to improve the clarity and transparency in EU gender legislation82004EMPL (EU)MATISSE
Pre-packed products – directive on the size of packaging of products92004ENTR (EU)TEP
INFSO – communication on the information society for growth and employment102005INFSO (EU)TEP
Postal service – proposed directive on internal postal service112006MARKT (EU)TEP
Maritime transport – EC rules at EC level for the technical investigation of maritime accidents122005TREN (EU)EVIA
Working family – extension to maternity and paternity leave132005DTI (UK)EVIA
Disability – access to public transport for people with disabilities142005DFT (UK)EVIA
Working time transport – regulating drivers' hours commercial road transport152005DFT (UK)MATISSE
Offshore – policy is aimed at reducing oil discharge from offshore drilling platforms162005DTI (UK)MATISSE
Climate change – assessment of options for addressing climate change in Europe post-2012172005ENV (EU)MATISSE
Legal protection of design – regulation of the secondary market (repair and replacement) in automotive sector182004MARKT (EU)MATISSE
Groundwater protection – directive to help better protect groundwater sources from pollution192003ENV (EU)MATISSE
Timber imports – proposal concerning the establishment of a voluntary licensing scheme for imports of timber into the European Community202004ENV (EU)MATISSE
Air pollution – thematic strategy on air pollution212005ENV (EU)TEP
Plant protection – directive regulating products for plant protection222006SANCO (EU)TEP
Credit union interests – policy to improve access to affordable credit by increasing the market share of credit unions232006HTM (UK)EVIA
National lottery bill – policy on the allocation of lottery funds242005DCMS (UK)MATISSE
Prescriptions – regulation of pharmaceutical services252005DOH (UK)MATISSE
Landfill – policy for Implementation of the EU directive on reducing waste to landfill262004DEFRA (UK)MATISSE
Kyoto – policy on linking Kyoto Protocol project credits to the European Carbon Emissions Trading Scheme272005DEFRA (UK)MATISSE
Home office – policy on the policing of UK borders282006HO (UK)MATISSE
Euromed – policy toward a liberalized trade in the Mediterranean292005RELEX (EU)MATISSE
Environment and health – plan for preventive action on environmental sources of health impacts302004ENV (EU)MATISSE
Equivalent access to law – policy to improve level of cross-border information exchange to a level adequate for law enforcement cooperation312005JLS (EU)MATISSE
Total31N/AEU n= 19MATISSE n= 15
UK n= 12EVIA n= 10
TEP n= 6

We acknowledge that our case studies vary in terms of depth, the number of interviews conducted, type of documents examined, and more generally the resources invested in the preparation of the report. Moreover, research questions vary by project. This creates different sources of bias when the findings are aggregated. To reduce this bias, we generated an analytical template. Table 2 (i.e. the codebook) lists the variables. This analytical template was first piloted on four RIAs scored independently by each author to check inter-coder reliability, then discussed and adjusted, before we proceeded to the full analysis. Reliability of the full analysis was then checked through meetings and discussion between the authors. The categorizations for the RIA usage for each case study were agreed by the team after having read the case studies.

4. Identifying RIA usage through univariate analysis

We found perfunctory (7 cases for the UK, 10 for the EU), political (4 UK, 9 EU), instrumental (6 UK, 6 EU, plus 1 EU case with weak learning effects) and communicative RIA usages (0 UK, 5 EU) as summarized in Table 3. The total exceeds 31 because, as expected, there are multi-purpose usages. The lack of communicative cases in the UK seems to confirm the trend toward centralized control, in line with previous research (Radaelli 2010a).

Table 3.  Summary of univariate analysis
Political usage9413
Instrumental usage7 (including weak case)613
Communicative usage505
Perfunctory usage9716

4.1. Political usages

ID 1, 2, 3, 5, 9, 17, 18, 19, 24, 25, 28, 29, 30

These cases provide evidence of political control, as well as political interest in using the findings to inform the strategic behavior of elected policymakers. Political usage also means the attempt to shape appraisal by injecting conflict inside and around the process of preparing and finalizing the RIA. In classic Schattschneider (1960) fashion, actors that are to incur losses expand the scope of conflict and seek to frame more politically the function and scope of appraisal. We should be clear, the success or otherwise of these political activities is not the matter at stake here. Rather, we are simply interested in the political usage of RIA rather than the outcome.

To illustrate, case ID 18 shows that the appraisal served to magnify the conflicts that already existed in the policy sector and, importantly, provided a renegotiation of the regulatory space for the consumer lobby on an issue which had hitherto been dominated by the car industry. The proposal for extended company reporting requirements in the form of an Operating and Financial Review (OFR; ID 5) was first raised by an independent steering group in the context of formal review of company law. OFR was adopted in early 2005. It was due to come into force in 2006. But the government announced the decision to revoke this measure to showcase its commitment to cut “unnecessary red tape.” Friends of the Earth challenged the decision through a judicial review procedure on the basis of a “breach of legitimate expectation.” They argued that only selected business groups were consulted on the withdrawal of the OFR. Instead – the Friends of the Earth argued – the RIA behind the 2005 OFR had shown that a wider range of stakeholders was affected. In consequence, they should have been consulted before announcing the decision to abandon the OFR.

Other cases show that the process of appraisal is used to negotiate attention and priority for one instrument or another – which is also a way of negotiating political priorities around regulatory reform. Pre-packed products in the EU (ID 9) are instead about the tension created by ideational ambiguity about the overall direction of regulatory policy. At the outset, pre-packed products were supposed to be a showcase of the revamped better regulation strategy of the Barroso Commission in 2005, pro-business and deregulatory. The overall nature of this measure induced the lead Directorate General to come up with a short RIA – since it was felt that there was very little to justify in a measure leading to freer markets. Yet the whole case became highly problematic when the European Parliament started to politicize the discussion and introduce issues concerning the protection of consumers. Meuwese (2008, p. 253) explains that

For that very reason (its flavor of deregulation) this dossier was selected by the rapporteur in the EP to be the first parliamentary RIA. (. . .) The EP accused the Commission of selective consultation and of ignoring the social impacts of the proposal on weak consumers. The European Commission found the parliamentary RIA utterly un-convincing and went on to propose (and secure) liberalization of pack sizes in even more sectors than originally envisaged.

We also observed more deliberate politicization, in the sense that the appraisal process was geared toward the policy options favored by ministers or commissioners (e.g. cases ID 24, 25, 28, 29). For instance, with the UK lottery case (ID 24) the RIA was conducted with the sole purpose of getting ministerial clearance, rather than learning about alternative options. In case ID 25 the chosen option was that “prescribed by the minister.” With the EUROMED case (ID 29) the final decision was politically driven, not evidence driven.

Overall, nine cases provide evidence of involvement of the central unit beyond technical assistance and providing help. Ten cases show involvement of political actors in the RIA. Finally, there are four cases of political RIAs that have been openly questioned by stakeholders who have produced counter-impact analyses.

4.2. Instrumental usages

ID 1 (weak effects), 3, 4, 11, 12, 13, 14, 15, 20, 21, 25, 26, 28

We expected to find a low number of RIAs oriented toward instrumental usage. One reason for this expectation is the frequent observation that RIAs start late in the policy process and are constrained in terms of time and resources for economic analysis (NAO 2006, 2007, 2009). However, in our sample, 23 cases did not report severe constraints. This perhaps explains why we have found several RIAs whose uses were instrumental.

Constellations of actors implement RIA requirements to learn how to use economic analysis or to structure their relations with other departments at the stage of formulation of cross-cutting proposals. RIA is therefore reframed and negotiated to find out the legitimate boundary and autonomy of department A, in relation to departments B, C, and the central oversight unit. At the level of the Commission, there is considerable appreciation for RIA as a tool that has made the dialogue between one lead Directorate General (DG) and the others relatively smooth and evidence-based. Case ID 1 is mostly perfunctory, but there is modest instrumental usage in that the DG in charge learned how to calibrate the choice of regulatory instrument and to perform multidisciplinary analysis. Another of the Commission's RIAs (case ID 3) portrays some limited instrumental usage, in that interservice consultation led to amendments to the assessment.

Classic instances of instrumental use via economic analysis are case ID 26 (where appraisal was done “as part of the policy development process to shift through ideas and work through options”) and ID 13 (RIA had an important role in informing the detailed policy design and influencing significant decisions). One explanation is that the RIA process started early enough to allow time for analysis. Another is that all the Commission's RIAs we examined belong to an early phase of development of EU impact assessment: several officers at the Commission wanted to showcase their initial work on RIA as best-practice and to dampen suspicions about their capacity for evidence-based appraisal.

The use of external experts is systematic, both in the UK and the Commission: in total, we found 17 cases in which experts contributed to appraisal, presumably improving on the knowledge base available to the officers. Sometimes the experts become quite internal to the process and can be effectively considered part of the team developing the RIA within the bureaucracy.

4.3. Communicative usages

ID 2, 11, 18, 20, 21

Having read the case studies and discussed them in our team, we concluded that five RIAs were communicative in the broader sense that we defined earlier. In case ID 2, the actors placed the negotiation of priorities and transparency above everything else. Case ID 21 portrays a communicative process in terms of balancing arguments and analysis. In ID 18, the consultation process was conflictual. On the one hand it made the lines of conflict clearer and therefore more manageable. On the other, it allowed consumer associations to actually take part in appraisal processes whilst at the beginning the only actor involved was industry. The other cases show a wide range of communication beyond formal consultation from public events to demonstration workshops, and forums targeting specially identified audiences. Thus, consultation is not a single event, but a process.

4.4. Perfunctory usages

ID 1, 2, 6, 7, 8, 10, 15, 16, 22, 23, 24, 26, 27, 29, 30, 31

There are several cases that strongly suggest a perfunctory engagement with RIA. Cases ID 16 23, 26 and 30 imply that RIA was a box–ticking exercise to comply with formal requirements. The RIA in case ID 2 was perfunctory; “most significant elements of it would have been addressed . . . in the absence of a formal IA. The main purpose of the IA was to build acceptance within the social partners” (EVIA 2007, p. 61). The analysis of cases ID 23 and ID 31 was more explicit in suggesting that the IA was part of a rubber-stamping exercise where this process was used to add a veneer of evidence and data to a policy deal which had already been done.

It would be wrong to assume that the perfunctory aspects of RIA are always deliberate choices to dress up prefabricated positions. In some cases, there is genuine difficulty of handling the kind of rational process described in the guidelines when it is clear to all those who are involved that the end of the story has already been written – for example at the domestic level when the RIA is about the implementation of a directive of the EU.

Other cases highlight the difficult synchronization between the timing of impact assessment and the actual formulation of policy proposals. Although developed at the same time, the proposal in ID 7 was not the result of deliberations based on RIA. Instead, and contrary to the logic of appraisal, the two documents were fine-tuned and harmonized to make sure they would not contradict each other. Yet another group of cases show that the involvement of experts can become part of the perfunctory element of RIA when experts come from narrow ranges of expertise and do not engage with analysis, but rather with imaginative ideas about the presentation of the proposed rule.

5. Exploring the interplay of variables using Qualitative Comparative Analysis (QCA)

To go beyond univariate analysis and look at the interplay between contexts and policy actors, we explored the conditions for different RIA usages by examining our data with dedicated Qualitative Comparative Analysis (fs/QCA) software.3 The two core analytical intuitions in this approach are configurational analysis and equifinality (Ragin 1987). Configurational analysis points to the joint effect of conditions. Thus, crisp–set QCA draws on the Boolean algebra's logical operations (AND/OR) to identify combinations of conditions, rather than viewing cross-case patterns through the classic lenses of the “net” effect of independent variables on dependent variables. Equifinality means that there may be multiple configurational paths – or “recipes” (Ragin 2008) – leading to the same outcome. Of course, this does not mean that there are an infinite number of recipes. Indeed, with QCA, researchers use their theoretical expectations to explore a finite number of paths (Ragin 1987, 2008). Conditions can be necessary and/or (jointly) sufficient for the outcome. A necessary condition is a condition that must be present in order to observe the outcome, while a sufficient condition is a condition that, if observed, guarantees the presence of the outcome. The analysis of necessary and sufficient conditions is based on set-theoretic relations, meaning that they are assessed by comparing the membership scores of each case in the conditions, and, respectively, in the outcome. Fuzzy set QCAs allow for degrees of set membership in the interval between 0 and 1. Our data can be coded as 0 (absence) and 1 (presence); hence we use crisp-set QCA (csQCA) (Rihoux & Ragin 2008).

The goal of our application of QCA is not to test one single hypothetical-deductive model, but it is more tentative and heuristic: to explore whether the usages of RIA are related with different, coherent configurations of variables, and to discover “unexpected” recipes (for a similar approach see Jackson 2005). Thus, by using QCA our aim is not to explain the impact of any single independent variable, but rather to illustrate the complex configurational paths leading to the different outcomes (Grofman & Schneider 2009). As mentioned, one important function of QCA is to establish conditions that jointly lead to a given outcome. This is important because the prescriptions of international organizations tend to stress the same combinations. QCA can appraise these normative propositions as well as finding out cases of equifinality, for example different combinations of variables leading to the outcome “instrumental usage.” Thus, we use QCA to widen our peripheral vision to explore combinations that have not been considered yet.

We present the analysis of sufficient conditions, using the standard procedure for crisp-set “truth tables” implemented in the fs/QCA software.4 In line with our research goals, and given the presence of missing data on some conditions, our interpretation of “sufficiency” is restricted to the exploration of different configurational paths leading to each outcome, without any strong claim of causality, and with limited generalizability.5 Furthermore, to deal with these missing values, consistency and coverage scores have been recomputed, recipe by recipe, by using the results of prior truth table analyses.6 This procedure allowed us to check the robustness of our analyses by maximizing the number of cases included in each recipe at a time. QCA results reported in Table 4 relate to intermediate solutions that are robust after recomputation; if the re-analysis produced more complex, contradictory results, the recomputed recipes and scores are not reported as it is considered inconclusive.7 As shown in Table 4, six models were analyzed, following different analytical goals about RIA usages. To explore the combinations of conditions, we used the dichotomized 0–1 variables in Table 2 (the model labels elaborated in Table 2).8 In the following QCA tables, when the sign ∼ precedes a variable, it means that that condition is “absent”: so for example ∼ tradeof means “absence of trade-offs.”

Table 4.  Summary of QCA analysis
NGoalModelSolution (intermediate)CoverageConsistency
1Paths to intervention of the regulatory oversight bodyCentralu = f(tradeof, electgov, distrib, affect, polnov, isscompl)∼tradeof*∼distrib0.710.91
2Paths to political usagesPolusage = f(informcons, distrib, consultwho, affect, timing)informcons*∼affect0.131
3A communicative view of political usagesPolusage = f(dialrespon, balanc, polnov, issvis, affect)dialrespon*∼balanc*affect0.431
4Instrumental usageInstrumental = f(tradeof, outexp, timing, polusage, centralu, electgov)outexp*timing0.431
5Communicative responses to consultationDialrespon = f(polnov, affect, issvis, timing, electgov, resou)Inconclusive  
6Combinations leading to transparencyTransp = f(informcons, electgov, resou, centralu, balanc, affect, timing)resou*affect0.331

To begin with, we had to determine the role of the Better Regulation Executive and the Secretariat General of the Commission – the regulatory oversight units (model 1). We took “centralu” from the list in Table 1, with a value of 0 for no active role of the oversight unit, 1 elsewhere. To explore conditions for an active role of the central unit, we followed our expectations and examined the presence or absence of the following: (i) the analysis of trade-offs, an indicator of genuine complexity in the assessment; (ii) the role of elected officers; (iii) distributive effects; (iv) the “affect” variable (1 when industry and/or citizens are affected, 0 when other departments are the main entities affected); (v) the novelty of the issue; and (vi) its complexity (all variables are defined in Table 2).

The crisp-set intermediate solution for sufficient conditions of the outcome “active role of the central unit” is composed of two consistent paths.9 The more important path defies our prior expectation. It revolves around a combination of lack of distributive effects and lack of trade-offs. A plausible explanation is that the oversight unit in this path prefers to avoid meddling with RIA when there are distributive effects and trade-offs, perhaps with the intention of staying out of conflicts with the stakeholders. This is reflected in the case studies with “low politics” characteristics, where conflict was low and stakeholder cooperation and consensus was high. And so the absence of conflict may have contributed to the availability of space for the central actor to assume a key role.

The second path to “active role of the central unit” is characterized by explicit trade-offs in the process of appraisal, some degrees of political attention, distributive effects, the role of firms affected by the proposed regulation, and complexity of the issue. After having recomputed the consistency and coverage scores by maximizing the number of cases included in the analysis both results appear unaffected by missing values. The second recipe displays, however, poor coverage so that it seems highly contingent on specific cases.

Another way of looking directly at political usages contemplates: elected policymakers learning how to manage policy formulation, how to utilize the RIA for political decision-making, and how to make the agencies and departments responsive to their principals. We therefore examined a model with the following variables on the right-hand side: role of stakeholders in the process beyond consultation, distributive effects, types of actors that were consulted, type of actors that were affected, and timing of the appraisal process (model 2 in Table 4). We had a prior expectation that political usage is higher when stakeholders are active outside the perimeter of consultation, for example with media campaigns or direct lobbying. Principal–agent theory, in fact, provides the expectation that pressure groups affected by proposed regulation ring the fire alarm and alert the principal. As regards this solution, the involvement of industry in consultation seems to play a role in combination with other factors. For example, in the case of the UK RIA on extending company reporting requirements (case ID 5) it was an NGO appeal against the government's plans to relax reporting procedures that triggered early industry intervention and politicized the appraisal process. Similarly, the RIA concerning the organization of working time at the EU level suggests that politicized and “heavy negotiations” (EVIA 2007, p. 58) was largely the result of the dominance of social partners in the appraisal. The two combinations composing the overall solution show the important role played by the involvement of actors outside formal consultation, even if the RIA did not start early. This finding can be regarded as robust because the condition “informcons” is stably present in the recipes after having recomputed consistency and coverage by considering one recipe at a time.

Political usages can also be examined in a different framework that blurs our initial distinction between political control and communicative RIAs. One can reason that RIA can be used to initiate communicative responses that are politically profitable. To check this argument, which runs counter to what we said earlier about RIA types, we considered model 3 in Table 4. The intermediate solution is supported by the robustness check for missing values. It does provide some leverage for the counter-argument we are considering, since communicative aspects of RIA and the nature of stakeholders affected by the proposals are present in both sufficient conditions – one time with high issue visibility and another with skewed consultation. “Skewed consultation” suggests that political control is associated with RIA as a fire alarm for the constituencies that matter to the principal – hence RIA is politically useful exactly because it stacks the deck favoring some stakeholders over others.

Let us turn to instrumental usages. Different combinations of variables lead to this outcome. We expected political use, active intervention of the central unit, and the presence of elected policymakers in the process of appraisal to hinder evidence-based usage. This prior expectation, however, may not be necessarily correct: the central unit can help departments and promote instrumental usage. Political attention may increase the motivation and rewards associated with engagement with evidence-based usage. Elected policymakers may also be beneficial to the process of appraisal, since they can direct the economic analysis toward conclusions that are easier to utilize in decision-making processes.

We explored this somewhat ambiguous prior expectation with model 4 in Table 4. The model shows a combination of two conditions that are particularly relevant. One is an early start of the appraisal process (variable “timing”) and the use of consultants from outside public administration (variable “outexp”). The importance of embedding external expertise before the RIA process begins was particularly evident in the EU appraisal of proposals to restructure the sugar sector (case ID 4). The potential abolition of intervention mechanisms in the EU sugar market regime meant the threat of judicial review prompted by stakeholders was very real and resulted in the adoption of an evidence-based approach to policy change. As a result, the appraisal process was described by analysts as one underpinned by “mutual learning” (EVIA 2007, p. 50).

There exists another, more complicated sufficient combination of conditions. This alternative path postulates an active role of the oversight unit, presumably concerned with the quality of RIA and the presence of elected policymakers in the process, combined with the absence of trade-offs.10 This second path seems, however, highly contingent on missing cases, because when recomputed it led to inconclusive results. Therefore, it is not reported in Table 4.

Turning to our third type, the communicative RIAs (model 5 in Table 4), one variable in our template highlights an important dimension (“Dialrespon,” coded at 1 when the case study included evidence of communicative responses to the stakeholders, typically by showing how the concerns raised in consultation processes were taken into consideration). We looked for some obvious variables that could possibly affect the outcome, such as the novelty of the policy issue (variable “polnov”), whether industry or other actors were affected (variable “affect”), the visibility of the policy problem (variable “issvis”), the timing of RIA in the policy formulation cycle (variable “timing”), the presence of uses of the RIA by elected officers (variable “electgov”), and whether resources were severely constrained or not (variable “resou”) (model 5 in Table 4).

The examination of crisp–sets relations led to results that are inconclusive after robustness checks. We can, however, mention two paths, which are sensitive to missing cases, and hence should be interpreted with extreme care (not reported in Table 4). In both cases the communicative outcome is associated with the presence of industry-level stakeholders being affected by the proposed regulation, in combination with other variables. In one path, “affect” is combined with elected politicians kept at bay from the RIA; in a context where resources (human and financial) are not constrained and issue visibility is low, more communicative usage is found. The appraisal process associated with the investigation of accidents in the maritime sector (case ID 12) is illustrative. Here communication, as fostered by Member States, involved being limited to sharing best practice in maritime passenger transport and insights from accident case histories (EVIA 2007, pp. 36–38).

To improve the exploration of communicative RIAs, we consider a model for the variable “transparency” (model 6 in Table 4). The model includes different variables, specifically information about stakeholders' involvement beyond consultation (variable “informcons”), the elected policymakers, resource constraints (value of 1 for severely constrained resources), the central oversight unit, balanced consultation, who is affected, and timing of the RIA.

We had a prior expectation that transparency is higher when economic stakeholders are involved, consultation is balanced, and time and other resources are not severely constrained. The presence of elected politicians can push the agencies to be more transparent. In the appraisal process of the EU's thematic strategy on air pollution (case ID 21), Commission officials involved in drafting the impact assessment were mindful that the proposals had to stand up to the wider scrutiny of the Commission Cabinet, European Parliament, and Council (TEP 2007, pp. 306–307). The oversight unit similarly combines with affect and balanced consultation to create a path to appraisal transparency. As Table 4, model 6 illustrates, a balanced consultation is important in three combinations of conditions. Overall, this intermediate solution for crisp sets is also very stable after having dealt with missing values.

6. Discussion and conclusions

This article contributes to the literature on policy instruments and knowledge utilization by relaxing the assumption that instruments unambiguously embody theories, beliefs, and ideas. This may be true at the abstract level – there is no doubt that RIA incarnates cognitive and normative beliefs about the role of economic analysis and cost–benefit principles (Turnpenny et al. 2009; Kysar 2010). However, when the guidelines written by governments are implemented by constellations of actors, the “ideas behind the instrument” look more ambiguous and pliable – a point that chimes with Jabko's notion of ideas as “talismans” (Jabko 2006). They can be negotiated on the ground. The whole RIA can be reframed in one direction or another. The initial commitment of the government to carry out the RIA is an incomplete contract that can be shaped by implementation actors.

Since appraisal is reframed by communities of practice, it often leads to outcomes that are different from the original aims set by the government. Ours is another way of looking at the expectations-capability and adoption-implementation gaps that emerge in recent studies of RIA (Jacob et al. 2008). Our results, however, come with several limitations. Our analysis is tentative and exploratory. We scored cases that were influenced by the early period of the Commission's RIA strategy. And we did not explore wider comparisons, by adding other European countries and, perhaps, the North American cases – something that future research could usefully do.

Empirically, our sample illustrates that there are at least four types of manipulation of RIAs – this is our answer to the first research question. The analyses of the causal conditions enrich our understanding of the types by relating conditions to outcomes. Contrary to our prior expectations, the sample shows that instrumental usage is not so rare. Additionally, given certain conditions, communicative and political usages of RIA are closer than we thought. The combinations we examined by using crisp-set QCA to answer our second research question show that the prior expectations are not necessarily wrong. But they focus on one variable at a time. An example is consultation, often seen as a panacea for learning and dialogue with the stakeholders. Another is the role of elected politicians, often seen as “political meddling with the RIA” whilst it can also lead to transparency. Overall, the QCA of our 31 cases points toward more complex combinations of conditions.

Methodologically, we have demonstrated the advantage of using meta-analysis and QCA to explore case studies in a cumulative fashion, thus providing a more comprehensive and systematic coverage than previous case-study research. By examining the rich data generated from cumulative case studies, our findings also complement the scorecard-based studies of compilations of RIAs.

Finally, our third research question: what are the implications of this analysis? We found that constellations of actors reframe the appraisal process by using regulatory innovations in different ways. They negotiate “what's the impact assessment to do” when they “sit down” and attend to a specific RIA. In some cases this is a good thing – the local actors define what RIA should do for them, given the balance of power, resources, and problems at hand. Arguably, actors have to invent an “art of convening” (Rein 2006, p. 397) by pre-negotiations in which shared agreements about the purpose of the RIA in that case can be developed. In other cases, shaping meanings may lead to an explosion of expectations about what the RIA is for. This variability of RIA usages could be reduced if governments were clearer on expectations. In terms of policy recommendations, the QCA analysis shows that there are different paths to desirable outcomes. Governments, international organizations, and audit bodies should therefore explore different combinations of conditions leading to the usages they deem desirable rather than arguing for a fixed menu of variables.


This paper is based on research carried out with the support of the European Research Council (ERC) grant on Analysis of Learning in Regulatory Governance (ALREG), directed by Claudio Radaelli Dunlop and Radaelli gratefully acknowledge the support of the ERC. The authors wish to express their gratitude to the participants to the ECPR Joint Sessions in Munster, 22–27 March 2010 and ECPR Regulatory Governance conference in Dublin 17–19 June 2010. We are grateful to Oliver Fritsch for research assistance, Andrew Jordan and John Turnpenny for cooperation on the IA library (see endnote 2), and to seven anonymous referees and the journal editors for their insightful comments.


  • 1

    The European Commission started with only 21, 27, and 73 RIAs in 2003, 2004, and 2005 respectively, but in recent years it stepped up to 94 RIAs (2007), 120 (2008), and 75 (source: our calculation from the impact assessment website of the European Commission). In the UK 183, 277, and 315 RIAs were conducted in 2003, 2004, and 2005 respectively. Since 2007, UK RIAs can be found on the government library of impact assessments ( Earlier UK RIAs can be found by tracing the Command Papers.

  • 2

    We have gathered all the case studies we are aware of in a library at

  • 3

    Provided by Charles Ragin at

  • 4

    It is worth noting that relatively few contradictory configurations are present in the truth tables (2 in models 1 and 5; 3 in model 2; 1 in model 3 and 0 in models 4 and 6). Following Ragin (1987), these contradictory configurations are treated as “unclear” and recoded as [0] on the outcome value.

  • 5

    For the same reason, we considered the analysis of necessary conditions to be unproductive.

  • 6

    For example, the recalculation of the first recipe of Table 4 implied to recompute consistency and coverage scores by evaluating them only with respect to ∼tradeof and ∼distrib, in addition to standard crisp-set analyses.

  • 7

    The full analysis is available upon request.

  • 8

    All cases were analyzed in each model, when possible. As usual, cases with missing values for one or more variables were not included in the configurations in the truth table.

  • 9

    A few definitions are in order here. Raw coverage concerns the share of the outcome that is explained by a certain alternative path (Schneider & Wagemann 2010) Consistency is the degree to which a subset relation has been approximated (Ragin 2006). We also need to explain the meaning of solution coverage. The solution coverage is the share of the outcome that is explained by the solution – that is, its empirical relevance. The decision on a threshold of ‘decency’ depends on the researcher. For some research questions, coverage can be of secondary importance, for other research questions, especially if you want to portray some general patterns, it is more important. We assume that over 60 percent is not too bad (in the text we refer to ‘decent’ solution coverage), over 80 percent is certainly good. A solution with poor coverage is not false, it is simply rare (but it may still be of interest).

  • 10

    Note that a value of 0 for this variable simply means that there are no trade-offs in the published RIA; we do not know whether the trade-offs were ignored or simply did not exist, although if ignored and important they should have shown up in other variables, such as “low legitimacy” of the RIA.