Governance by Evaluation for Sustainable Development: Institutional Capacities and Learning – Edited by Michal Sedlacko and André Martinuzzi


Governance by Evaluation for Sustainable Development: Institutional Capacities and Learning , edited by Michal Sedlacko and André Martinuzzi , published by Edward Elgar , 2012 , xi + 314 pp., £85.00, hardback .

This collection of fifteen chapters is a selection of contributions which were first presented at two international conferences held as part of the project EASY-ECO 2005–2007. This project included a series of conferences and training courses supported by the European Commission's Marie Curie Initiative under the Sixth Research Framework. EASY-ECO (Evaluation of Sustainability: European Conferences and Training Courses) was launched by the Research Institute for Managing Sustainability at the Vienna University of Economics and Business Administration. Twelve European research institutions were involved in this project aimed at building sustainability evaluation capacity and facilitating the exchange of relevant experiences. The project was prolonged in the project EASY-ECO 2008–2010 and its final conference took place in Brussels in November 2010. This book is the third volume in the Evaluating Sustainable Development series published by Edward Elgar. The editors announce in their preface a soon-to-be-published fourth volume, edited by German academic colleagues.

The introductory chapter by the editors mainly explores the theoretical background of the themes of this volume. Thematically, this book focuses on the institutionalization of monitoring and evaluation systems and procedures in policy making across Europe. In line with the context of its project, this collection of contributions elaborates also on learning and capacity-building through evaluation, on the challenges of evaluating the participatory policy processes and on providing insights into recent methodological developments. Starting with a very short but comprehensive overview of the literature regarding (good) governance of/for sustainable development, Michal Sedlacko and André Martinuzzi move quickly towards their main points of interest of ‘evaluation’ and ‘learning’. According to the editors, there is very little information on the politics of sustainability evaluation and this book should be considerated as an effort to remedy the current scarcity. However, they acknowledge the problematic relationship between individual and organizational learning, and they seem to accept learning opportunities for organizations (and society) through a variety of learning processes. If such processes, inter alia based on evaluation exercises, may lead towards changing decision-making rules, one may speak about transformational (value-driven) governance. As an introduction, this text offers a clear and useful overview for readers who are not familiar with the public management terminology, but it lacks some explanation for the readers without any sustainable development (SD) background.

Part Two of this volume deals with learning through evaluation. The two chapters in this part are critical of the rationalist policy cycle model and share a perception of how evaluation can support the change of the conventional modernist/rational approach to steering. The contribution of Pregernig et al. offers an analysis of three Austrian strategies for SD: the Strategy for Sustainable Development, the Forest Dialogue and the Biodiversity Strategy. They apply three main conceptual categories (rationales, approaches and goals of evaluations). Their empirical work confirms the indications from their heuristic framework: evaluations serve multiple objectives. In this sense, the authors state that sustainability evaluations are no pure technical exercises, but they are more than ‘political power play’ (p. 42). However, one may question whether the choice of the strategies that were evaluated has influenced the outcome of this analysis. Would it not have been a surprise if the evaluations of all three strategies (all fitting into the SD policy context) should have revealed other outcomes?

The second chapter in this part offers an interesting, but rather theoretical overview of tools for learning-oriented environmental appraisal. Måns Nilsson presents not only the well known tools based on – mostly environmental administration – practices (like multi-criteria analysis and scenario analysis), but also looks to policy analysis concepts. He wants to go beyond the monitoring limitations and argues for a toolkit that includes both analytical and deliberative instruments useful for ex ante and ex post appraisals.

Part Three on ‘institutionalising sustainable development concerns in European policy making’ contains three chapters and focuses on the use of evaluations by the European Union institutions. The central theme of Chapter 4 by Franz and Kirkpatrick is the impact assessment (IA) process as introduced and implemented by the European Commission. IA illustrates a linkage between governance for SD and the (former) Better Regulation – now called ‘Smart Regulation’– agenda. The latter is to be considered as a tool for addressing socioeconomic concerns with a focus on reducing administrative burdens, being part of the former Lisbon Strategy, whilst the Sustainable Development Strategy was an outcome of the Gothenburg process. The new Europe 2020 Strategy aims to incorporate both strands. This was illustrated by the Commission communication of 20 June 2011, ‘Rio+20: Towards the green economy and better governance’. Although the latter does not mention IA, it does include a lot of wording about governance. The research by Franz and Kirkpatrick offers valuable insights in the Commission's IA practice, which includes a kind of quality assessment by the Impact Assessment Board (IAB):

While there is limited evidence in the IAB Opinions reviewed of explicit attention to sustainability issues, there is evidence in the IAs of greater treatment of the SD principles across the [Directorates-General]. In particular, attention to the principles of subsidiarity and proportionality were well addressed by the IAs reviewed. There was, however, a consistent recognition from the IAB that more attention should be given to quantifying impacts, particularly in the form of assessing a greater range of costs and benefits. (pp. 79–80)

This excerpt from the authors’ conclusions illustrates their particular focus and understanding of SD, which is rather econometric. In general, however, and given the broad scope of the research, this chapter contains a number of interesting observations about the Commission's IA approach and its current limitations.

The fifth chapter by one of the editors, André Martinuzzi, focuses on the evaluation of EU-funded research in the EU innovation context. Referring to the EU's Sustainable Development Strategy as renewed in 2006, this chapter draws some lessons learned for ‘governance by evaluation’. This includes two aspects: the necessity to build institutional capacities, and the mechanisms of institutional learning. In this particular case and chapter, both aspects are elaborated around the monitoring approach as established for the EU's research programme (the Seventh Framework Programme):

Knowledge transfer is only possible when information is transparent and publicly available in a form meeting the needs and contexts of the specific target group. The FP7 monitoring system consequently addresses the different target groups with different means. (pp. 103–104)

This includes a public platform that includes an on-demand analysis service. Without exaggerating its importance, this approach presents a certain openness that could – or even should – inspire the IA practitioners in the Commission.

The sixth chapter concerns policy learning in the area of official development assistance (ODA). Olearius et al. reveal in their contribution how an audit report by the European Court of Auditors played an important role in reinstating the importance of environmental considerations in ODA decisions after a previous weakening in the Commission's decision making. This chapter clearly highlights the difficulty of integrating environmental concerns in comprehensive strategic frameworks – even the ones aiming at SD. The authors – all non-academics – also wrote a brief addendum including some relevant information from 2011, such as the observation (p. 127) about the loss of institutional memory by the huge transfer of ODA personnel in the Commission. A very critical point when dealing with ‘organizational learning’!

‘Dealing with multi-stakeholder contexts’ is the subject of the third part of this volume. It contains three case studies and a theoretical contribution. All four chapters describe and analyze different tools and methods to engage stakeholders in decision making. The approaches that are presented reflect a variety of disciplines and perspectives. Özerol (Chapter 7) presents a Turkish water management case by using the (institutional) common-pool resource management approach.

Chapter 8 by Hollaender and Stokman presents a Dutch case about dynamic decision (making) analysis. They focus on the measurement of the impacts of stakeholder dialogues as used in the Costa Due approach initiated in 2005 in the Dutch province of Groningen to explore promising uses of biomass. This project aimed also at implementing innovative ideas, but the researchers state that ‘during the dialogue process it became apparent that innovation vs implementation might require different settings in order to be carried out successfully’ (p. 170). As in the previous chapter, the authors refer also to the guiding principles for identifying relevant stakeholders in collective-choice arrangements, proposed by Ostrom.

Illustrated by a project in Cambodia supported by the German ODA and the Food and Agricultural Organization of the United Nations (FAO), Strele presents in Chapter 9 the method of participatory livelihoods system appraisal. This assessment approach is rather simple and flexible as it focuses on the most crucial factors with a potential to trigger an effective positive change in the sense of SD. Reading this grassroots contribution was quite refreshing and enlightening compared to – at least – some other chapters in this volume.

Chapter 10 by Grafakos et al. brings the reader back to the academic world. Using the multi-criteria analysis, they propose a weighting process for the selection of criteria and stakeholder involvement in ex ante evaluation of climate policies. Their outcome leads to the following – quite obvious – implications (p. 207): learning and capacity-building, in-depth discussion, transparency, legitimacy and acceptance and delivery effectiveness. But they also mention the need for further research.

Part Four also contains four chapters about the topic of ‘developing learning capacity in organisations’. All four contributions present case studies and are written by practitioners. Chapter 11 by Powell et al. presents the use of outcome mapping (OM) – a new approach to project planning, monitoring and evaluation – to assess a Swedish ODA support programme in Bosnia and Herzegovina. The programme to support civil society organizations was established because at that time the government was not yet functional. The authors give an overview of OM's weaknesses and strengths (compared to results-based management (RBM)), including its potential to develop organizational capacity. However, in their conclusions they state that: ‘The clearest success of projects is to help individual civil society actors, among not only boundary partners but also members of implementing partner organisations themselves’ (p. 228). In their recommendations, the authors seem not to stick to OM exclusively and they suggest the possibility of a hybrid OM/RBM form (p. 230).

Chapter 12 by Deprez, on ‘the development of a learning-oriented monitoring system for sustainable agriculture chain development in Easter Indonesia’, illustrates how the Belgian non-governmental organization (NGO) Vredeseilanden has changed its Indonesian VECO-programme. VECO is a sustainable agriculture chain development programme that supports farmers in Eastern Indonesia. Its head office is based in Bali and it operates with 24 staff. For the last decade, VECO applied a project cycle management approach based on the logic framework approach (LFA) by using OM. One of the criticisms often directed to LFA concerns its top-down approach that undermines debate opportunities with stakeholders. The new VECO approach is called ‘planning, learning and accountability system’ (PLA) and includes an integrated monitoring and evaluation process which facilitates learning and improvements. As the experience with the new VECO approach is still young, the author's conclusions remain rather prudent and aware of potential pitfalls and challenges.

The process monitoring of impacts and its application in Structural Fund programmes of the EU is the subject of Chapter 13 by Hummelbrunner. He refers to the well known problems and limits of the current monitoring systems for these funds. Within the different political fora of the EU, the reform of these Structural Funds is being debated and the next programmes (as of 2014) are being prepared. In its general presentation of the reform proposals –inter alia based on the critical Barca Report about the deficiencies of the current 2007–2013 Structural Fund implementation – the Commission stated explicitly that:

The establishment of joint monitoring committees and joint monitoring and reporting systems can lead to cost savings for national authorities. Joint planning will facilitate an integrated approach to the delivery of cohesion policy.1

Hummelbrunner states that the use of indicators is not sufficient as the processes are completely overlooked. Therefore, he favours the process monitoring of impacts (PMI) instrument, as developed within Austria.

A Swiss case is presented in Chapter 14 by Störmer and Truffer. They describe the strategic planning approach of the Regional Infrastructure Planning (RIF) for the sanitation sector as developed and tested in the Swiss national research programme. It is an alternative approach based on foresight and scenario development techniques to map alternatives and it includes also a participatory sustainability criteria assessment. The authors claim that participatory foresight is necessary for creating awareness on deviations from personal future expectations. However, their description of the RIF method illustrates also a rather technocratic development, certainly when reading about the need for further integration or coordination with other sector and cross-sector strategies. While institutions and other organized stakeholders might acknowledge obvious advantages in the RIF method, the individual or personal participation or interest remains quite questionable.

The final part of this volume contains only one contribution: Chapter 15 by Wilfing and Bechtold. They analyze the role of visualization within sustainability evaluation processes. This contribution offers mainly an interesting theoretical introduction on this issue, inter alia, given its perspective on sustainability research:

Taking into consideration the need for legitimating decisions, science must be reframed in terms of disrupting merely disciplinary approaches and involving real-world perspectives[:] how to promote more meaningful interaction among policy makers, scientific experts, corporate producers, and the public. (p. 283)

The different chapters in this book offer a challenging variety of theoretical analyses, case studies, critical empirical observations and opinions about a whole range of assessment and evaluation approaches. This collection illustrates a growing European research stream which has to cope with the pitfalls of balancing between academic rigour and political and public expectations. In this sense, this publication should attract interest from a wide audience. Not only academics might broaden their predominantly theoretical learning perspectives. Moreover, policy makers could gain more than some abstract insights as this volume – especially Part Four – offers valuable lessons learned from particular evaluation cases, such as the Swedish ODA efforts in Bosnia and Herzegovina. Finally, stakeholder organizations could enhance their practices on the basis of this book. It might contribute to new reflections about participation, but also teach such organizations to keep realistic expectations concerning institutional learning!