A study of the Resilience Analysis Grid method and its applicability to the water sector in England and Wales

This paper presents an adaptation of the Resilience Analysis Grid (RAG) method framed on the Resilience Engineering theory as a proposal for tackling the lack of organisational resilience metrics in the water sector of England and Wales. The method was adapted to the sector by selecting 16 indicators and the addition of a ‘Resilience Ethos’ section to the grid. Its applicability was tested by analysing the evolution of resilience in Ofwat’s Price Reviews 2009 (PR09), 2014 (PR14) and 2019 (PR19). Key conclusions obtained were that: (I) PR14 acts as a pivot between PR09 and PR19, as it installs a new regulatory approach; (II) the sector has advanced significantly towards greater consideration of resilience and its management in PR19; and (III) the PRs lack instruments for in‐period performance assessment. The RAG method proved to be simple and flexible to use, delivering clear and straightforward graphical results.


Introduction
The impact and importance of the concept of resilience have increased over the past few years, particularly in the face of threats of climate change, increasing urbanisation and growing population (Butler et al., 2014). A simple search using the word 'resilience' in the Web of Science yielded over 76,000 results in a wide range of disciplines (Clarivate, 2019). While traditional safety promotes a 'fail-safe' approach and the illusion of creating a system that never fails (Wharton, 2015), resilience endorses a new approach in the analysis of organisations focusing on a 'safe to fail' ethos (Holling, 1996).
In the United Kingdom, after a series of extreme weather events between 2007-2014 and the first results of the UK Climate Impact Programme, the need for a broader look into resilience was evident. The amendment of the Water Act in 2014 gave the Water Services Regulation Authority (Ofwat) the primary duty to further its resilience objective, making resilience a priority issue in the water sector in England and Wales. For the operationalisation of the resilience concept, benchmarking methods, standards and metrics are needed (Resilience Task and Finish Group, 2015). Although seven resilience principles were defined in Ofwat's latest policies (Ofwat, 2017a;2017b), there is still a lack of a framework, guidance and indicators for objectively assessing resilience outcomes (Safe & SuRe, 2015). Various methods have been reported to benchmark resilience in existing literature (McManus, 2008;Hollnagel, 2011b;Lee et al., 2013;Cai et al., 2018), however, there are no specific methods developed and applied in the water sector.
This paper proposes an analysis of the sector based on the Resilience Engineering (RE) framework (Hollnagel and Woods, 2006). The framework is defined in the context of socio-technical systems, and its operationalisation is founded upon the understanding that technical and social elements are highly integrated, influencing one another to maintain their activity and existence in the pursuit of a goal (Patriarca et al., 2017). RE promotes a systematic approach to understand resilience and organisational management for business continuity under deep uncertain conditions (Pęciłło, 2016;Patriarca et al., 2018). Its focus is on resilience performance, rather than resilience as a property, and stresses the importance of how the systems respond to both disturbances and opportunities (Hollnagel, 2011a(Hollnagel, , 2018. In order to understand how resilience performance works in organisations and to develop an operational resilience assessment method, RE suggests four 'abilities that characterise resilient organisations, referred as the 'four cornerstones of resilience': responding, adapting, monitoring and learning (Hollnagel, 2011b). Monitoring refers to the system's ability to address the critical and knowing what could affect performance in the near term, by analysing the environment and the system itself. Anticipating addresses the potential by knowing what to expect from developments further into the future, such as disruptions or changes in operating conditions. The differences between monitoring and anticipating are the time horizon and the type of threats that are studied. Learning is associated with the importance of knowing what has happened and being able to learn from experience. The main focus on learning should be on success and not failure. Responding refers to the capability to address the actual and knowing what to do when exposed to regular and irregular disturbances; the system must be able to adjust its functioning by activating prepared actions or by adjusting their mode of functioning (Dekker et al., 2008;Hollnagel, 2011b). While resilience may be increased through manipulation of the abilities, this alone is not guaranteed to increase system performance. The abilities are not independent, and a resilient system needs to be balanced among them (Patriarca et al., 2018).
Based on these four cornerstones, the Resilience Analysis Grid (RAG) method is a tool that aims to assess how resilient an organisation is by ranking four sets of questions, one for each ability (Hollnagel, 2011b;Patriarca et al., 2017). The answers are used to create resilience profiles that constitute a powerful graphical way to benchmark resilience and understand its development over time. It must be highlighted that these RAG profiles do not provide an absolute rating of resilience performance; however, these are a starting point to understand how the organisation functions and the consequences that planned interventions would have on the evolution of resilience (Hollnagel, 2018). The RAG method has been applied to a range of organisations and sectors, such as the oil and gas industry (Apneseth et al., 2013), the Swedish Civil Aviation Administration (Ljungberg & Lundh, 2013), polish enterprises (Pęciłło, 2016) and the health care sector (Hedge et al, 2015). The application of the framework in such different systems demonstrates its flexibility and applicability in different types of organisations and at different scales.
This study is the first development and application of the RAG method to the water sector in general and that in England and Wales, in particular. The study aims to evaluate the method's utility and to propose a systematic approach to the evaluation of organisational resilience in the sector. It will also help to gain insight into the progress on the approach and adaptation of practices that are more resilient in the water sector, analysing the evolution of resilience and its operationalisation in the sector. This application will allow to providing a critique of the method based on the results obtained, helping to identify what could be improved in the proposed adaptation and determining what are its main advantages and disadvantages as an organisational resilience tool for the water sector.

Methodology
The RAG building process consists of four phases: the description of the system to be analysed, the selection of questions to assess the different abilities, the ranking of the questions and finally, the unification of the results in a star plot (Hollnagel, 2011b). The next section details the application of the RAG method to the water sector of England and Wales, using the four phases' process.

Phase 1: Definition and description of the system analysed
A clear and concise definition of the structure and boundaries of the organisation is crucial for the success of the method, as it restricts the field of application and contextualises the analysis (Patriarca et al., 2018).
After the privatisation of the water sector in England and Wales in 1989, the provision and regulation of services provided were established and distributed between an environmental regulator (Environmental Agency), a drinking water regulator (Drinking Water Inspectorate) and an economic regulator (Ofwat). Ofwat, although it is the economic regulator of the water sector, must also ensure that water companies properly carry out all their statutory functions, including what it is exhorted by other regulators . In addition, through the modification of the Water Act in 2014, Ofwat was given a new statutory duty to 'further the resilience objective to secure the long-term resilience of water companies' water supply and wastewater systems and to secure that they take steps to enable them, in the long term, to meet the need for water supplies and wastewater services' (Water Act, 2014).
Every five-years, Ofwat reviews price limits and elaborates a methodology for controlling the prices, incentives and services packages that customers receive -the Price Review (PR). The PRs aim to balance the customers' and companies' interests while assuring that the sector is able to finance their services and meets all their legal, environmental and social duties. The PRs provide a five-year framework within which the companies and the sector as a whole will work, establishing the challenges that must be tackled and creating controls that aim to enable icentivisation and encouragement of the companies to achieve their duties . As PRs determine the regulated planning, evaluation and operation of the water sector, they are ideal source of information on the state of resilience thinking and planning within any particular five-year period.
As a first exploratory study, this paper evaluates the PR in 2009, 2014 and 2019 (PR09, PR14 and PR19). PR09 and PR14 were developed before the modification of the Water Act, 2014 and were in operation between 2010-2015 and 2015-2020, respectively. In December 2017, the final methodology for PR19 was published, which will set the regulatory environment for the sector between 2020-2025. These three PRs are crucial to understanding the impact of the new resilience duty introduced by the Water Act in 2014 on Ofwat and the water sector. Their study promotes the understanding of the evolution of a more resilient sector and enables the detection of any areas where more development is needed.

Phase 2: Selection of relevant indicators and questions for each of the four cornerstones
This phase consists of the selection of four sets of indicators and questions concerning the four cornerstones of resilience, which will define the analysis grid. The questions must successfully illustrate the critical functions of the system and be relevant for the assessment of the case study. The approach to the development of the questionnaire varies in the literature, and it is very much dependent on the type of system studied.
In this study, the selection was performed entirely by the authors and it was based on the original questionnaire proposed by Hollnagel (2011b) and the indicators used in applications in other sectors (Apneseth et al., 2013;Ljungberg & Lundh, 2013;Hedge et al, 2015;Pęciłło, 2016;Patriarca et al., 2017). The final indicators and the questions were adjusted in an iterative process when analysing the different PRs. Each of the four cornerstones is evaluated using four indicators, and each indicator has a question associated with it that illustrates which is being assessed (Table 1).
An overall description of how resilience is approached and defined appeared as a necessary addition. In the literature, Lee et al. (2013) modified the Relative Overall Resilience framework developed by McManus (2008), incorporating a 'Resilience Ethos' section, which served to understand how the organisations address resilience. This modification is applicable and imperative to the RAG method, as the sector that does not have a mature definition and perception of what is being resilient means. This is an initial questionnaire and its purpose is a first approach to evaluate the applicability of the RAG analysis to the water sector. Consultation with experts and stakeholders is crucial to create an effective resilience metric method, and this questionnaire could be used as a foundation for further developments. Ljungberg and Lundh (2013) proposed two different approaches for rating the indicators: using assertions or open-ended questions. Although both approaches are valid and complementary, the success of the tool is based on the fact that all the questions are rated using the same terminology (Hollnagel, 2011b).

Phase 3: Rating the selected indicators and questions
The RAG rating used here was qualitative and udertaken by the authors based on extensive research and evaluation of the information available in the final methodologies for PR09 (Ofwat 2007;2009), PR14 (Ofwat, 2013) and PR19 (Ofwat, 2017a), their appendixes, documents that explain their application and the multiple consultations made for their elaboration. The evaluation of the abilities was established using a 6-point 'Likert scale' (Table 2; Likert, 1932;Hollnagel, 2011b), where each indicator in Table 1 is assigned a rating based on Table 2.
The 'Resilience Ethos' Section is not ranked and is merely used to illustrate how resilience is addressed in the PRs. To analyse the 'Resilience Ethos', the authors identified all references to resilience in the PR texts. After a primary identification, the references were evaluated on their approach and the context where the term was used.

Phase 4: Combinations of the ratings for each cornerstone, and for the four cornerstones of resilience combined
The results for each ability are presented in a star plot, and together provide a snapshot of the organisational resilience of the sector. The star plot axes represent the indicators used to rate each cornerstone ability and deliver a visual illustration of the balance between them (Hollnagel, 2011b;Patriarca et al., 2018).
By using the associated numerical value for each rating (Table 2), an arithmetic mean of the indicators' ratings for each ability is calculated. This allows a straightforward way to assign an overall rating for each of the abilities and aggregate them for the final RAG star plot (Hollnagel, 2011b). The concluding star plot has the four cornerstones of resilience as axes and provides a compact illustration on how the system as a whole is performing based on the different abilities.

Results
This section summarises the results of the analysis of the PRs using the adapted RAG method for the water sector ( Table 1). The star plots obtained by the aggregation of the results for each of the resilience cornerstones abilities are shown in Figs 1 and 2. A more extensive evaluation of each of the abilities and the aggregation of the results is shown below.

Resilience Ethos
Although the 'Resilience Ethos' section is not rated and consequently not added in the final aggregation of the results in the star plots, it is crucial to understand the evolution of the term resilience in the documents as a preamble to the analysis of each of the cornerstone abilities.
The analysis shows an important contrast between the PRs. PR09 does not have a clear definition of resilience, and the references to the term found in the final methodologies are mainly focused on infrastructure and asset resilience to flooding hazards. However, there are more references to the term in PR09 than PR14. In PR14, the references to resilience are related to water supply, environmental change and finance. Although 'Resilienceoutcome-focused regulation' (Ofwat, 2012) with a primary definition and nine planning principles was published, the document is not mentioned in the PR14 final methodology.
In PR19, resilience is seen more holistically and has a definition produced by the Resilience Task and Finish Group. Resilience is one of the key themes in PR19, providing a methodology to approach the subject. Resilience is considered to have three main aspects: operational, financial and corporate. Ofwat (2017a) is emphatic about the importance of companies taking an integrated systems approach to the three aspects mentioned and provides an overall approach to links between them.

Ability to monitor
The ability to monitor was studied using four indicators: 'Indicator Lists', 'Validity of the Indicators', 'Analysis and Interpretation' and 'Organisational Support'. The analysis showed that there is a clear progress towards a more comprehensive monitoring ability in PR19 (Fig. 1a).
'Indicator Lists' identifies the system performance indicators proposed in each PR. Ofwat provides a set of mature common indicators in PR09, which evaluate the key challenges set by their customers and stakeholders. However, their impacts are financial incentives only, and they are mostly linked to financial aspects and the development of the business plan. These changes in PR14, where a new framework to evaluate the companies' outcomes, are introduced. Although the new framework is a significant break in Ofwat's evaluation system, its first application had important deficiencies, as the system did not have common definitions and this hindered a consistent evaluation of the sector. The main constraints of the PR14 Outcome Delivery Incentive system are tackled in PR19, where Ofwat proposes a set of common and clear performance indicators. Even though the importance of evaluating all aspects of resilience is emphasised, PR19 indicators lean towards operational resilience.
'Validity of the Indicators' evaluates how appropiate the indicators used are, and how their relevance is justified in the PRs. Examination showed a similar evolution to the 'Indicator List' indicator in the PRs. PR09 offers a clear framework to choose and assess the indicators, contrary to PR14, which introduces a new determination and evaluation system that lacks common definitions that allows flexibility in their evaluation. The introduction of important tools in PR14, such as the performance commitments, is consolidated in PR19 by the consistent definition of indicators aligned with the priorities set for the sector.
'Analysis and Interpretation' detects the tools and means foreseen in the PRs for the communication and evaluation of the companies' assessment results. This indicator shows important differences between PR09 and PR14. The former acknowledges the importance of the June Reports, the Annual Performance Report (APR) predecessor, for the communication of results in the sector. The latter not only fails to mention this instrument in its methodology, but also the outcome delivery system is only evaluated at the end of the five-year period. This changes in PR19, where in-period incentives are created. Here, Ofwat facilitates the evaluation of performance by creating a more transparent environment and generates discussion based on the ARP.
'Organisational Support' aims to understand what the organisational mechanisms are for the creation and evaluation of the indicators, as well as how often these are revised. The indicator showed that PR09 and PR14 have organisational mechanisms mostly concentrated before and after the five-year period. PR19 introduces various in-period tools, such as the company-monitoring framework and the outcome delivery incentives.
The rating of the indicators and the star plots shows the evolution of the monitoring ability in the sector. However, Table 2 Likert scale for the evaluation of the abilities (Likert, 1932;Hollnagel, 2011b) Rating Numerical Value Description Missing 0 No capability whatsoever to address the specific item Deficient 1 Insufficient capability to meet the criteria addressed by the specific item Unacceptable 2 The system does not meet the nominal criteria addressed by the specific item Acceptable 3 The system fully meets all reasonable criteria addressed by the specific item Satisfactory 4 The system fully meets all reasonable criteria Excellent 5 The system exceeds the criteria addressed by the specific item the change in the regulatory paradigm in PR14 resulted in a lower grading in the ability to monitor compared to PR09, due to the lack of clarity in the new concepts. PR14 can be seen as a transition between PR09 and P19, where the importance of the outcomes and performance commitments is settled. PR19 sets a more consistent approach towards monitoring the sector, establishing common goals and various tools to further into the performance indicators during the five-year period.

Ability to anticipate
The indicators used for the assessment of the ability to anticipate on the PRs were 'Assumptions about the Future', 'Acceptability of Risks', 'Aetiology' and 'Time Horizon'. The results are summarised in Fig. 1b. 'Assumptions about the Future' identifies what are the models for the future in the PRs. Although the business plans constitute the companies' future models for the three  PRs, there is a different view on their strategic planning points. PR09 proved to be very clear about the plans and information needed from the companies, as well as being emphatic on the importance of the long-term view in the sector shown in the various plans. The long-term approach is lost in PR14 and in addition, there is a less prescriptive approach in the business plans. An important change is made in PR19, which introduces a clear framework for resilience planning and encourages companies to look further ahead, prioritising innovation and resilience.
The main objective of 'Aetiology' is to understand how future threats and opportunities are modelled, and what their assumed nature is. The progression in the PRs is very marked, and there are important developments made in every five-year cycle. The sector's key challenges and the way companies should approach them are described in PR09, but there is no information about how the future should be modelled. PR14 introduces a risk-based approach with scenarios based on the key risk drivers in the sector, such as household growth and rainfall variation. Although PR19 continues the scenario planning approach developed in PR14, it presents a new framework with a set of different scenarios and a strong economic focus.
'Acceptability of Threats' detects if there is a threshold for threats to be seen as acceptable or unacceptable, meaning how the evaluation of the future models is conceived in the PRs. The future strategy assessment is not explicit in PR09. However, PR14 and PR19 introduce clear frameworks for the evaluation of the business plans and their models for the future. While there are similarities in the key areas tested in both PRs, the frameworks do not show continuity. PR19 proposes a categorisation of the companies based on the quality of their models and proposes financial, procedural and reputational incentives.
'Time Horizon' evaluates how far the PRs look ahead. Both PR09 and PR19 are clear on their long-term approach, stressing the importance of the link between long-term plans, such as the Water Resource Management Plan (WRMP) and 25-year Strategic Direction Statements and the companies' business plans. However, in PR14 the fiveyear business plan is the sole future model expected from companies. This is a short-term view approach, especially when considering operational resilience and the companies' ability to secure supply.
In conclusion, as seen in the ability to monitor, PR14 appears to be a pivot between the PR09 and PR19. PR14 settles the basis for a new approach towards the regulation of the sector, but PR09 is arguably more mature in terms of the concepts used and the promotion of a longerterm view. PR19 crystallises the evolution of the sector towards resilience and innovation, as it settles consistent frameworks for the development of future models and their assessment.

Ability to learn
The study of the ability to learn is performed using the following indicators: 'Learning Basis', 'Data Collection', 'Implementation and Communication' and 'Frequency'. The star plots and rating of the indicators are aggregated in Fig. 1c.
'Learning Basis' recognises how learning is developed in the PRs. The PRs analysed have pathways for learning during the creation of the PR methodologies and after the five-year period, with important participation of the sector's stakeholders. PR19 is the first to integrate in-period learning with the development of new tools, such as in-period ODIs and customer engagement models to create learning and communication.
'Data Collection' identifies the organisational support for data collection and its analysis and learning. Data collection and its analysis are effective before and after the five-year period, and there is a culture developed for information sharing. However, PR09 and PR14 lack tools that promote learning and effective data collection for the sector inperiod. The only clear method to collect information yearly was the June Report, which incorporated important modifications during the 2011-2015 period, to be transformed into the ARP. Ofwat creates a scheme for data collection in PR19, and there is an initial approach to the integration of new channels of communications, such as social media. Furthermore, PR19 promotes the use of customer data and big data for the enhancement of the sector.
As the name implies, 'Implementation and Communication' searches to understand how acknowledged lessons are applied. Similar to the 'Data Collection' indicator, PR09 and PR14 offer significant information and documentation on their methodologies and their impact, yet there are no instruments to follow-up during the five-year cycle. PR09 goes further than PR14, as it mentions the yearly June Reports. PR19 offers a variety of tools for communication such as the ARP and Discover Water, a website where the companies' performance is shared with the public. Additionally, in PR19, the internet and social media are now seen as a valid source of communication between stakeholders and customers.
'Frequency' refers to the continuity of learning. In PR09 and PR14, learning is discrete and there are no clear references to learn in their methodologies. The new schemes developed by PR19, such as the ODIs, increase the learning frequency.
In summary, Ofwat has clear pathways to learn and communicate during the formulation of the PRs and for the evaluation of past PRs, however, in-period learning is lacking and there are no clear methodologies to incentivise it. PR19 introduces elements of an in-period learning approach by creating tools such as in-period ODIs and a clear reference and discussion of the APR.

Ability to respond
The PRs' ability to respond was analysed with 'Event List', 'Background and Relevance', 'Response List' and 'Resources'. Fig. 1d shows the evaluation and final scores on the ability to respond for PR09, PR14 and PR19.
'Event List' asks if the PRs have a list of events for which the sector has prepared responses. PR09 is very descriptive on the main challenges that the Sector faces and proposes a clear approach to tackle each one of these areas. However, the approach is not comprehensive and it is nuanced to financial aspects. Companies are required to provide strong delivery plans and coherence between the business plan, the WRMP and the drought management plan. This is lost in PR14, where although companies are expected to explain how they will overcome constraints, Ofwat does not explicitly request how the companies are going to respond to threats and opportunities. The creation of new tools that incentivise the companies to create responses and be prepared for unusual events is provided by PR19.
'Background and Relevance' recognises the mechanisms of selection of the relevant events and how the lists are kept up-to-date. In PR09, the key areas are in line with the challenges detected in the Sector and are based on a holistic view accounting for the needs and opinions of the main stakeholders. PR14 proposes a framework for scenario modelling and the evaluation of threats, however, the companies' business plans are not evaluated in terms of how the company is going to respond in the case of serious events. PR19 uses scenario modelling, but unlike PR14, states that the companies are expected to show actions and management responses to impacts that are predicted from scenarios, making a clearer emphasis on the response.
'Response List' refers to the evaluation of the responses and how it is determined that they are adequate for the situations they refer to. The analysis showed a progression in the assessment of how companies cope with disruption to systems and services. There are no assessment methods in PR09, and the challenges detected are generally approached in a financial way and independently. Moreover, PR14 provides information on how to develop scenario modelling but there is no emphasis on how the responses to threats and opportunities are evaluated. PR19 grows into the direct assessment of the mitigation and responses to disruptions in the company's business and strategic long-term plans.
'Resources' appraise the PRs' tools for the detection of the companies' resources to respond. The companies are accountable for what their resources are and as a part of the requirements of PR09, they must provide an asset inventory. However, PR09 is nuanced to financial resources.
Although companies must justify financeabilily, PR14 lacks consideration of the companies as a whole (operational, financial and corporate) and there are no instruments to identify resources. Different tools in PR19, such as plans and performance outcomes, foresee the direct and indirect identification of the resources available in companies.
The three PRs require companies to build their response ability on the different plans, but the instruments used were not specifically aimed at creating valid responses to threats and opportunities in the sector. PR19 is more emphatic about the importance of the responses and contingency plans, and in the evaluation of the business plans, there are specific questions devoted to their assessment. While this shows an enhancement of this ability, there is still a lack of a systematic framework that allows in-period upgrading and evaluation of these responses.

Aggregation of the results: the conclusion of the RAG analysis
The RAG analysis showed an important advancement towards a more resilient approach in the regulation for PR19 (Fig. 2). There is a clear impact of the Water Act, 2014 in PR19 and the desire of Ofwat to fulfil its duty to secure resilience in the Sector. The evolution of the four cornerstones of resilience shows that Ofwat was successful in creating tools and frameworks that should promote progress on these resilience abilities.
The PRs are stronger in their ability to monitor and to anticipate (Fig. 2). This positive development over time appears to be obvious as these methodologies are meant to regulate the companies and set the context for how the sector is going to work in the next five-year period. Nonetheless, it is important that Ofwat conceives instruments that promote learning and require the sector to have a level of responsiveness to regular and irregular threats.
PR14 appears to be a pivot between two approaches in Ofwat's regulation, as it installs a new risk-based approach and a regulation centred on the companies' outcomes. In addition, there are several new definitions on the price settlement, as this PR introduces a separation between wholesale and retail controls. As an unintended consequence of reducing regulatory burden, PR14 lacks important information such as the change between the June Reports and the ARPs and loses the link between the different stakeholders and regulators.
An important aspect that comes to light when analysing the different PRs is the introduction of new instruments and concepts every five-years. Although the justification and articulation around the new regulatory methodologies are clear, it generates a lack of continuity. However, the regulator is creating a new approach in the regulation, and the introduction of the new risk-based approach in PR14 is based on the necessity of developing a more resilient and sustainable sector.
Finally, the analysis has shown that Ofwat must create mechanisms that allow in-period analysis and development of the four cornerstones of resilience. The creation of the PRs is an interactive process, but the ability to monitor and learn appear to be focused before and after the fiveyear period. PR19 is more explicit in the description of the tools that Ofwat uses during the five-year cycle, such as the APR, and there are references to the interest in increasing the sector's transparency. Greater regularity support more uniform progress towards a more resilient future and hope to avoid the sharp changed of approach currently experienced every five-years.

Discussion
The application of the RAG method to the water sector in England and Wales, by evaluating the PR09, PR14 and PR19, has demonstrated its applicability to this industry. This systematic approach has enabled an understanding of the evolution of the resilience concept and its operationalisation in the PRs, and consequently the evolution of resilience in the sector in general.
The three main advantages of the method are its flexibility, simplicity and the easy-to-read graphical results. First, the method is flexible, and by changing the formulation of the questions, it can be applied to different organisations of the water sector. The analysis of the PRs allowed not only the evaluation of the final methodologies but also the instruments that Ofwat uses to regulate the sector. The application of these instruments could then be evaluated at a company level, by applying the RAG questionnaire to individual company's business plans. Second, the application was straightforward as the phases were explained in detail, which allowed a correct adaptation to the water sector in England and Wales. The method proved to be simple, as the ranking of the questions is based on an uncomplicated scale and the results are summarised in a visual way. Finally, the star plots constitute one of the main advantages of this method, as the systems' strengths and weaknesses are easily understood.
A possible disadvantage of the method concerns the validity of the RE framework. Hopkins (2014) argues that the abilities used in RE are based on the five cardinal features of Highly Reliable Organisations (Weick and Sutcliffe, 2001;Vogus and Sutcliffe, 2007). The definition of resilience has been a great subject of great debate, and small subtleties in the definition may result in very different approaches to regulation. In addition, the property/performance duality is not distinctively acknowledged by the method, and the metrics used could be seen as property-based. Safe & SuRe (Butler et al., 2014(Butler et al., , 2016) defines resilience as a required level of performance in the system and argues that although properties may contribute to performance, they do not guarantee it. Such similarities and contradictions with other frameworks could be an interesting point to build upon and achieve a combination of different currents of thoughts in a more holistic approach to resilience.
Another issue detected is that the method does not prescribe weighting of the four abilities (Hollnagel, 2011b). Apneseth et al. (2013) argue that different abilities could have different levels of importance depending on the system. The PRs aim to regulate the sector and to create a policy framework for the subsequent five-year period. Consequently, it was predictable that the PRs ranked higher in their ability to monitor and anticipate. However, Hollnagel (2011b) reiterates the importance of addressing all the abilities to some extent in order to achieve resilience. The different weights of the abilities could be studied in future research by using the analytical hierarchy process framework proposed by Patriarca et al. (2017). This would allow the representation of the effect of each of the resilience abilities into the overall organisational resilience.
This study is the first step in the development of a method for the sector, and consultation with stakeholders and experts is clearly required. The questionnaire presented by this study could be evaluated and changed based on the opinions of different stakeholders and experts. Later, different participants of the sector could rate the adapted version of the grid. Future research could apply the questionnaire to future PRs, polices and plans, making this method a potentially powerful tool for policy development in the water sector.

Conclusions
(1) The RAG method was adapted and applied to the water sector in England and Wales. The RAG resulted in four indicators for each of the resilience cornerstones and the addition of a new section 'Resilience Ethos'.
(2) The application of the method was successful and enabled understanding of the evolution towards a more resilient approach of the regulations and in the sector. The most relevant conclusions obtained from the RAG analysis are: (I) PR14 could be seen as a pivot between the PR09 and PR19, where important changes in the approach to the regulation were installed; (II) the results showed an important improvement of the four cornerstones of resilience in PR19; (III) although the progress is significant, the analysis revealed the need for consistent in-period performance follow-up and enhancement of the abilities.
(3) The method showed three main attractions: its simplicity, its flexibility and its graphical results. Its implementation was straightforward, demonstrated important adaptability for the systems studied and the star plots were an important instrument to exhibit the results clearly. The main disadvantages concern the validity of RE framework and its resilience definition, as well as the method's lack of prescription of the weights of the four cornerstones.
(4) This study represents a first step towards the application of the RAG method as an organisational metric tool in the water sector in England and Wales. The adapted method could be applied in previous and future PRs, as a potentially powerful tool for policy development. The final grid produced could be used for assessing the companies and their business plans, by maintaining the same indicators and modifying the questions. The involvement of experts and stakeholders in the development of the RAG would provide a more comprehensive analysis due to their insight into the sector.