Bad Piano? Bad Music? A Response to Comments on Lowe and Pemberton, The Of ﬁ cial History of the British Civil Service , Volume II : 1982 – 1997

The reforms made to the civil service during the premierships of Margaret Thatcher and John Major were unparalleled in scope in peacetime. Undertaken in the name of ef ﬁ ciency and better management, they served to Balkanise the service, shrink the number of civil servants, subordi-nate them to the will of ministers, and effectively privatise a swathe of public services. Their legacy, however, was a relatively weak centre struggling to cope with a fragmented and extremely complex governmental machine, an overly managerialist senior civil service sometimes side-lined from policy making, and a system lacking slack as a consequence of the quest for ‘ ef ﬁ ciency ’ and dependent on private contractors of sometimes dubious worth. Over the past ﬁ ve years that legacy left the country poorly placed to triumph over the challenges of Brexit and then of Covid-19.

I AM IMMENSELY grateful to the preceding three authors for their thoughts on the second volume of The Official History of the British Civil Service. In this response I address many of the issues they raise and in the process reflect on the findings of that volume in the light of the events of the past five years (the manuscript having been submitted in late 2015, before the referendum on EU membership, and subsequent, politically sensitive, Brexit-related amendments to the conclusion having then been resisted by the Cabinet Office). More recently, although Covid-19 has revealed governmental shortcomings in almost every advanced capitalist democracy, in the UK its conjunction with Brexit has created an administrative challenge on a scale unseen since 1939, and exposed problems in both policy making and public administration. Most obviously, Covid-19 and Brexit have together revealed underlying tensions between politicians and civil servants, and that the civil service was poorly prepared for the pandemic (despite earlier identifying it as the number one risk facing the country), struggled to direct and control a highly diverse set of administrative institutions, and was forced to deal with its lack of spare capacity by reflexively falling back on private sector providers and consultants, many of whom have disappointed. This should not surprise us, for the roots of these failures lie in decisions taken in the past with little regard for potentially malign consequences.
In the first volume of The Official History, the late Rodney Lowe noted that the institution faced three major challenges after 1945. 1 First, a challenge of expertise as it sought to acquire the skills necessary for it operate in a world in which government was committed to an interventionist strategy across numerous fronts on both the economy and social welfare. Second, a challenge of administration in translating such expertise into policies that were compatible, practicable and democratically acceptable. Third, a challenge for management, which was required cost-effectively to deploy the necessary policies to an ever larger and more diverse bureaucracy. Although that era saw something of a golden age for cooperative working between civil servants and their ministers (both Labour and Conservative), the volume concluded that although 'modernisation's moment' came in the 1960s (with the 1961 Plowden report significantly more important to modernisation than the Fulton report of 1968) the opportunity to overhaul the civil service to meet these challenges to greatest effect was imperfectly grasped. 2 In retrospect, we can see that the failure of 1960s modernisation had significant and long-term consequences which still condition political debate about and attitudes toward public administration in Britain. The key problem with the failure of the 1960s modernisation project was that it handily positioned the civil service as a scapegoat when politicians began to cast around for someone to blame for the country's perceived 'decline' in the post-imperial era. As the search for remedies began to push the two major political parties away from the centre in the 1970s, each came to embrace a very different ideological vision; on one thing they could agree, however, the civil service was no longer the solution to the country's major problem, it was the problem.

Civil service reform, 1979-97
As both Theakston and Diamond note in their earlier comments, the election of the Conservative government in 1979 was a decisive moment. The new prime minister wanted a radical overhaul of the civil service, an institution which Thatcher saw as resistant to change, wedded to the idea of a mushy middle-of-the-road political consensus, and more interested in its own institutional interests than in those of the nation. As Volume One of The Official History showed, she was initially stymied by the resistance of the civil service's trade unions and by the unwillingness, or inability, of its senior management to seek anything but consensus on reform of the service. Those two key impediments to radical change were swept away in 1981, however, with the failure of that year's four-month-long civil service strike and Mrs Thatcher's abolition of the Civil Service Department (created in the wake of the Fulton report to manage, professionalise, and defend the service). This opened the way for a major shrinkage of the service whilst ministers also sought to reorientate it towards a new managerial ethos better focussed on delivering ministerial objectives. The era of partnership between ministers and civil servants was over; now a new world of what one might term 'ministerial supremacy' was definitively to be ushered in.
Volume Two of The Official History, which focusses on the years between 1982 and 1997, explores the mechanics and consequences of the Conservatives' assault on the civil service and of the emergent cross-party political assumption that it had no constitutional role other than to implement the will of ministers. 3 This era is often rightly seen as one of profound crisis for the civil service, its greatest since that which precipitated the 1854 Northcote-Trevelyan report, and the end of the model of government precipitated by that earlier convulsion. The reform programme pushed through under the successive premierships of Mrs Thatcher and Mr Major stands accused of having produced the return of patronage, the erosion of the principle of appointment on merit, and the politicisation of a civil service no longer characterised by a unified, permanent and career-based cadre of staff.
Unusually for an era so closely identified with her, Thatcher's tenure as Prime Minister looms quite small in the positive civil service reform equation. Her major achievements were twofold. First, the appointment of Derek Rayner, then joint managing director of Marks and Spencer, as her adviser on improving government efficiency and effectiveness and as head of a new Efficiency Unit within Number Ten. That unit undertook a succession of often brutal departmental 'scrutinies', with ministers and their senior officials summoned to justify themselves in front of Mrs Thatcher, which served to drive down both costs and inefficiencies. Second, the 1982 Financial Management Initiative, a form of central control which was designed to determine the budgets of and clarify both the goals and output measures for individual managers; a landmark initiative albeit one that essentially built on similar initiatives in the 1960s and 1970s.
However, Thatcher's abolition of the CSD, with its functions divided between the Treasury and Cabinet Office (with the Cabinet Secretary taking on the mantle of Head of the Civil Service), produced an effective three-way operational schism-between Number Ten, the Treasury, and the new Management and Personnel Office in the Cabinet Office-which served to hollow out the centre's operational capability. Thatcher's campaign against the civil service also had other long-term effects. She had come to see the civil service as engaged merely in the management of national decline not in its reversal, as both inherently less efficient than the private sector and pursuing its own agenda, and thus a barrier to the realisation of what came to be called the 'Thatcher revolution'. As Theakston notes, her tenure in consequence was a profoundly uncomfortable one for civil servants, who saw their pay squeezed, their conditions of work deteriorate, and their number shrink, with all that implied for plummeting morale and problems with recruitment.
Another long-term effect of Mrs Thatcher's war on the service, and her desire for it to cast aside its former model of partnership with ministers and accept its full subordination, was that it forced Robert Armstrong, the then Cabinet Secretary and Head of the Civil Service, to codify the role of the modern civil service in 1987. In the so-called Armstrong memorandum he set out its constitutional position: 'Civil servants are servants of the Crown. For all practical purposes the Crown in this context means and is represented by the Government of the day. … The Civil Service as such has no constitutional personality or responsibility separate from the duly constituted Government of the day.' And he laid out the duties of civil servants to ministers thus: The Civil Service serves the Government of the day as a whole, that is to say Her Majesty's Ministers collectively, and the Prime Minister is the Minister for the Civil Service. The duty of the individual civil servant is first and foremost to the Minister of the Crown who is in charge of the Department in which he or she is serving. 4 Egregiously, the Armstrong memorandum, whilst it reaffirmed the civil service's duty of impartiality, made no mention of a hitherto uncodified function: the service's permanence requiring it to serve and retain the trust of governments of different political complexion over time, thus ensuring that it functioned as a constitutional counterweight to the typically short-term nature of ministerial tenure. Armstrong also dodged the thorny question of 'the national interest' in the context of a breakdown of two-party politics and the increasing chance of large Commons majorities being delivered on relatively small vote shares and turnouts. In short, the memorandum had subtly recast the civil service's function within the constitution and, crucially, established what was effectively the new doctrine of 'ministerial supremacy' even as the political legitimacy of ministers was coming into question.
Most of the positive agenda of civil service reform in this era came not under Thatcher but under her successor, John Major, although the seed of probably the most significant, the Next Steps initiative, was sown under Rayner's successor, Robin Ibbs, in the Efficiency Unit's 1986-7 'scrutiny of scrutinies'. 5 The resulting Next Steps programme of reform, the main administrative success story of the 1990s, unfolded between 1989 and 1998, but with most of the action occurring from 1990 to 1996. The Next Steps analysis was a very different one from that of Fulton in 1968, for it concluded that the civil service was too big and too diverse to manage as a single entity. The resulting initiative inaugurated a new world of arms'-length management of semi-autonomous 'executive agencies' that would themselves decide on the most effective means by which the objectives set for them by ministers might best be achieved. In the process of implementing this agenda the civil service was fundamentally reshaped, the most obvious effect being the progressive transfer of staff out to Next Steps agencies such that by 1997 only a quarter of civil servants worked in 'Whitehall' departments. Next Steps placed the UK in the vanguard of the 'new public management' revolution (although it owed remarkably little to academic thinking within that political science sub-discipline) and was widely seen as a success across the British political spectrum and abroad.
At its margins the Next Steps project overlapped with privatisation and, via the Competing for Quality agenda from 1991, particularly with the contracting-out to private companies of services hitherto provided by central government. 6 That programme's initial target of market testing £1.5 billion of publicly provided services was not achieved, but by 1996 it had managed £1 billion with the private sector winning 53 per cent of those tests, and 71 per cent of their value. 7 Thus, whilst ministerial hopes of a wholesale privatisation of market-tested functions were disappointed, a substantial amount of effective privatisation took place. Competing for Quality persisted despite the election of a Labour government, which could remark in its 1999 Modernising Government White Paper: 'We must not assume that everything government does has to be delivered by the public sector'. Thus reliance on private contractors was to be an enduring change. 8 By 1992, Stephen Dorrell, then Conservative Financial Secretary, could observe that the achievements of privatisation and civil service reforms had transformed the agenda, replacing, 'what can we sell?', with the question 'what must we keep?'. 9 Yet, for the first time, this represented an acceptance that some services were never going to be amenable to contracting out or privatisation, and this assumption lay at the heart of John Major's Citizen's Charter in 1991. This focussed on raising the quality of public services, which should be 'run to suit the convenience of customers, not staff'. 10 In seeking to shift power from public sector 'producers' of service to the people who consumed them, the charter ran with the grain of existing policy in that privatisation and Next Steps had sought in different ways to devolve decisions about service delivery away from Whitehall and closer to the final consumers. At the same time it challenged long-standing civil service culture such as the assumption that those using public services were (and perhaps should be) essentially passive; its tradition of top-down service design and delivery; and an assumption within the Treasury of a zero-sum equation between expenditure and standards. 11 Almost entirely failing to capture the public imagination, and much derided at the time and since, in fact the Citizen's Charter had significant effects. Not the least was that it was enthusiastically embraced by front-line public servants, who were attracted by the idea of providing higher quality service. Likewise, though after quite a significant time lag, senior civil servants too came to recognise that whilst the charter challenged prevailing civil service norms and practices, it also embodied an opportunity via its acceptance of an inescapable core of public services and positive engagement with civil servants (civil service unions, entirely antipathetic to the government's broad agenda on the service, welcomed the charter). Shortly before the 1997 general election the House of Commons' Public Service Committee concluded that by working with the grain of both consumer culture and the desire of public servants to do a good job the charter had helped to transform service quality. 12 Far from a government blunder, therefore, the charter came to be seen as an example of where a 'government policy is widely regarded as risible when first introduced but turns out in the fullness of time to have been a considerable success.' 13 Over time, however, the impact of the charter on quality declined as the initiative gradually morphed, under pressure from the Treasury, away from qualitative targets to a plethora of quantitative targets focussed on cost and efficiency. Thus 'value for money' came once again to trump quality, and the civil service definitively embraced a culture of management by target.
With Major's acceptance of a continuing role for public services came an opportunity for the civil service to stem the tide of reform that was shrinking, reshaping and fragmenting British government. An explicitly federal civil service was taking shape, calling into question the historic commitment to maintain a unified and uniform service. As Theakston observes, staff were demoralised and demotivated and there were also fears both inside and outside the service that its traditional values were being undermined, that its politicisation was becoming a problem, and that lines of political accountability were becoming unclear. Political expectations of the civil service had also changed, with an increasing assumption in both major political parties that its purpose was simply to implement the policy of the party in power. Such changes raised deep questions about the role and function of the civil service.
Robin Butler, the then Cabinet Secretary, sought to deal with these issues by persuading Major that it was time for a White Paper on the Civil Service, which duly appeared in 1994 with the telling title Continuity and Change. 14 The continuities were intended to be the civil service's time-honoured traditional values of integrity, political impartiality, objectivity, selection and promotion on merit and accountability through ministers to Parliament. The most significant change other than the further embedding of the 1990s reform agenda was the heralding of a new 'senior civil service'. This was a defensive move instigated by Butler, but a successful one in that it effectively ring fenced this elite cadre whilst also securing ministerial endorsement for the civil service's 104 H U G H P E M B E R T O N role in both policy making and the delivery of public service. In doing so, however, it did nothing to challenge the doctrine of 'ministerial supremacy'. It also institutionalised the logic of the core-periphery model inherent in Next Steps and, effectively, recreated the distinction between administrative and executive classes that had been eliminated after Fulton. 15 Legacy By 1997, such had been the pace and scope of change since 1979, the civil service's basic structure and mode of operation were almost unrecognisable. It was also a much smaller institution than it had been, having lost a third of its complement since 1979. The pursuit of 'efficiency' in these years improved civil service productivity, which was welcome, but despite a brief emphasis on quality in the wake of the Citizen's Charter White Paper the relentless focus on driving down costs had deleterious effects. A smaller service with less slack was not necessarily a more capable one, or one that was better at delivery. Nor were the demoralisation of civil servants and constant political attacks on the service likely to improve either delivery or quality, not least because they had begun to affect recruitment and retention of high-quality staff.
Was the service better managed? More attention was paid to management and to the acquisition of management skills than hitherto, certainly, but for all the reforming activity between 1987 and 1997 the hole at the centre was not filled. Nor did reform of that centre keep pace with the array of changes demanded of it. The logic of Conservative reforms since 1979 demanded the decentralisation of public services whether via the creation of Next Steps agencies, contracting out, or outright privatisation with public regulation. It was an approach that implicitly recognised that, in the language of new public management, the business of the centre should increasingly be about 'steering not rowing'. That demanded a shift at the centre from day-to-day tactical management to medium-and long-term strategic management, but it also required that the centre retain strategic control of an increasingly fragmented system with multiple centres of power and responsibility. It struggled to achieve either, despite the recentralisation of managerial responsibility for the service with the transfer of central management functions to the Cabinet Office (thus effectively recreating the Civil Service Department which Mrs Thatcher had abolished in 1981).
In addition, fragmentation had unforeseen side effects. First, ministers' faith in private sector contractors and their assumption that it must necessarily be able to provide better services at less cost proved to be misplaced. This was clear by 1997 from surveys of staff and consumers of public services which indicated that contracted-out services were generally no better, in fact often worse, than those delivered 'in-house'. Fragmentation therefore began to degrade service quality. Second, fragmentation demanded very different forms of management: by ministerial definition of operating frameworks and priorities in the case of executive agencies; or by legal contracts with private providers of contracted-out public services. In each case the centre specified objectives and delegated decisions on delivery. But this was a much more inflexible form of management, particularly in the case of contracted-out services where the need to account properly to Parliament for the spending of public money demanded detailed contracts and, crucially, any change of political or administrative priorities required (generally expensive) contractual change. Thus, service delivery was now less flexible; change was slower; and over time delivery was more expensive.
In short, although the UK was the poster child of the new public management revolution of the late 1980s and 1990s, its achievements were slight. Hood and Dixon later concluded that the overall effect of all this activity was that government probably worked a bit better, but far from being cheaper, it actually cost a bit more. 16 Our research corroborates that analysis. But we also find growing concerns at senior levels of the civil service about management by contract, and fears that the Balkanisation of the civil service had caused departments to lose contact with the reality of front-line service delivery to the detriment of both good policy and effective management.
A harder judgement concerns claims about degraded political accountability and politicisation. Certainly, delegation to executive agencies of responsibility for day-to-day decision B A D P I A N O ? B A D M U S I C ? 105 making did raise the question of who was responsible when things went wrong. The intention was that agency chief executives would answer directly to the relevant select committee; but for all Robin Butler's attempt to distinguish between accountability and responsibility, the committees (and the media) proved most unwilling to accept this and continued to hold ministers accountable for agency failings. The problem was exemplified by the 1995 furore over IRA escapees from prison, which resulted in the Home Secretary sacking the chief executive of the Prison Service (even though he had a good case that the service had actually performed well in achieving ministerial targets).
As Diamond observes, politicisation had undoubtedly increased by 1997. This was driven by a multitude of factors, including the long tenure of officials such as Bernard Ingham who were publicly identified with the political project, the growing influence of special political advisers (SpADs), and the tendency for 'can-do' civil servants to be favoured for promotion at the expense of those more willing to speak truth to power. We should also note that disinterested advice from civil servants was increasingly unwelcome to ministers, who more and more drew their advice from their special advisers and from politically sympathetic think tanks. Again, these proved to be enduring changes. Moreover, growing politicisation was inevitable because of the new doctrine of ministerial supremacy, for by design it demanded a more supine civil service.

Conclusion
It would be comforting to think that the changes described above are of merely historical interest; that would be a mistake for, as Diamond notes, they had malign long-term effects. Whilst things did not stand still, there was, for example, some rolling back of the executive agency model as part of a reassertion of central control, the greater complexity of the administrative apparatus and its greater fragmentation persisted, and persists. So too did the weakness of the centre, highlighted by all three commentators above, which consistently struggled to reconfigure itself adequately to adapt to strategic management of a much more fragmented and complex machinery.
The weakness of the centre became more glaringly obvious as British politics became increasingly focussed on the Prime Minister but, as Rutter notes in her commentary, Number Ten remained relatively under-resourced (for all the attempts of the Blair administration to address this). And whilst successive cabinet secretaries told their Prime Minister that the Cabinet Office could effectively act as a Prime Minister's Department, it remained focussed on the day-to-day coordination of administration, formally responsible to the Cabinet as a whole rather than to the Prime Minister, and with a surprising lack of powers to direct departments-which remained ministerial fiefdoms. 17 What was less obvious was that in the new world of 'ministerial supremacy' there was a much greater need for ministerial competence. As one former permanent secretary put it in 1992, it mattered little how good the civil service piano was if the ministerial music it was required to play was not well written. 18 Ministers were convinced the problem lay with the piano; they did not consider whether they were good composers. Yet, unfortunately, ministerial competence was often lacking, not least because of the often relatively short tenure of many ministers, and concerns about fundamental quality have also mounted in recent years.
Since 2016, Britain has confronted two major and recently parallel crises in the shape of Brexit and Covid-19 which, taken together, amount to by far the biggest challenge to British public administration since 1939. The challenge was all the greater for the election in December 2019 of a government profoundly antipathetic to the civil service and to its values of integrity, political impartiality, objectivity, selection and promotion on merit and accountability through ministers to Parliament. In meeting the challenges of the past five years, and particularly the past year or so, it has to be said that many ministers have been found wanting (one notes, for example, that the executive agency Public Heath England entered the Covid-19 emergency with its key ministerial priority being Brexit and with no ministerial priority attached at all to pandemic planning). 19 We should acknowledge, therefore, that the reforms of the 1980s and 1990s played a key role in weakening the effectiveness of British public administration. A