Decision making in NICE single technological appraisals: How does NICE incorporate patient perspectives?

Abstract Context The National Institute for Health and Care Excellence (NICE) has an explicit mandate to include patient and public involvement in the appraisal of medicines to be available for funding on the NHS. NICE involves an appraisal committee who are required to take on board experiential evidence from patient experts alongside population‐based evidence on clinical and cost‐effectiveness when making a decision whether to fund a drug. Objective This paper considers how NICE Single Technological Appraisal (STA) committees attempt to incorporate the views of patients in making decisions about funding medicines on the NHS. Methods A prospective design was employed to follow three pharmaceutical products involving three different appraisal committees. Three data collection methods were used: analysis of documentary evidence sent by NICE, non‐participant unstructured observations of the open and closed sessions of meetings and qualitative interviews. Settings and participants Unstructured non‐participant observations were carried out at nine STA meetings, and 41 semi‐structured interviews were undertaken with committee members from NICE's STA committees, patient experts, analysts from NICE's project team and drug manufacturers. Results Our analysis showed how the committees displayed a preference for an ideal‐type of patient representative, disagreement among the committee when weighing‐up patient statements in the STA process and more pre‐preparation support for patient involvement. Conclusions Although NICE has attempted to adopt an approach flexible to patients and carers through formal decision‐making arrangements that incorporate patient views, nonetheless, the processes of the STAs can in fact undermine the very evidence collected from patient representatives.

however, the politicization of these decisions has resulted in NICE facing much criticism and challenges to its legitimacy. 3,4 STAs are one of NICE's decision-making processes in which evidence about a selected technology (often medicines) is evaluated in three distinct phases (scoping, assessment and appraisal). In the last phase of this process, an independent appraisal committee evaluates evidence in a meeting (or multiple meetings), partly held in public but with the latter half taking place in a "closed" session. During the meeting(s), the committee considers evidence based on clinical and cost-effectiveness, as well as statements expressed by patients, commissioning experts and clinical specialists. 5 The Institute encourages patient experts attending the meeting to provide both written and oral commentary about their personal view in the current management of the condition and the expected role and use of the technology-in particular how it might provide benefit to patients.

| NICE incorporating patient perspectives
Patient and public involvement (PPI) within the planning and development of health services is by no means a new concept in the domain of user involvement; 6 in particular, PPI has in recent years in the UK become a growing feature in the organization and delivery of health care. [7][8][9][10] Since the mid-1990s there have been moves towards an, "open, accountable and patient-centred service and an attempt to establish the involvement of service users in health-care services," and within the UK NHS, there have been a number of initiatives giving individuals and groups a stronger voice within the health service in activities such as planning and development, extensive lobbying within hospital Trusts 11 and involving the public in commissioning decisions and strategies. 12 Although PPI has become increasingly prevalent, relatively little is known about which approaches work best, when or why, or under what circumstances better outcomes can be achieved making it difficult to draw conclusions about the effectiveness of different involvement approaches. 13 One of the key aspects underpinning such uncertainty is how PPI contributes to decision making in the review and appraisal of new drug therapies. Researchers have questioned whether the arrangements for engaging and involving users in the decisionmaking process are effectively addressed through formal structures set up by organizations such as NICE in their appraisals processes. 14 This paper considers the contribution patient experts make in NICE's STA committee meetings, identifies what level of participation patient experts currently have, and what types of health-care decision-making contexts or domains the current model of involvement reflects.
The difficulties in incorporating patient perspectives when appraising new drugs are noted by Milewa. He explains how in one respect NICE is charged, "with the production of evidence-based deci-sions…But in a second regard NICE is politicized in that it has to afford interested parties access to the decision-making process, demonstrate cognisance of these myriad voices and reflect transparency in attendant rationales for its rulings." 15 Thus, NICE is required to demonstrate that it has consulted with various active specialists-not only typical experts including clinicians, health economists, NHS managers, pharmacists and researchers from health technology manufacturers, but patient groups who provide detailed witness statements. Milewa argues that inclusion of various active patient participants, "appears to constitute a more inclusive, deliberative, approach to decision making." In pragmatic terms however, Milewa describes the difficulty in implementing these two positions, as NICE attempts to balance these two decision-making logics-reasoning according to explicit criteria (clinical and cost-effectiveness) and upholding an equality of status with all participants in the appraisal process. In his 2005 qualitative study, Milewa found that clinical and professional actors (as noted above) appeared to play a major role in debarring, reshaping or admitting the more subjective accounts or experientially based forms of evidence submitted by patients and carers. 16 In the last 5 years, policy reforms in the NHS and in social care organizations denote a far greater expectation that patients, service users, carers and the public will be involved in decisions regarding the delivery of health-care services, as reflected in four key policies: the Health and Social Care Act (2012), the NHS Constitution (2012), "Putting people at the heart of care" (2009) and "Essential Standards of Quality and Safety" (2010). In line with these wider tendencies in English health policy, 17 NICE has outlined its own key principles to promoting PPI: • That lay people, and organizations representing their interests, have opportunities to contribute to developing NICE guidance, advice and quality standards, and support their implementation, and • That, because of this contribution, our guidance and other products have greater focus and relevance for the people most directly affected by our recommendations. 18 Through its 2013 policy paper on Patient and Public Involvement Policy, NICE's commitment to involving patients, service users, carers and the public appears integral across its range of guidance, products and quality standards documentation. In relation to STAs, NICE's continued commitment to ensuring this level of PPI appears evident. For every STA, NICE invites submissions from all patient and carer groups with interests relevant to the appraisal, and their written and oral statements must be taken on board by the committee involved in the decision making.
In the context of technological appraisals the main purpose of qualitative research is to explore areas such as patients' experiences of having a disease or condition, their experience of having treatment and their views on the acceptability of different kinds of treatments 5 NICE specifically outlines the areas where patient and carer perspectives would provide the most benefit to the appraisal committee; this includes consideration of both "majority views and views held by only a few patients even if they contradict the majority"; therefore, NICE appears to invite a spectrum of patient views to be taken into account, even in contexts where views may differ. In addition, NICE states that the appraisal committees are keen to incorporate patient perspectives in areas where there are gaps in published research that do not adequately capture outcomes that are important to patients, and help gauge valid clinical outcomes and standardized generic instruments in measuring quality of life to the disease or condition. 5 Therefore, NICE and its committees might find themselves in a potentially incongruous position: how to take on board the experiential evidence from individual experts alongside evidence on clinical and cost-effectiveness when reaching a decision about whether or not to recommend a drug as cost-effective for NHS use. economists, statisticians and public health practitioners), as well as lay experts. NICE therefore seeks to appoint members of the committees based upon their relevant experience or their specific technical skills. 19 In terms of the patient experts asked to provide witness statements to the STA committee meetings, the Public Involvement Programme (PIP) manager contacts them in advance to explain what their role is, to complete their personal statement and prepare them for the meeting. 20 While there is a growing emphasis on patient involvement in NICE technological appraisal meetings, nonetheless prospects for greater participation remain somewhat unclear, and therefore, the expectation and assumption of what patient experts may bring to the decisionmaking process varies considerably. Charles and DeMaio present a conceptual framework based on three elements of a decision-making domain (treatment, service delivery and broad macro-or system level decision-making contexts), which incorporates different role perspectives (a user of health services, and a public policy perspective), taking into account a range of levels of participation (consultation, partnership and lay control), thus providing a step towards understanding the contribution lay participation may make in health-care decision making. This framework helps to delineate the types and levels of lay involvement that take place at STA meetings (discussed further below), and the implications of this involvement for decision making at the meetings. 6 In other aspects of its work, NICE appears to have a broader commitment to involving patient experts (such as expert eyewitnesses from the voluntary or community sector) through its PIP offering practical and detailed advice on how individuals can be involved in NICE's individual programmes in public health, health technologies, clinical practice or guidance products. 21 While NICE's express commitment to incorporating PPI is evident, there have been notable barriers to patient experts' participation in NICE technology appraisal meetings.
In a survey published by NICE in 2012 of patient experts, respondents reported that the level of patient expert participation varied across different STA committees, and that their contribution to the appraisal meetings was largely dependent upon the committee Chair. 20 Gibson and colleagues argue that in order for lay engagement to gain full recognition to complement the analytic and reductionist approaches of scientific reason, professionals, including the committee Chair, must engage in deliberations outside their traditional professional terrains.
Thus, the value provided by lay experts is dependent upon the committee Chair to provide "knowledge spaces" to help establish a new terrain where experiential evidence, interpretations and opinions, and on the other hand, rigorous evaluations and scientific rationality, come together to build "creative dialogues to inform health-care policy decisions." 22 Patient experts also reported that the meetings were "very large, formal and with a lot of technical language, and that the emphasis on clinical and cost-effectiveness overshadowed patient issues." 20 They described not feeling able to contribute to issues of paramount importance, and they did not feel sufficiently sure or confident of their role in the meetings. Worryingly, some patient experts felt intimidated at meetings and, less surprisingly had difficulty following the presentations. The barriers to lay persons' participation are noted by Gibson and colleagues in their examination of the concept of PPI. They observed how difficult, if not impossible it was for lay persons to achieve participatory parity in the presence of professionals. They argued that current PPI statutes may instil a semblance of equality in participation, nonetheless in specific meeting settings, "the abstract and technical forms of knowledge that professionals tend to have acquired through formal education are highly valued than the practical, experiential knowledge that patients or members of the public may possess." Unfortunately when lay persons sit on prestigious medical committees their contribution may be unconsciously undermined, because "they do not have access to dominant forms of capital that professionals have access to." 22 The patient experts felt that their presence at meetings was largely "tokenistic" and that the patient experience had little to do with the committee's overall decision. 20 The literature has emphasized such tokenism as one key reason why PPI has thus far had limited influence on the organization, planning and delivery of healthcare services, despite its emphasis in recent policy. Yet, in exploring the role of patients in NICE appraisal decision making, it is important to go beyond this narrative to consider the possibility of greater influence as well as to nuance analyses of the process of marginalization where this takes place.

| METHOD
The study used a prospective design to follow three pharmaceutical products through the STA process, which were chosen for variation in the socio-cultural resonance of the illness they were designed to treat. The final selection included: a drug treating a less "prominent" but prevalent chronic illness (Case Study X); a drug treating a high profile condition (Case Study Y); and lastly, a drug which treated a rare but life-threatening condition (Case Study Z). 23,24 The study used an ethnographic research approach using three distinct methods of data collection: first, an analysis of documents released by NICE on each drug/therapy made available to the public online, documents submitted by the pharmaceutical company and review documents prepared by independent academic advisors.
Documentary analysis was undertaken to identify points of contention, areas of uncertainty in the data, as well as providing the researchers contextual information about the drug under assessment.
Second, for each drug appraisal, semi-structured interviews were conducted with a purposive cross-sectional sample of the key informants (see Table 1). Forty-one interviews, both face-to-face (15) and telephone (26) (three with NICE staff plus 14 in case study X, 12 in case study Y and 12 in case study Z), were carried out. Each interview lasted between 45 and 90 minutes, was voice-recorded and was then transcribed. The semi-structured interview guide covered topics which probed around their experiences of the meeting, how informants dealt with issues of uncertainty, whether they felt there were influences in the wider social-cultural environment that influenced decision making, other external influences in the appraisal process on decision making, and how they found working with NICE as an organization.
Lastly, nine non-participant unstructured observations of open and closed sessions of the committee meetings were carried out across the three case studies. Observational data were collected by two researchers who attended each meeting; one researcher recorded detailed discussions from the meeting verbatim, while the second researcher collected observational data on key discussions and debates, including any major points of uncertainty. It was not possible to utilize a structured observational schedule due to the technical nature of discussions. The field work was carried out between 2012 and 2014.
Three researchers were involved in the data collection activities for the documentary analysis, interviews and non-participant observations. All three members of the research team brought with them a range of meaningful perspectives, experiences and standpoints which influenced the research process in terms of data collection and analysis (see Analysis section below). 25

| Patient and public involvement
The original study was funded by the Economic and Social Research Council (ESRC) on a project entitled "Managing uncertainty within NICE technological appraisals: the nature and impact of the 'social features' of decision-making." While the proposal originally set out to study difficulties in reaching decisions due to multiple logics and inputs, the specific issue of patient involvement was a theme that emerged from the data in the subsequent analysis phase. Thus, further data on how much experience the participants had of PPI in research and in the overall STA process were not covered in the study's objectives.

| Ethical considerations
Ethical approval for the study was provided by the Ethics Committee The main ethical issues encountered concerned potential identification of participants, in particular as membership of each of the committees is published on NICE's website. It was ensured that reference to the drug under appraisal, the corresponding committee and any information relating to the participants (including age, gender, years working on STAs) was removed to ensure anonymity.

| Access and recruitment
The participating departments of NICE gave their approval for the research to be carried out. The initial approach was then made to the committee Chairs who provided an agreement by which we gained permission to undertake the non-participatory observations; in addition, individual consent was obtained from the committee members for the closed part of the meetings.
The first contact to recruit interviewees was made face-to-face by the researchers who approached all potential participants including NICE committee members, as well as external contributors such as patient experts, clinical experts, members of specialist assessment groups and representatives from the pharmaceutical industry (see Table 1). Consent was taken from each of the informants.

| Analysis
In the analysis presented below, it was considered how the commit- The analysis undertaken focused upon manifest themes, which were themes drawn directly from the interviews and observable content within the data set, rather than latent themes, which are tacit deductions, where participants refer to social distances from certain groups. 28 Daly et al. 29

| Contextualizing the disease alongside the benefits of the drug
The relevance and importance of patients' perspectives were recognized by the committee members, who generally expressed that these views were central to enable a complete appraisal of the drug technology under assessment:  1 We have withheld reference to each specific case study for the interviews with informants in order to preserve anonymity.

| Conflicts in decision making and credibility
Alongside the usefulness of these insights, however, committee members reported that patient accounts were also an area of disagreement between members of the committee: The comment above suggests that when patient experts were brought in to discuss the nature of the disease, rather than whether the drug had an effect, this was less than helpful and raised doubt about the credibility of the patient expert instead.
As noted above, the power and credibility of patients' testimonies led to some misgivings about their influence. 30 A further concern noted by several committee members was that they felt mistrustful of the patient experts due to how they were selected. Several accounts suggested that the patients were often selected via patient groups, who in turn were seen as having close links with manufacturers due to their connection with the manufacturer for example through receipt of funds from the pharmaceutical industry: More often than not, committee members were suspicious and disapproving of patient experts and patient representative organizations who were seen to have a close association with the drug manufacturers.
Committee members also seemed to feel that patient experts' views were too narrowly concerned with having the drug available, rather than seeing the bigger picture of rationing and the broader consequences for the NHS.

| Tokenism
The patient experts themselves were mindful of their inclusion at the appraisal meetings and acknowledged that they were in many respects, merely representatives without a dominant presence or with any significant degree of persuasion in the final decision: The two patient experts quoted above were commenting on two separate STA appraisals, yet, both referred to feeling marginalized and surmised that their viewpoints were not being taken into account. They both questioned why they were invited to attend when they felt they "weren't needed to be there." The first participant described feeling quite aggrieved by the way she was interrupted when asked to comment.
In contrast, another patient expert's contribution at the meeting was commended as particularly compelling as she was able to articulate in the language appropriate to the committee the problems with the disease as well as her recovery after taking the medicine (Observation notes). In an interview with this patient expert, she reflected positively about her experience and felt the committee were receptive to her account. She was also conscious that she was not a typical patient and presented herself well: In this sense, the individual characteristics and capital of the patient expert were influential, while the role of the Chair was also influential.
Within our observations, the Chairs appeared conscious of the need to demonstrate publically that they welcomed the views of patient experts: The actions of the Chair reflect how it was necessary to acknowledge the contribution of the patient experts of how they viewed the drug, in this case, as innovative. In another case, the Chair drove a discussion regarding the plausibility of the quality of life model (which initially was questioned) by advocating that the committee would be more flexible in accepting this model (even though the case was not entirely clear) as a way of reflecting the compelling arguments put forward by the patient expert.

| More support for incorporating patient perspectives
We have already noted that the characteristics and conduct of individuals (patients or Chairs) could shape the relative influence and input of patients: The question posed by the committee member about patient progress was quite unexpected and the patient expert felt compelled to remind the committee that she did not have the expertise to comment, which in turn could be seen as publically devaluing her presence at the meeting.

| DISCUSSION
In the analysis presented above, the aim was to consider how NICE incorporates patient perspectives within the framework of the STAs process. Although NICE provides the formal structures and processes in its appraisals for involving and engaging patient groups, the role of such groups is confined to the realm of "representation" rather than that of a key stakeholder in decision making. 14  In many respects, patient experts are tasked with performing a conflicting role where they are expected to present themselves as a credible patient while at the same time performing the role of a charismatic patient representative. In this study, it was found that patient experts who appeared at committee meetings were atypical, as they had been able to contribute to formal meeting settings, and, those invited did not reflect the archetypal patient. Patient experts who attended the meetings had two main reasons for their participation, the first as a patient sufferer with the condition or disease, then as the enigmatic storyteller or marketer advocating the patient group. In these circumstances, the committee placed the patient experts in a position where they foresaw their contribution as an aid to reaching a decision in a neutral public forum, but when presented with a patient statement, they found their impassioned accounts too emotional or hard to handle. 35 Evidence from this study did show that the overriding priority of the committee was to appraise the technology in terms of costeffectiveness and the extent to which it fell in or outside the threshold set by NICE. 24 In one of our cases, the technology was far too expensive and a decision not to recommend was made after two meetings which may explain why there was limited patient expert involvement.
However, in the other two cases, there was considerable more uncertainty about the incremental cost-effectiveness ratio and possibly therefore more space for patient expert influence.
Notwithstanding the study showed that throughout the process for all three cases, social influences on decision making were both explicit and implicit, suggesting that the discourse on cost-effectiveness was overriding and apparent in formally documented meetings, but contributions from patient experts also had a tacit impact on the appraisal throwing open debate between committee members around rigorous evidence vs patient witness statements. This study also only focused on three of the four appraisal committees, although there is no reason why the one not included should adopt a different approach to the importance of incorporating patient expert perspectives.
Although NICE has implemented some pioneering initiatives to incorporate patient perspectives, these initiatives have left some patient groups feeling that their role at the STAs has been largely peripheral and perhaps even tokenistic. Some patient representatives questioned whether they would accept an invitation to attend another meeting in the future. Patient experts described feeling unsupported in their attendance at meetings and were in some instances dismayed that other lay experts would not be able to contribute to meetings given the technical nature of discussions. 20 NICE's attempts to engage with patients experts in STAs have helped to exclude the very groups the appraisal system has intended to involve. While NICE declares values and aspirations appearing to be open, transparent, participatory and pluralist in its involvement structure, yet in practice, the process is a closely managed, unitary system and a top-down model, where salaried professionals run a highly regulated public participation process.
Their approach to public involvement has been predominantly technicist or an instrumental approach, without any overriding concern for a pluralist involvement structure where a diversity of views can be publically articulated. 22 It has been shown that the role of patient representatives at appraisal meetings differed considerably-while some patient experts were described as helping the committee understand the disease/ condition, in other respects they were also seen as advocates of the new drug therapy. It would appear that NICE needs to provide much greater guidance and clarity over the roles and contributions it expects patients to make and how their statements and submissions might fit into the decision-making framework. 6 The blueprint for positive outcomes and greater impact for PPI is recommended in the RAPPORT study by Wilson et al. who put forward six salient actions for effective PPI, the first of which is aimed at defining the purpose, role and structure of PPI. Having a clearly defined role is paramount for NICE to be able to carve out a bespoke function for patient experts, to understand where in the decision-making framework patient experts reside and what purpose their contribution makes at the appraisal meetings and within wider STA deliberations. 36

| CONCLUSIONS
The evidence from this study has shown that patient experts have