Involving patients and caregivers to develop items for a new patient‐reported experience measure for older adults attending the emergency department. Findings from a nominal group technique study

Abstract Context Patient experience is an important component of high‐quality care and is linked to improved clinical outcomes across a range of different conditions. Patient‐reported experience measures (PREMs) are psychometrically validated instruments designed to identify where strengths and vulnerabilities in care exist. Currently, there is no validated instrument available to measure patient experience among people aged over 65 years attending the emergency department (ED). Objective This paper aims to describe the process of generating, refining and prioritising candidate items for inclusion in a new PREM measuring older adults' experiences in ED (PREM‐ED 65). Design One hundred and thirty‐six draft items were generated via a systematic review, interviews with patients and focus groups with ED staff exploring older adults' experiences in the ED. A 1‐day multiple stakeholder workshop was then convened to refine and prioritise these items. The workshop entailed a modified nominal groups technique exercise comprised of three discrete parts—(i) item familiarisation and comprehension assessment, (ii) initial voting and (iii) final adjudication. Setting and Participants Twenty‐nine participants attended the stakeholder workshop, conducted in a nonhealthcare setting (Buckfast Abbey). The average age of participants was 65.6 years. Self‐reported prior experiences of emergency care among the participants included attending the ED as a patient (n = 16, 55.2%); accompanying person (n = 11, 37.9%) and/or as a healthcare provider (n = 7, 24.1%). Results Participants were allocated time to familiarise themselves with the draft items, suggest any improvements to the item structure or content, and suggest new items. Two additional items were proposed by participants, yielding a total of 138 items for prioritisation. Initial prioritisation deemed most items ‘critically important’ (priority 7–9 out of 9, n = 104, 75.4%). Of these, 70 items demonstrated suitable inter‐rater agreement (mean average deviation from the median < 1.04) and were recommended for automatic inclusion. Participants then undertook final adjudication to include or exclude the remaining items, using forced choice voting. A further 29 items were included. Thirty‐nine items did not meet the criteria for inclusion. Conclusions This study has generated a list of 99 prioritised candidate items for inclusion in the draft PREM‐ED 65 instrument. These items highlight areas of patient experience that are particularly important to older adults accessing emergency care. This may be of direct interest to those looking to improve the patient experience for older adults in the ED. For the final stage of development, psychometric validation amongst a real‐world population of ED patients is now planned. Patient and Public Contribution Initial item generation was informed using qualitative research, including interviews with patients in the ED. The opinions of patients and members of the public were integral to achieving outcomes from the prioritisation meeting. The lay chair of the Royal College of Emergency Medicine participated in the meeting and reviewed the results of this study.


| INTRODUCTION
Patient experience is an important component of high-quality, patientcentred care and is associated with improved outcomes for a range of acute conditions including pneumonia, acute coronary syndrome and asthma. [1][2][3] Older adults currently account for about a quarter of emergency department (ED) attendances and this proportion is likely to increase further given the ageing global population. 4,5 Older adults may have a range of additional care requirements and psychosocial needs when accessing emergency care, compared to younger adults. 6,7 Capturing older adults' experiences of care may identify where vulnerabilities and subsequent opportunities for improvement in the provision of emergency care exist.
Patient-reported experience measures (PREMs) are validated, self-reported questionnaires that are directly reported by patients and aim to provide standardised evaluation of individual experiences of care. PREMs differ from patient-reported outcome measures (PROMs), which measure patients' views of their health status, and satisfaction surveys, which measure to what extent care meets patients' subjective expectations. 8,9 Hodson and Roberts 10 suggest that patient satisfaction measures often exhibit a ceiling effect, whereby responses are predominantly positive. Hence, satisfaction surveys may be less likely to identify negative determinants of experience compared to PREMs. This is important, as negative determinants of experience may represent particularly useful areas for performing quality improvement. As such, the use of PREMs to capture patient experiences of emergency care is suggested within the International Federation of Emergency Medicine framework for quality and safety in Emergency Medicine. 11 However, a systematic review of existing PREMs in emergency care determined that there was significant variation in the quality of existing instruments, including uncertain validity, reliability and responsiveness. 12 These findings are reflected in a further systematic review of 88 PREMs which reported inconsistent adherence to established criteria for the selection of health instruments. 13,14 Recently, PREMs have been developed to capture older people's experience of hospital and community care, although no instrument specific to the ED yet exists. 15,16 The PREM for patients attending the ED, aged over 65 (PREM-ED 65) aims to address the current gap, by developing and validating a PREM for use in older adults accessing emergency care. The first stage of PREM-ED 65 development aimed to generate a comprehensive understanding of determinants of older adults' experiences of receiving ED care. Initially, a systematic review of qualitative studies was conducted leading to the formulation of a conceptual framework for patient experience in the ED. 17 This framework highlighted the importance of meeting patients' communication, emotional, care, physical/environmental and waiting needs. Confirmation of conceptual validity and expansion of the framework was then achieved by undertaking semi-structured interviews with older adults during an emergency care episode, and focus groups with staff responsible for the provision of emergency care to older adults across three EDs. 18,19 This study aims to describe the process of generating and prioritising a list of suggested items for PREM-ED 65 by involving multiple stakeholders including patient and public representatives, healthcare professionals and advocates for older adults.

| Item generation
An initial list of candidate items was developed by two researchers (B. G. and J. M. L.) following methodological triangulation of findings from prior studies conducted by the research team. These consisted of a qualitative metasynthesis of 22 studies of patient experience in the ED 17 ; interviews conducted with 24 patients aged over 65 attending the ED 18 ; and interprofessional focus groups with 37 ED staff. 19 Methodological triangulation describes the use of multiple data sources to study a phenomenon, and is useful to confirm findings, enrich data and increase overall validity. 20 Therefore, similar findings that occurred across more than one of the studies were identified as particularly relevant as a focus for future measurement of older adults' experiences of ED care. Item generation focused on these recurrent areas. To enrich understanding, excerpts of relevant findings were highlighted, extracted and grouped together. Each group of excerpts was then summarised by the two researchers and translated into a single suggested item for inclusion in PREM-ED 65.
To ensure the conceptual underpinnings of the study were respected, the research team discussed the meaning of each item and categorised each item according to one of the five analytical themes: communication, emotional, waiting, care needs, physical and environmental needs, or team attitudes and behaviours.
Following identification, the wording of each of the draft items was subjected to a readability assessment, accomplished by calculating a Flesch Reading Ease (FRE) score. The FRE provides a simple formula for assessing semantic difficulty and is commonly used to interpret the readability of health information. 21 The score signifies how easy a statement is to read on a scale of 0 (most difficult [postgraduate reading level]) to 100 (least difficult [9-year-old reading level]). Typically, a score of 70 is assumed to be accessible to the average adult. 22 In practical terms, this represents the reading age of an average 12-year-old. Therefore, candidate items with a score of less than 70 at the initial assessment were modified by simplifying the vocabulary, syllable count and structure of the statement. Readability was considered satisfactory when a postadjustment score of greater than about 70 was attained.

| Item familiarisation and comprehension assessment
For the first workshop exercise, participants were asked to provide a comprehensibility assessment of items. For each item, participants were asked to determine whether the item was (i) 'easy to read' (Yes/ No) and (ii) 'easy to understand' (Yes/No). Participants were invited to suggest new items if any gaps were identified.

| Initial voting
The second workshop exercise was initial prioritisation. During this voting exercise, participants were presented with each item and asked to individually vote on the perceived importance for inclusion in PREM-ED 65. This was accomplished using a nine-point interval scale; priorities 1-3 were labelled 'less important', priorities 4-6 as 'Important, but not critical' and 7-9 were 'Critically Important'.
The median priority and measure of inter-rater agreement (absolute deviation from the median [ADM]) was calculated for each item. 23,25 The mean ADM (MADM) across all items was then calculated, and individual items with an ADM greater than 50% of the mean value were deemed as having insufficient inter-rater agreement. This was used to determine whether the item was eligible for inclusion, exclusion or final adjudication in a second round of voting (Table 1). Data collection and analysis for initial voting was accomplished in real-time by members of the research team (F.B. and B.G.) using a preformulated instrument developed in Microsoft Excel.

| Final adjudication
The third workshop exercise was the final adjudication. This consisted of dichotomous voting for items which did not meet inclusion or exclusion criteria during the first round. During this exercise, participants were presented with the item and requested to vote to either 'include' or 'exclude' the item. To facilitate inclusion of only those items for which there was clear positive consensus, a majority threshold of at least 75% was prospectively agreed to determine the criteria for inclusion. This threshold is comparable with other studies. 26,27

| Participant evaluation
Participants were invited to complete an optional 10-item anonymised paper-based survey at the end of the workshop. This aimed to evaluate overall satisfaction with the NGT process, the ability to meaningfully participate and invite suggestions for future improvements.

| Workshop participants
Twenty-nine participants attended the consensus workshop (

| Item familiarisation and comprehension assessment
To reduce the burden on participants, the 136 items were divided between four groups (34 items/group). Each group was facilitated by either a member of the study team or a volunteer who was a final- year medical student. All facilitators received prior training in the study protocol and NGT method. Group members were encouraged to assess allocated items for comprehension using a 'think aloud' technique, led by a group facilitator. 28 All items were retained and were assessed as being easy to comprehend. Two additional items were added and agreed between participants, both following a large group discussion relating to the perceived importance of recognising disabilities in the ED (Quotations 1 and 2).
My disability did not get in the way of my care.
Staff recognised my hidden disability. As a result, a final list of 138 items was generated.

| Initial voting
The Real-time data analysis of first-round prioritisation data yielded 70 (50.7%) items meeting criteria for automatic inclusion in PREM-ED 65 (priority 7-9 and MADM < 1.04). By way of example, the highest ranking 10 items are presented in Table 3. All remaining items (n = 68, 49.2%) required further voting; this included the four items identified as less important, as inter-rater agreement was insufficient to justify automatic exclusion.

| Item final adjudication
The 68 remaining items were subjected to final adjudication. Of these, 39 (57.3%) items received insufficient favourable votes, resulting in their suggested exclusion from the PREM-ED 65. The lowest ranked 10 items are presented in Table 4. Notably, all four of the items originally prioritised as 'less important' were excluded during this round (average proportion of 'favourable' votes for these items, 32.4%).

| Final prioritised list of candidate items for inclusion in PREM-ED 65
An additional 29 items were prioritised for inclusion because of final adjudication. Hence, a total of 99 out of 138 items remained eligible for inclusion in the instrument, representing 71.7% of the original items.
The finalised full prioritised list of included and excluded items are presented in Electronic Supporting Information Material S2.

| DISCUSSION
This paper describes the process of generating and prioritising a list of candidate items for the PREM-ED 65. There is currently no accepted gold standard for generating or prioritising items for inclusion in either PROMs or PREMs, despite this being an essential T A B L E 3 Top 10 ranking items included via initial prioritisation (presented in rank order based on median priority and then inter-rater agreement (MADM).

Item Median priority MADM
Staff who were learning were always supervised. 9 0.11 The pain relief medicine worked well. included the presence of disability as a discrete geriatric condition when evaluating characteristics of older adults attending an ED in Denmark. In this study, the presence of one or more geriatric conditions was associated with poorer health outcomes following ED attendance. Furthermore, improving transitions from ED care to community settings may prevent functional decline and increased disability that occurs in older adults following ED attendance. 41,42 Our experience is that conducting NGT amongst a population of older adults is an achievable and rewarding means to effectively prioritise items for inclusion within a PREM. Using this approach it was possible to assess and prioritise all items within a single day. To this end, NGT may be more efficient than other consensus-building methods, most notably the Delphi method, where ongoing participant engagement is required during multiple asynchronous rounds of voting, often spanning months in duration. This requires high levels of participant engagement throughout the process, to avoid attrition. 43 Furthermore, NGT may yield the highest levels of accomplishment and satisfaction compared to either the Delphi method or unstructured groups. 44 This is reflected in the high satisfaction reported amongst participants in this study, as reported through postevent feedback.
For the NGT, the first round prioritisation revealed that most candidate items were deemed of 'critical' importance. Therefore, the method was effective in identifying very high-priority items for inclusion in the instrument-that is, those assigned 7-9 out of 9 and meeting the predetermined criteria for inter-rater agreement. The  importance. The latter is reflected in recent literature, highlighting that prolonged ED length-of-stay is independently associated with the development of hospital-acquired pressure sores. In the current international context, where ED crowding and prolonged length-ofstay is the norm, adequate tissue viability assessment and pressure sore prevention during the ED stay is essential. 45 Additionally, the importance of many of the other themes are prominently recognised in the literature. For example, stakeholders within this study were almost unanimous in emphasising the importance of clinical supervision for trainees in ensuring an optimal experience. Indeed, supervision of trainees in the ED has been recognised as essential to both ensuring patient safety, and facilitating clinicians' professional development. 46 In relation to pain management, older people may be more susceptible to receiving inadequate pain relief in the ED, compared to younger patients. 47 Although the first round of voting was very effective in highlighting items for inclusion, it was not possible to exclude any item using this initial round, and it was, therefore, necessary to Furthermore, shortened questionnaires have been shown to effectively measure experiences of care. 51 The NGT has provided an initial means of reducing items for PREM-ED 65.
To validate the psychometric properties of PREM-ED 65, a quantitative study will be conducted with a population of ED patients. This study will aim to confirm how each item performs in a real-world setting by assessing participant engagement, floor/ ceiling effects and differential validity of the items. Any items with low engagement or problematic validity will be removed to reduce the length of the questionnaire. The remaining items will undergo exploratory factor analysis to confirm structural validity. Additionally, the study will assess the internal consistency of measurement scales and test-retest reliability. The goal is to make PREM-ED 65 suitable for assessing the experiences of a wide range of older adults in the ED.