Developing the Resident Measure of Safety in Care Homes (RMOS): A Delphi and Think Aloud Study

Abstract Objective This study aimed to develop a measure of contributory factors to safety incidents in care homes to be completed by residents and/or their unpaid carers. Introduction Care home residents are particularly vulnerable to patient safety incidents, due to higher likelihood of frailty, multimorbidity and cognitive decline. However, despite residents and their carers wanting to be involved in safety initiatives, there are few mechanisms for them to contribute and make meaningful safety improvements to practice. Methods We developed 73 evidence‐based items from synthesis and existing measures, which we presented to a panel of stakeholders (residents/carers, health/social care professionals and researchers). We used two online rounds of Delphi to generate consensus (80%) on items important to include in the Resident Measure of Safety in Care Homes (RMOS); a consensus meeting was later held. The draft RMOS developed through the Delphi was presented to participants during ‘Think Aloud' interviews using cognitive testing techniques. Results The 29‐item RMOS was developed. Forty‐three participants completed Delphi round 1, and 27 participants completed round 2, 11 participants attended the consensus meeting and 12 ‘Think Aloud’ interviews were conducted. Of the 73 original items, 42 items that did not meet consensus in Delphi round 1 were presented in round 2. After the consensus meeting, it was agreed that 35 items would comprise the RMOS questionnaire and were presented in the ‘Think Aloud’ interviews. Participants suggested numerous changes to items mostly to improve comprehension and ability to answer. Conclusion We have a developed an evidence‐based RMOS, with good face validity, to assess contributory factors to safety in care homes from a resident/carer perspective. Future work will involve psychometrically testing the items in a pilot and developing a complementary simplified, dementia‐friendly version to promote inclusivity. Patient or Public Involvement Four patient and public contributors worked with researchers to develop the online questionnaires. Patients (residents) and carers participated on the consensus panel. One member of the research team is an expert by lived experience and was involved in design and analysis decisions. The item list and instructions for the questionnaires were reviewed for face validity, understanding and acceptability by a patient and public involvement group and modified.


| INTRODUCTION
Ensuring patient safety in care homes is a longstanding challenge in the United Kingdom, which has been further evidenced to the public during the SARS-CoV-2 pandemic. 1 This is complicated by the fact that older residents in care homes often live with long-term conditions and functional decline, take multiple medications and experience recurrent hospitalizations. 2 Care home residents are particularly vulnerable to experiencing patient safety incidents, which can lead to severe but avoidable harms including hospital readmissions, infections, medication harm and mortality. 3 They are twice more likely to experience emergency admissions than the rest of the population over the age of 75 years, and 40% experience adverse events within 45 days after their discharge from hospitals back to care homes. 4 A recent publication from WHO suggests that improving patient safety should begin with engaging patients. 5 In primary and secondary care settings, evidence suggests that patients are able to identify the factors that contribute to patient safety incidents. These contributory factors are often defined as proximal and latent causes of error. 6 Examples include ineffective communication between professionals, poor quality of treatment or inappropriate physical environments. 6 Capturing patient feedback about contributory factors to patient safety incidents and using this feedback for making safety improvements is a novel and promising field of research. It is known that patients, carers and family members are able to provide a different and valuable perspective on patient safety in hospitals and primary care; however, less is known about the care home setting. 7-10 A measure of patient feedback on safety is therefore needed in this setting. This would ensure that resident and carer preferences are accounted for, and provides another measure for safety outside routinely collected data and case record reviews. 11 Patient safety improvement projects in the UK care homes sector could considerably benefit from engaging residents and their relatives. 12 Enabling residents to provide feedback on safety has the potential to improve person-centred care, which is now em- implementation of real-time service improvements, embracing the view that healthcare staff are more likely to engage with interventions that they have been involved in developing. 14 At present, no measures utilize residents' perspectives of contributory factors when making safety improvements for older people/their carers living in residential care homes. 15 Although a proportion of older residents in care homes might not be able to engage with a self-reported patient feedback measure due to advanced cognitive decline, most residents and/or their carers will be able to provide some level of feedback on patient safety and can play a vital role in the success and longevity of patient safety improvement interventions in care homes. A theory-driven, evidence-based measure, derived from James Reason's Swiss Cheese model of accident causation theory, which identifies the latent conditions (contributory factors) responsible for patient safety incidents, 16 and similar to the PC PMOS would be particularly helpful to capture information about the contributory factors to patient safety in care homes from the residents'/carers' perspective.
The aim of this study was to develop a resident/carer measure of safety that can be used as a basis for proactively preventing and improving safety in care homes.

| Study overview
The Resident Measure of Safety in Care Homes (RMOS) was developed in four stages, and involved residents, carers and healthcare professionals at each stage: (1) a long list of potential items was generated from conducting a systematic literature review 17 and through exploration of existing measures; (2) the list was presented to stakeholders in an online Delphi process (2 rounds); (3) the results of the Delphi survey were appraised at a consensus meeting; and (4) the resulting measure was presented to a small group of participants in a 'Think Aloud Study' for face validity testing; see Figure 1.

| Participants
Participants (from the United Kingdom) were recruited in several different ways. Academic researchers were recruited if their research had been included in our systematic review; 17 in addition, known researchers in the field identified by the team were also approached.
They were approached by email. Care Home residents were recruited through a 'gatekeeper' in three care homes. 'Gatekeepers'' were care home managers who identified and approached residents. Information leaflets were shared and residents self-identified. We asked participants to read the information sheet and to summarize it; this was used to assess capacity, and those who could not summarize it were excluded. We recruited a number of care home professionals through these sites. Sites were selected to represent different demographic areas; however, most care homes did not have the capacity to take on any research during covid, so this limited our choice. The remaining healthcare professionals, care home professionals and carers were recruited through our teams' existing professional networks and social media (Twitter and Facebook). One author (B. S.) also emailed multiple UK care homes registered on www.carehome.co.uk. Participants were eligible to participate if they had personal or professional experience of transfers between care homes and hospitals.
The same participants were involved throughout the Delphi research process. Consensus meeting participants who had completed both Delphi rounds were asked to indicate whether they would be interested in attending a face-to-face meeting. We invited a random sample of interested participants to attend, while ensuring representation of all key stakeholder groups. If a participant declined the invitation to the consensus meeting, a similarly matched participant was invited from the Delphi panel. They were selected based on their stakeholder view, i.e., care home staff, family member or patient safety expert.

| Stage 1: Gathering information
To develop our initial pool of items, we synthesized items derived from existing evidence in this field and items from exiting measures in other settings. Our systematic review of contributory factors to safety in care homes, 17 which informed this work (submitted), identified 18 contributory factors with two additional categories that were added to the existing contributory factors to the safety in hospitals framework 9 (family-related factors and organization and care planning). Items from two existing patient-reported outcomes measures (developed for hospitals and primary care) 8,18 were combined to generate a long list of potential items. This list was discussed by three authors N. T., S. G., B. S.) in a structured meeting.
Each potential item was considered in turn and each member had the opportunity to present arguments for or against inclusion. For each item, it was decided whether the item should be a stand-alone item, edited for the care home setting (e.g., we adapted from the PMOS, 'the hospital departments were very clean' to 'the care home was very clean'), combined with another item of a similar thematic nature or removed if not relevant to a care home setting (e.g., 'I was able to make an appointment at a time that suited me'). If all three authors agreed to remove or merge an item, it was amended; otherwise, the item remained as a stand-alone item.

| Stage 2: Delphi survey
The Delphi technique is a research method aimed at generating consensus. 19,20 It solicits opinions from stakeholder groups in an iterative process of answering questions about which items should be included and why. After each round, the responses are summarized and redistributed for discussion in the next round. We chose a priori to hold two Delphi rounds in this study. The item list that was finalized after group discussion in stage 1 was used to populate the first Delphi questionnaire. Any items without consensus after the first round were re-presented in round 2.
Surveys were distributed to participants who consented to take part. For contacts in the researchers' networks and those recruited using social media, this was via an online link. For residents, these were in paper format due to lack of technology in care homes.
Managers also distributed electronic copies via email to staff. Round (see Section 4.1). We conducted the Delphi survey using Qualtrics, a secure online hosting platform. 21 After reading a participant information sheet and giving informed consent (by ticking a box), participants selected their stakeholder group. Participants answered a number of demographic questions, gender, ethnicity, etc, and selfgenerated an identification code for follow-up.
In each round, participants were asked whether the items should become part of the RMOS measure. A 9-point Likert scale was used: ranging from (9) Highly relevant to (1) Highly irrelevant; items 8-2 were presented as numerical values with no text labels. There was a free-text comments box and participants were encouraged to provide comments that would be fed back anonymously to the whole group.
Participants could suggest additional items/contributory factors to safety at the end of round 1, which were reviewed by the core research team. Any contributory factor not already represented was added to round 2.
In round 2, median group scores for each item and anonymous comments for and against from the previous round were presented, and participants were asked to reflect on the information and score each item again. The percentage of participant agreement with each item on a scale of 1-9 was calculated from the scores obtained during round 1 and again in round 2.
The literature suggests that consensus levels should be set a priori at a minimum of 70%. 22,23 We unanimously chose an 80% consensus level to increase sensitivity. Consensus criteria were defined a priori: items scored as relevant or highly relevant (7-9) by 80% or more of the group reached consensus for inclusion and were included in the first RMOS draft. Items scored as irrelevant or highly irrelevant (1-3) by 80% or more were defined as having reached consensus for exclusion and were excluded. Items with >80% agreement but many comments were sent back to the panel in round 2 to improve readability. Items not fulfilling criteria for consensus inclusion or exclusion were defined as not having reached consensus and were therefore re-presented in round 2 for reconsideration. Decisions for inclusions and exclusion criteria were made a priori to reduce burden between rounds by removing items that fulfil criteria. We decided a priori to limit the process to two Delphi rounds due to the complexities of conducting research during the Covid-19 pandemic; stopping a Delphi after a predetermined number of stages is common in the existing literature.

| Stage 3: Consensus meeting
The results of the Delphi survey were presented at a consensus meeting. The main goal of the consensus meeting was to decide which items will be included in the first draft RMOS. This was chaired by four members of the team (N. T., S. G., C. P., M. P.) with expertise in consensus methodology and patient involvement and engagement.
Participants were sampled to achieve a balanced representation of carers, healthcare professionals, care home professionals and researchers. Unfortunately, due primarily to Sars-CoV-19 restrictions on care home access, we could not recruit residents to this component. We aimed to have a small representative group of between 9 and 12 participants to enable meaningful small group discussions, similar to previous consensus meetings. 24,25 The format of the consensus meeting comprised of (a) a short overview of the study and (b) a summary of the Delphi results sorted by stakeholder group, beginning with the items that were agreed to be included after round 2 of the Delphi, any items that might be contentious and finally all of the remaining items presented in round 1. 26 Participants were asked if there were any fundamental reasons why these items should not be included in the RMOS measure.
Divergent views were actively sought and the chair ensured that everyone had the opportunity to participate in discussions. Data were collected to ascertain which items should proceed to the think aloud stage and reasons for exclusion or reintroduction.

| Stage 4: Think Aloud
A selected sample of participants was then asked to complete 'Think Aloud' interviews. This involved completing the RMOS and describing their thought process when arriving at their answer. Three researchers conducted the 'Think Aloud' interviews, using a standardized prompt sheet that provides example 'Think Aloud' questions to familiarize participants with the process and then questions about the items covering 1. Comprehension, 2. Retrieval, 3.
Judgement and estimation, 4. Documenting a response and standardized associated prompts (i.e., can the participant make a judgement based on the information that they are able to retrieve).
Think aloud methodology was based on work by Tourangeau. 27 Interviews were audio-recorded.
From an analysis perspective, each interviewer coded each item (for each participant) for 1. Comprehension and 2. Ability to answer (based on retrieval/judgement/estimation/documenting a response) into one of three categories (Yes, no, partially). An overall percentage score based on comprehension and ability to answer the question was calculated for each item and then each item was discussed in person at a team meeting (S. G., C. P., N. T.) to generate the final version of the RMOS presented in this paper.

| Patient and public involvement (PPI)
Four patient and public contributors worked with researchers to develop the online questionnaires. Patients (residents) and carers were also represented alongside professionals and researchers in the consensus panel. One member of the research team is an expert by lived experience and was involved in design and analysis decisions.
The item list for round one and instructions for the questionnaires were reviewed for face validity, understanding and acceptability by a PPI group and modified according to feedback. All public contributors were remunerated for their time.

| Stage 1: Item generation
Our systematic review has been described in detail elsewhere. 17

| Stage 3: Consensus meeting
Twelve participants attended the consensus meeting, with equal representation from each of the three categories: four participants were researchers, four identified as carers/relatives and four identified as healthcare/care home professionals.
Each of the items was considered individually and discussions indicated that many could be made clearer. Subsequently, twentyone items remained the same, 2 items were combined, 12 items were adapted and 3 items were excluded. Four items that were excluded in round 1 and 2 items that were excluded in round 2 were discussed in the consensus meeting and reintroduced either through combination with an existing item or by adapting the wording. Thirty-five items were presented in the 'Think Aloud' study.

| Stage 4: Think Aloud
Twelve participants completed the 'Think Aloud' interview. Nineteen items were adapted as a result, which typically involved changing one or two words, for example, 'my friend/relative is aware of how to report a concern' was changed to 'we are aware of how to report a concern'. Nine items remained the same and six items were removed.
Most items were removed because participants felt that they were not able to make an adequate judgement to sufficiently answer the The final PMOS measure developed after the Think Aloud interviews is presented in Figure 2, Table 1.   There are enough staff to look after the needs of residents 5

| DISCUSSION
Things get done in a timely manner (e.g., help with going to the toilet) 6 There is always a member of staff available with the knowledge/skills to perform specific tasks 7 Residents are always given the right food for them 8 The care home is very clean 9 The physical environment of the care home feels safe (e.g., good lighting levels, enough space, no clutter) 10 All equipment needed for residents' care always work properly These statements are about the staff that work in the care home (e.g., carers, nurses, nursing assistants, managers) rather than staff that visit from outside agencies These statements are about sharing information 20 We are informed of mistakes made 21 We have been informed of any infection outbreaks and the action to control further spread 22 We are always offered enough information about their care 23 We always receives answers to all the questions we have regarding care 24 We are encouraged and supported to be involved in decisions about care 25 We are aware of how to raise a concern about a member of staff The key concern was regarding the ability of carers to make judgements about some contributory factors to safety that they have not witnessed or experienced. Examples of these domains/items include how staff communicate/work with one another or things that happen infrequently, such as abusive language. Many carers agreed that the SARS-CoV-2 pandemic social-contact limitation policy reduced their ability to make judgements about contributory factors to safety. It was also agreed that these items would be difficult for many carers to answer when social-contact limitations are lifted, based on visits to a care home.
Therefore, items that would be difficult for most carers to answer for this reason were removed following the 'Think Aloud' process.

| Limitations
The key limitation of this study was the attrition rate: 39% after round 1. Although attrition is very common in Delphi studies, 34 we found the attrition rate for this study to be more pronounced than that in other Delphi studies. 35,36 We attribute the attrition rate to a number of factors, the first being the SARS-CoV-2 pandemic and associated governmental restrictions, which meant that access to care homes during the study period was not permitted; even the carers who contributed to the study reported only seeing their relatives for 30 min a week. Although several gatekeepers (care home volunteer/managers) circulated the paper versions of round 1 to care home residents, they were unable to collect/oversee the round 2 responses due to employment changes, policy changes and resource constraints. Also, recruiting residents who were able to participate proved difficult as many had dementia, sight or movement disorders.
Therefore, we lost contact with the majority of the residents who In comparison with measures developed for other settings (primary and secondary care), during our stakeholder engagement and consensus process, concerns were raised about the appropriateness of items that measure safety in care homes in the measure's current format. The stakeholders key concern was the ability of residents without capacity to use the measure (due to later stage dementia or organic mental health problems) to use the measure was considered problematic. Our 'Think Aloud' process and stakeholder consultation work indicated that the measure would be most useful for carers and residents with capacity and using cognitive testing in the 'Think Aloud' was a strength of this study. Also, stakeholders involved in the study found it difficult to make judgements on some key elements of safety, such as medication management, and as a result, this measure may not cover all elements of safety.

| Future work
This study established consensus about the items that should be included in the RMOS measure. Face validity of the measure was also determined. Future research is needed to establish the psychometric properties of the RMOS measure and make targeted adaptations.
This will result in an RMOS that can be used for residents with capacity and their informal carers to improve safety in care homes. In parallel, we recommend that the RMOS be refined into a complementary dementia-friendly simplified version that can be used by people who might lack capacity, due to dementia or other cognitive decline. A dementia-friendly version should be developed in combination with appropriate methods for data collection such as displaying the items as visual graphics, in line with recent advances in inclusive dementia research. 33 The measure also has potential to be used by agencies that assess the quality and safety of care homes such the Care Quality Commission.

| Conclusion
To the authors' knowledge, this study has developed the first measure to enable residents and carers to give feedback on contributory factors to safety incidents within care homes using established measure development science including Delphi, cognitive testing and consensus methods. The RMOS is therefore a theory-and evidence-based measure, which utilized items from existing validated measures, 8,37 and from a recently completed systematic review. 17 The RMOS enables residents (and/or their carers) to identify factors that could contribute to safety incidents, to enable care home settings to take meaningful action to prevent harm.