Impact of using data from electronic protocols in nursing performance management: A qualitative interview study

Abstract Aim To explore the impact of using electronic data in performance management to improve nursing compliance with a protocol. Background Electronic data are increasingly used to monitor protocol compliance but little is known about the impact on nurses’ practice in hospital wards. Method Seventeen acute hospital nursing staff participated in semi‐structured interviews about compliance with an early warning score (EWS) protocol delivered by a bedside electronic handheld device. Results Before electronic EWS data was used to monitor compliance, staff combined protocol‐led actions with clinical judgement. However, some observations were missed to reduce noise and disruption at night. After compliance monitoring was introduced, observations were sometimes covertly omitted using a loophole. Interviewees described a loss of autonomy but acknowledged the EWS system sometimes flagged unexpected patient deterioration. Conclusions Introducing automated electronic systems to support nursing tasks can decrease nursing burden but remove the ability to record legitimate reasons for missing observations. This can result in covert resistance that could reduce patient safety. Implications for nursing management Providing the ability to log legitimate reasons for missing observations would allow nurses to balance professional judgement with the use of electronic data in performance management of protocol compliance.

electronic devices the resulting data create new ways of measuring nursing omissions. For instance, the National Health Service (NHS), the national, free-at-the-point-of-entry healthcare system, provides access to advice from healthcare professionals via its 111 telephone call service. Time-stamped electronic data collected during these calls are used to compare nurse response times and numbers of enquiries resolved within the time required by their protocol (Pope et al., 2017;Prichard, Turnbull, Halford, & Pope, 2014;Ruston, 2006).
Previous research has shown that nurses can resist electronically based protocol implementation in overt and covert ways (Pope et al., 2017;Prichard et al., 2014;Ruston, 2006;Timmons, 2003), thereby undermining the aims of the protocol (Timmermans & Berg, 2003).
The research to date has focused on NHS 111 (Pope et al., 2017;Prichard et al., 2014;Ruston, 2006) electronic patient records (e.g. Timmons, 2003), or interactive whiteboards (Allen, 2014), which are not time critical. It is therefore important to understand the impact of using electronic data in time-dependent protocols for performance management in a ward with multiple task demands.
In this research, we explore the use of early warning score (EWS) protocols embedded within bedside electronic handheld devices.
Taking and interpreting vital signs are a fundamental aspect of nursing care (Kitson, Conroy, Wengstrom, Profetto-McGrath, & Robertson-Malt, 2010). Early warning scores were introduced as a standardized way to measure patient deterioration (Morgan, Williams, & Wright, 1997), weighting physiological readings based on degree of deviation from agreed normal ranges to create a score capturing overall severity  (Smith, Recio-Saucedo, & Griffiths, 2017). There has been a high uptake of EWS protocols with 99% (530/538) of UK hospitals using them to monitor deteriorating patients, with 97.9% using linked escalation actions (National Confidential Enquiry into Patient Outcome and Death [NCEPOD], 2015).
One reason the use of electronic EWS protocols has increased is because of Francis report recommendations that automated vital signs taking could improve patient safety (Francis, 2013(Francis, , p. 1599 and research highlighting inaccuracies in paper-based EWS scores (Niegsch, Fabritius, & Anhøj, 2013;Odell, 2015). The introduction of electronic EWS systems has been associated with decreased mortality in hospitals (Schmidt et al., 2015) and significant improvement in clinical responses (Credland, Dyson, & Johnson, 2018;Jones et al., 2011). Yet non-compliance persists at night and for patients with the highest EWS values Hands et al., 2013;Jones et al., 2011). While some non-compliance is associated with low staffing levels, most is not . This paper explores how electronic data were used to performance manage ward-level nursing compliance with a EWS protocol. A 'technologies of practice' perspective is employed, which emphasizes the importance of understanding the impact of technology on a social setting and vice versa (Timmermans & Berg, 2003).
Aim: To explore the impact of using electronic data in performance management to improve nursing compliance with a care protocol.

| Background
The data in this study derives from a wider study exploring why diurnal variation in vital signs observations persisted in the study hospital following the introduction of a EWS protocol embedded within bedside handheld devices. The study hospital uses commercially available, small handheld electronic devices in all adult inpatient wards except maternity and high care areas. The embedded software requires nurse input of a full vital signs set required by the EWS algorithm. Once a full set is entered, a timestamp is assigned to the data, which is saved to the hospital's central database. The EWS value is automatically calculated and any required actions are displayed on the device screen.

| ME THODS
The project received University ethical approval on 30/03/2015 with governance approval gained from the hospital's Research and Development Office on 15/06/2015. This was a qualitative interpretative study using semi-structured interviews. A qualitative approach was chosen as it is the method of choice for exploring implementation of technical solutions in health care (Berg, 1997). Members of staff in a general hospital in the South of England were recruited through a survey exploring night-time compliance with the protocol. Seventy survey respondents indicated their interest in participating; forty-eight were eligible and provided accurate contact details. For inclusion, staff were required to have worked night shifts immediately prior to the survey launch. Attempts were made to build a "deviant case sample" (Patton, 2015), recruiting wards with the highest and lowest nighttime protocol compliance. However, this had to be expanded to all eligible interested staff to ensure recruitment was high enough to reach data saturation. Seventeen staff members participated in interviews in June 2016. This sample resembled a maximum variation sample in terms of ward compliance level, years of experience and speciality (see Table 1). Ward compliance was measured by using administrative data from the hospital to stratify wards into quartiles reflecting percentage of scheduled vital sign observations taken on time at night according to the EWS protocol (see Table 1). Wards represented had six to 65 beds (including trolleys and chairs). The hospital runs at high occupancy (90%-95%). The average Registered Nurse Care Hours Per Patient Day varied between 3.8 in surgical wards to 7.4 in the Medical Assessment Unit (MAU), and average Health Care Assistant hours per patient day from 2.8 in medical wards to 3.9 in older people's wards.
Interviews, lasting between 19 and 61 min, were conducted face-to-face in a private room or over the telephone. The interview topic guide covered: • patient characteristics, care needs and ward specialty, Qualitative analysis was carried out using the constant comparative method described by Lofland, Snow, Anderson, and Lofland (2006). This is informed by grounded theory but permits the use of relevant top-down codes. NVivo software was used. The lead author and fourth author each coded three transcripts, compared them, and reached a consensus on transcript coding and initial code framework. The remaining transcripts were divided between lead author and fourth author. Codes and coding were discussed through NVivo memos and regular meetings until consensus was reached.

| FINDING S
The first section describes nursing practices in relation to the protocol before performance targets were implemented. These are added to the process shown in Figure 1, with additional processes added in new circles ( Figure 2). The final section explores how these processes changed when ward performance targets were implemented, with further circles added to Figure 3.

| The role of an electronic EWS protocol before ward performance management
Interviewees described competing demands, which could interfere with their ability to take scheduled observations when the clock icon indicated they were due. This included clashes with other scheduled tasks such as completing the hospital-wide bed occupancy database with patient information.
It has to be done hourly. This created an observation schedule perceived by many interviewees as inappropriately frequent. Interviewees described sometimes partially completing a vital signs set for these patient groups that would be recorded as a late or missed observation on the centralized EWS database and local devices (see Figure 2). They also described omitting vital sign measurements entirely for such patients (particularly at night), negotiating this at ward level and recording it in patient notes. Prior to the introduction of ward performance measures, reminders on the device to take a full set of vital sign observations were treated as advisory, used alongside the nurse's clinical judgement, ward management considerations and competing priorities. One interviewee said she had never woken a patient at night, another that a nurse-in-charge (on a different ward) had asked her not to take observations at night. However, some interviewees described benefits of having external reminders via the device: time does fly when you're working in such a busy environment -and it does bring your attention to, 'There's a red clock there; when was the last time this was done?' (RN8, medium compliance ward) Similarly, some nurses used the EWS to explore the reasons why a patient was unwell, using clinical judgement to decide on the next step, rather than relying on the EWS to make that decision.

| Impact of ward performance measures using compliance data
Ten of the 17 interviewees described benefits associated with the introduction of ward performance targets. These included reducing the chance of accidentally overlooking patient monitoring, reducing the time required to delegate observations, redistributing observations throughout the shift, encouraging reconsideration of patient condition and increased patient contact time. Significantly, although interviewees described the intervals required between observations as too short, they also acknowledged instances where unexpected deterioration had been detected more quickly. you do sometimes catch -when you think a patient's quite stable, and you actually do find that overnight they become unstable, that it is picked up.
(RN11, medium compliance ward) Interviewees reported pressure from ward managers to maintain and improve ward performance, meaning some interviewees would carry out observations even when this conflicted with their clinical judgement and was unpopular with patients.
Due to the inability to report reasons for omissions, interviewees described feeling penalized when prioritizing care of rapidly deteriorating patients.
it's a little bit demoralising […] because you suddenly feel, "Goodness, we're going to have not a good report […]" but actually you've been tied up in […] preserving life (RN4, Low Compliance ward) The inability to report reasons for exceptions meant some staff used a loophole to avoid having an omission recorded.
I think some people do still find it frustrating that there isn't a way to turn it off when they know their patient's stable, but the only way you can do it is by [selecting a specific setting] which isn't ideal, but it stops the red clock and that's the way some people get around it (RN11, Medium Compliance ward) This highlighted the existence of invisible non-compliance -a disparity between the reality of nursing practice and the recorded electronic data. Figure 3 summarizes these findings to show the impact of ward performance targets on staff protocol compliance. New actions are included in additional circles. The reasons why staff members judge that observations should be missed remain, but staff response differs due to increased pressure to demonstrate compliance with the protocol. The lack of a way to feedback reasons for non-compliance led to nursing staff taking observations at odds with their clinical judgement or the use of a loophole to avoid penalization when care was omitted. This has the potential to erode a sense of professional autonomy or introduce new, unmonitored risks to patient safety.

| D ISCUSS I ON
This study highlighted some important issues relating to the wider use of electronic data in performance management of nursing compliance with protocols. Before this initiative began staff were able to negotiate exceptions with colleagues. However, once compliance targets were implemented this created covert behaviours that were invisible to the protocol database or forced protocol-compliant care to be taken regardless of clinical (or ward management) judgement.
This threatened nurses' sense of autonomy and their perceived ability to provide personalized care to some patients. This reflects prior research that highlights the need to use clinical judgment to assess deterioration alongside EWS protocols and a continued discussion as to whether the development and use of such judgment is undermined by the use of EWS (Downey, Tahir, Randell, Brown, & Jayne, 2017).
The key limitation of this paper is its reliance on the accounts of nursing staff about their experience of using a protocol embedded within bedside handhelds. However, as can be seen from the quotations, nurses disclosed specific instances of ignoring the protocol. A greater understanding of the reasons for protocol non-compliance could be gained through taking ethnographic observations of how nurses interact with the devices at the bedside.
This study allowed us to explore the role of professional judgement in the context of resistance to electronic protocols. As Allen has argued, protocols and supporting technologies often fail to account for the "invisible" work of nurses, which can involve emergent organizational care, focused around patients' care trajectories.
These can be complex and require protocol adaptation (Allen, 2014(Allen, , 2019. Some of these concerns were supported to varying extents by the evidence base. One issue -also found in an evaluation of an electronic EWS system -was the lack of flexibility or feedback opportunities (Lang, Pinchin, Brown, & Sharples, 2016). In particular, the frequency of observations required for patients with chronic conditions (particularly COPD), where higher EWS scores reflected a chronic vital signs derangement, a difficulty highlighted by Wood, Chaboyer, and Carr (2019) in a recent review. Indeed, some research has used different thresholds to differentiate patients with respiratory conditions (CREW: Lobo, Lynch, & Casserly, 2015), while others suggest developing different algorithms for groups with distinct physiological profiles (Downey et al., 2017). However, an increasing body of research has found EWS (particularly NEWS) may be suitable for diverse patient groups, including those with COPD (Hodgson, Congleton, Venn, Forni, & Roderick, 2018;Hydes et al., 2018;Kovacs et al., 2016;Pimentel et al., 2018;Redfern et al., 2018).
Interviewees also disputed the necessity of the frequency of observations overall and whether there was always a need to take a whole set of observations, which reflects gaps in the evidence base (Smith et al., 2017). Nevertheless, evidence identifying an elevated risk of death within 24 hr in patients with high EWS scores is strong (Prytherch, Smith, Schmidt, & Featherstone, 2010). Nurses' treatment of the protocol as advisory reflects other research where protocols were treated as prescriptive guides, to be particularized for specific patients (Rycroft-Malone, Fontenla, Seers, & Bick, 2009) or "circumvented, tinkered with and interpreted" (Berg, 1997).
However, as data from an emerging literature have demonstrated, nursing staff do not only use clinical judgement but also ward management decisions at night (Hope et al., 2018. Ward performance management using electronic data to assess compliance aims to deal with these kinds of nursing omissions. However, these findings demonstrate the use of loopholes to avoid detection and penalization, echoing the "passive resistance" used against the mandatory implementation of electronic patient records (Timmons, 2003), or the overt and covert resistance to algorithm-adherence performance targets for 111 nurse advisors (Pope et al., 2017;Prichard et al., 2014;Ruston, 2006). In addition, these findings suggest that allowed margins of "non-compliance" can drive systematic under-monitoring of some groups under the hospital's radar. As described here, tightening compliance expectations risks reducing the visibility of missed care and creating new, undetected risks to patient safety.
Broad lessons can be learned from this study to increase patient safety and staff morale where electronic data are used in performance management of nursing compliance with protocols.
Firstly, providing alternatives to replace the previously hidden mechanism of negotiating exceptions with colleagues, an example of an "invisible" example of nursing practice that may not be accounted for when new technology is introduced (Allen, 2014). This could be either a formal modification of the electronic protocol parameters, such as providing latitude around EWS baseline interval frequency (Jones et al., 2011). Such approaches reduce the need to use covert methods and generate evidence to fine-tune protocols. Secondly, regularly monitoring data to explore whether some groups are more likely to receive less protocol-based care.
Finally, there should be a way to positively identify cohorts of patients where the protocol may be less appropriate. For these groups, valid alternative protocols with proportionate responses should be developed.

| CON CLUS ION
Standardization has the ability to reduce human error, and the use of digital data can uncover missed or suboptimal care. However, as this study has shown, it is important to approach the imposition of digital surveillance targets critically, and to understand reasons for passive resistance. This can help to avoid the use of covert strategies, which can undermine the purpose of using digital data in performance management by creating false readings.

| INFORMATIVE
Electronic data are increasingly used to monitor nursing quality but little is known about the impact on nurses' practice in busy hospital wards. This paper presents a case study of the use of information management and technology in nursing performance management: the use of electronic data to measure ward compliance with an early warning score protocol delivered by a handheld device. Interviews with 17 members of nursing staff found that introducing automated electronic systems to support nursing tasks decreases nursing burden but removes invisible mechanisms for negotiation that provide a balance between nursing judgement and standardized protocols.
This can result in covert resistance that decreases patient safety.

ACK N OWLED G EM ENTS
We thank the other members of the Night Surveillance Study