Use and validation of a survey tool to measure the perceived effectiveness of insulin prescribing safety interventions in UK hospitals

To describe the use and validation of a survey tool to elicit the opinion of hospital pharmacists and medicines safety officers in the UK regarding the perceived effectiveness of strategies to improve insulin prescribing safety in hospitals.


Introduction
Insulin is one of the top medicines associated with prescribing errors and patient harm in the inpatient setting [1]. Junior (trainee) doctors in the UK commonly prescribe insulin, but often feel unprepared and daunted by this task because of lack of confidence, knowledge and experience [2][3][4]. As a result, insulin may be prescribed erroneously or incompletely. This may involve prescribing the wrong product, frequency, device, concentration or number of units, prescribing insulin for the wrong time of day, in an unclear or incomplete manner, or omitting it from the prescription altogether [5]. These errors remain commonplace in hospital, despite efforts to improve insulin safety [6].
National recommendations to help improve insulin prescribing safety include the use of interventions such as electronic prescribing, mandatory education of healthcare professionals, input of specialist diabetes nurses and pharmacists, use of the 'insulin passport', not abbreviating 'units' to 'u' or 'iu' on prescriptions, promoting patient self-administration and self-management with insulin, and fostering a culture of error reporting and quality improvement [5,7].
There is currently significant inter-hospital variation in the use of interventions to improve insulin prescribing safety across the UK [8]. The effectiveness of insulin prescribing safety interventions may be measured by a reduction in insulin errors or increased prescription accuracy/completeness. These outcomes have been reported with the introduction of dedicated insulin prescribing charts [9,10], specific requirements for insulin medicines reconciliation on admission [11], insulin discharge checklists [12] and structured education [13,14]. Comparing intervention effectiveness is, however, restricted by differences in implementation and use in different contexts, organizational factors (such as available staffing and resources), and data collection methods [5].
Scientific methods seeking to establish intervention effectiveness with any certainty may also be unsuitable for evaluating complex interventions in the hospital setting. Insulin self-administration policies, for example, rely on various social, behavioural and organizational mechanisms to 'work', and may produce multiple outcomes for both patients and healthcare providers. Evaluating their effectiveness, therefore, requires context-specific and complexity-sensitive methods that look beyond establishing linear causality [15].
Measuring perceived effectiveness of interventions is a way of circumventing the above limitations for the purposes of identifying salient strategies nationally [16]. Perceived effectiveness, 'the extent to which the intervention is perceived as likely to achieve its purpose' [17], can indicate levels of support, acceptability and scepticism towards adopting recommended interventions [18]. Using this notion allows responses from hospitals irrespective of current intervention use. Insights of perceived effectiveness from healthcare professionals can be used to garner support for the wider implementation of interventions and focus national insulin improvement efforts.
There is currently no tool to assess the perceived effectiveness of inpatient insulin prescribing safety interventions. The aim of the present study was to develop, use and validate a tool to understand healthcare professionals' perceptions of effectiveness regarding a range of insulin prescribing safety strategies. We aimed our survey at hospital pharmacists and medicines safety officers because of their broad and in-depth knowledge of prescribing systems, interventions, and insulinrelated governance issues within their organizations.

Tool development
The Perception of Effectiveness for Prescribing Safety Interventions for Insulin (PEPSII) questionnaire included a list of 22 interventions derived from a recent systematic review on insulin prescribing quality improvement strategies [5], along with the research team's working knowledge of current practice. Three specialist diabetes pharmacists from different hospitals across England provided input to enhance face validity. The local diabetes patient public involvement group were consulted throughout the conceptualization and design of the questionnaire, and items regarded as very important for people with diabetes were included in the tool (e.g. self-administration and self-management policies). Additional input from the multidisciplinary diabetes inpatient group at the host trust (including consultant diabetologists, registrars, diabetes specialist nurses and pharmacists) enabled representation of the views of a range of professions involved in insulin safety.
The questionnaire included both system-level interventions and prescriber-orientated interventions. Interventions were organized into the following categories: provider education, decision support, team changes, prescribing and dispensing systems, policy, and restrictive measures [5,8].
A five-point Likert scale was used to determine the perceived effectiveness of each intervention on a scale of 1 (not effective) to 5 (very effective). We did not ask respondents to base their answers objectively on local data due to the expected heterogeneity in practice and available measures across hospitals. If respondents used insulin prescribing interventions that were not included on the list, they could describe these in an open-ended question and indicate on the Likert scale how effective they were perceived to be. This enabled the description of a broader range of currently used interventions, which could be used in future iterations of the questionnaire.

Questionnaire validation
To examine the content validity of individual items, a panel of four specialist diabetes pharmacists, representing the target population, was appointed. Each panellist was asked to score individual items as either 'essential', 'useful, but not essential', or 'not useful'. Their opinions on the relevancy of items used in the tool were quantified using the calculated level of inter-rater agreement (Fleiss kappa). Internal consistency or reliability of the questionnaire was estimated using Cronbach's alpha (a). Score distributions across intervention categories were also examined.

Use of the questionnaire
After initial validity testing, the PEPSII questionnaire was used as part of a cross-sectional descriptive survey of insulin What's new?
• Insulin prescription errors still commonly occur in the hospital setting.
• Many strategies have been developed to overcome possible hindrances to guaranteeing the safety and quality of insulin prescriptions, but no common tool has been developed to identify the best fit for both strategy and local organizational requirements.
• Outreach team review, mandatory insulin education and local guidelines on managing hypoglycaemia were the highest-scoring interventions.
• The uptakes of both the insulin passport and insulin order forms were low, and these interventions were not perceived to be effective.
prescribing practice and intervention use across all National Health Service (NHS) hospital trusts in the UK [8]. The entire population of NHS hospital trusts was chosen as the sample. At the time of study, this included 150 trusts in England, 14 in Scotland, six in Wales and five in Northern Ireland, according to the respective NHS webpages. Questionnaires were disseminated by post and online between January and April 2019 to a single respondent at the main acute hospital for each trust. The responses were anonymized and data confidentiality was maintained throughout the study. The collected data were used to describe perceived effectiveness of insulin prescribing interventions in relation to self-reported variables such as hospital demographics and intervention use.

Data analysis
Data from completed online and paper surveys returned by 10 May 2019 were inputted into Microsoft Excel 2016. Data input was checked by a second researcher and any unclear responses were discussed and resolved by joint decision. The dataset was exported to SPSS (IBM v24) for descriptive statistical analysis. The main unit of analysis was the average score of perceived effectiveness (out of 5) for each intervention.
Independent-samples t-tests were used to identify any differences in the means between groups of binary categorical data (e.g. hospitals who had a diabetes pharmacist, hospitals that used the intervention), and one-way ANOVA with Turkey's range test was used to compare means among groups of >2, such as hospital type, size and country (England, Wales, Scotland and Northern Ireland). A P value below 0.05 was taken to indicate statistical significance.
The analysis was restricted to the recorded responses for each item in accordance with the descriptive univariate analysis performed [19]. Individual item non-response was not compensated for with imputation methods or weighted adjustment methods. Any 'unsure' responses regarding categorial data (e.g. the current use of an intervention) were excluded from the subgroup analysis.

Results
Validation of the questionnaire All 22 items included in the PEPSII questionnaire were regarded as 'essential' or 'useful, but not essential' by panellists (Table S1). The overall questionnaire and kappa values showed acceptable content validity and fair inter-rater agreement. The overall Cronbach's a value of the questionnaire was 0.867, indicating good reliability (Table 1). An analysis of the individual items showed that the respondents tended to answer all items and items were well correlated. The number of items, possible range and number of respondents for each category is presented in Table 2. There was no bunching of scores at either extreme. There was a modest ceiling effect for the provider education category (44% of respondents selected the highest possible score on the questionnaire), and a slight ceiling effect for the policy category (23% of respondents selected the highest possible score). Negligible floor effects were observed across all categories.

Questionnaire use
After excluding duplicate or incomplete responses (e.g. those including demographic data only or from unidentified hospitals), data from a total of 92 hospitals (53% of 175 organizations) were eligible for analysis. Responses were received from 79 out of 150 (53%) English hospital trusts, five out of six (83%) health boards in Wales, three out of 14 (21%) health boards in Scotland, and all five trusts in Northern Ireland (100%; Fig. 1). Most of the 92 respondents were from teaching hospitals (n=39, 42%) or district general hospitals (n=37, 40%) and provided specialist inpatient diabetes services (n=79, 86%).
The highest overall mean score for perceived effectiveness was achieved for outreach team review (4.26 out of 5), followed by mandatory education on insulin safety for clinical staff (4.15), and local guidelines on managing hypoglycaemia (4.09). Modest scores were reported for electronic prescribing (3.87) and dedicated insulin prescribing charts (3.76). The lowest-scoring strategies included the insulin passport (2.80) and the use of dedicated insulin order forms for dispensing (2.65). The mean scores for perceived effectiveness of all insulin prescribing safety interventions included in the questionnaire are presented in Table 3.
The factor that best predicted a higher average score of perceived effectiveness of an intervention was its current use by organizations. This was observed for most interventions, but was significant for Tallman lettering on prescriptions (P = 0.001) and restricting the use of concentrated insulin (P = 0.007). Self-administration policies, extra requirements for medicines reconciliation of insulin on admission, electronic prescribing and the insulin passport were regarded as less effective by organizations currently using them compared to those that were not (Table 3). Scores did not vary significantly among hospital types (e.g. teaching hospital, district general, community, mental health), countries, or between those with a specialist diabetes pharmacist and those without. The only intervention that produced a significantly different score between hospitals of different sizes was outreach team review (4.69 for >1000-bed hospitals vs 3.57 for <200-bed hospitals; P = 0.034). All interventions returned a minimum score of 1 and a maximum score of 5 amongst respondents. Figure 2 shows the percentage of organizations that classed each intervention as 'very effective' (4 or 5 on the Likert scale).
Some respondents used the open question to describe the use of interventions that were not included in the questionnaire. These are listed in full in a separate paper [8]. Highly rated interventions included pharmacist-delivered insulin prescribing feedback, pharmacy-led self-administration assessment, the use of prioritization software to enable pharmacist review of insulin prescriptions and brand nameonly prescribing of insulin (all scored 5 out of 5). The lowest rated interventions included non-mandatory insulin e-learning and insulin resource folders (both scored 2 out of 5).

Discussion
The PEPSII questionnaire has been developed and validated to describe the perceived effectiveness of a range of insulin prescribing interventions, with good reliability. Mandatory insulin safety education for clinical staff was one of the most highly rated interventions, despite education being regarded more generally as a relatively weak strategy to prevent error [20]. This aligns with the literature documenting prescribers' lack of confidence and knowledge with regard to insulin, and suggests the need for increased educational efforts at postgraduate level. This may include face-to-face teaching [13], structured educational outreach [21], and online e-learning modules, which are the most common digital learning resource shown to be effective in teaching prescribers the required knowledge and skills [22]. More novel approaches, such as short educational videos [23] and gamification [24], have also produced positive outcomes. Our results support the systematic review findings that educational interventions are most effective when they are mandatory rather than voluntary [5,25].
Localized hypo-and hyperglycaemia management guidelines for prescriber support were perceived by most respondents to be very effective for promoting insulin prescribing safety. Although it is acknowledged that guidelines are not always adhered to, they represent a measurable standard of quality and are often prescriptive in nature, which can modulate prescriber uncertainty. These factors may contribute to their perceived effectiveness in promoting safe insulin prescribing. Policy interventions for insulin   N = the total number of respondents answering the item (missing data for each item are not included). Results are presented in order of overall average score and have been subcategorized according to whether the intervention is in current use in the organization. Data were not included in the subcategorization if respondents answered 'unsure' to the use of the intervention at their organization. Results with asterisks (*) indicate statistical significance (P <0.05). prescribing safety were also scored highly by respondents. Insulin self-administration and self-management interventions were regarded by our patient public involvement group as extremely important for people with diabetes, and are recommended by both the National Institute for Health and Care Excellence (NICE), and the Joint British Diabetes Societies for Inpatient Care. Our results suggest that the implementation and impact of self-administration interventions in particular would benefit from further study using complexity-sensitive evaluative methods.
Prescribing systems, such as electronic prescribing and dedicated insulin prescription charts, were given only a modest rating with respect to their perceived ability to promote insulin prescribing safety. As electronic prescribing was perceived to be more effective by respondents not currently using it, this may indicate optimism regarding the ability of electronic systems to reduce insulin prescribing errors in hospitals using paper-based systems. In light of the current drive to implement electronic prescribing across all hospitals, we emphasize the need to carefully design and implement these systems in such a way that the potential benefits regarding insulin prescribing safety can be maximized, and the shortcomings, such as any negative impacts on healthcare professional working practices, can be minimized [26].
Although in general, restrictive measures have been cited as more effective systems-focused interventions compared to person-centred interventions, such as education [20,30], we found that these types of interventions were, overall, less highly regarded for insulin prescribing. This may be attributable to the need for system flexibility on account of the non-standard and individualized dosing regimens associated with insulin. The average scores for this category were, however, higher in hospitals that used them compared to those that did not.
We found that respondents did not consider the insulin passport to be effective for promoting insulin safety in their organizations. Although the insulin passport is currently recommended by NICE, other studies have reported low uptake and support because of concerns associated with its use [27,28]. These include its unmanageable trifold design, lack of availability, an absence of integration into standard processes (e.g. medicines reconciliation), uncertainty about who is responsible for both issuing and updating them, and increased input required from staff to educate patients about their use. The insulin passport therefore may only benefit a small number of patients: those with carers or high capabilities are unlikely to need it, and those with low capabilities would be unable to use it [27]. Modified versions of the insulin passport (e.g. that include dosage information) have been more successful locally, however, and the e-diabetes passport, which incorporates insulin prescription information, may provide a promising alternative [29]. Dedicated insulin forms for dispensing were also regarded as ineffective and were seldom used. This may reflect the implementation of integrated electronic prescribing and dispensing systems negating their use, or other factors affecting local processes for ordering medication.
Despite being able to identify interventions that are perceived to be more effective than others in general, both the standard deviations and the maximum/minimum scores for individual items indicate that opinion varied considerably among respondents. Perceived intervention effectiveness may, therefore, be influenced by individual organizational contexts, or other factors such as respondents' experiences and position/role. As intervention use was shown to be associated with higher perceptions of effectiveness, the heterogenous use of interventions across the sample may have resulted in some biased estimates. This may be further explored with qualitative research methods. Interventions were categorized a priori because the purpose of the survey was to determine the perceived effectiveness of many interventions across a heterogenous sample. As we were not developing a psychometric scale to determine the dimensions of perceived effectiveness as a construct, we did not seek to undertake factor analysis to inductively generate categories. However, this may be an interesting avenue to pursue for research relating to specific interventions.
Whilst the intervention list was obtained from the existing literature and frontline staff from multiple hospitals, we recognize that it is not exhaustive. Other interventions, such as insulin safety campaigns, prescriber feedback, insulin safety huddles and other bespoke strategies described by respondents are not represented in the statistical data analysis. Interventions targeted at insulin supply or administration were also excluded. As respondents were asked to describe their opinions on current practice, recall bias is unlikely to impact results, although non-response bias could have affected the results. We chose to sample a wide variety of hospitals from across the UK to provide a broad and generalizable insight into current opinion, and our response rate was comparable to other similar surveys. The PEPSII questionnaire may, however, be used by single organizations to elicit the opinion of a wider breadth of frontline staff regarding insulin prescribing intervention efficacy for the purposes of more localized quality improvement efforts.
Implementing and evaluating prescribing safety strategies requires time, commitment and resources, as well as behaviour and culture change efforts in the local context. The capacity of individual organizations to achieve this varies. Nevertheless, the pressing need to improve insulin prescribing safety remains. Results from this study highlight interventions that are regarded as effective by a large sample of hospitals, and may prompt further exploration of salient interventions that have been recommended nationally, such as insulin self-administration, electronic prescribing and insulin safety education.
In conclusion, insulin prescribing is a common but risky process that has the potential to cause unintentional harm to people with diabetes. With the use of a validated survey tool, we have described the perceived effectiveness of a variety of interventions to help improve insulin prescribing safety across a wide range of hospitals throughout the UK. Outreach team review, mandatory insulin education, local guidelines for managing hypo-and hyperglycaemia, and insulin self-administration policies were perceived to be the most effective interventions at promoting insulin prescribing safety overall. These results provide a useful insight into which interventions may be more successful at promoting insulin safety, which is particularly useful considering the paucity of evidence documenting the impact of interventions on insulin errors. As such, our results may facilitate the direction of future insulin prescribing safety improvement efforts for the benefit of patient care.

Funding sources
This study forms part of a PhD project and was not associated with any funding. implementation of patient handheld medication tools to improve information transfer: a qualitative study.

Supporting Information
Additional supporting information may be found online in the Supporting Information section at the end of the article. Table S1. N E = number of experts who rated the item as essential. N U = number of experts who rated the item as useful but not essential. No items were rated as 'not useful' (N NU ) by the panel. Agreement (%) = Pi value calculated using Fleiss Kappa.