Evaluating the impact of library user training programmes across Thames Valley Strategic Health Authority in the UK

Authors


Anne Gray, Staff Library, Milton Keynes General Hospital, Standing Way, Eaglestone, Milton Keynes MK6 5LD, UK. E-mail: outreach.librarian@mkpct.nhs.uk

Abstract

Background:  This study was undertaken to provide evidence that library outreach projects in the NHS in the Thames Valley Strategic Health Authority area were effective and could justify continued funding.

Objectives:  The object was to assess whether information literacy skills training taken up by health professionals impacts on their confidence and skills in using electronic sources of health information.

Method:  A total of 487 health professionals taking up training on a voluntary basis were assessed by completing a self-assessment questionnaire prior to and after training to establish their information literacy skills and confidence levels. Of these, 223 trainees (46%) also returned 6-week follow-up questionnaires.

Results:  Results showed that 464 trainees (95%) intended to apply what they had learnt in a variety of ways; use of specific resources greatly increased after training and confidence in using them increased substantially. Use of search techniques increased considerably overall, whereas use of Google decreased.

Conclusions:  The major findings of this research are that training health-care personnel in the use of electronic resources impacts positively on their information literacy skills and confidence, and that the information available to them is considered to have potential value across health-care practice.

Introduction

In December 2003 the Thames Valley Strategic Health Authority launched a library strategy. One of its main objectives was to work towards equitable access to library services for all NHS personnel in the patch. This was supported by marketing and outreach programmes, the key aims of which were to improve the awareness of library services, particularly amongst people not based on main hospital sites, and to give them the skills to access electronic resources for themselves. Project funding was allocated to each of the three counties in Thames Valley to fund outreach work. Two counties, Oxfordshire and Buckinghamshire, appointed temporary outreach librarians; Berkshire used the funds to extend existing outreach programmes.

Objectives

Because these services were funded as 2-year projects, some means of demonstrating their value was needed within a relatively short time frame if they were to be continued, not only to justify embedding them into service, but also to secure permanent funding. If primary care trusts (PCTs) were to be persuaded to provide ongoing funding in a climate of financial constraint, they would require evidence of the usefulness of library outreach services in their own areas. Summaries alone of the numbers trained would not be sufficient to make the case for the continuation of outreach services, nor would citing published studies from elsewhere demonstrating the need for training.

The Thames Valley Outreach Librarians’ Group therefore agreed that in-depth evaluation of the impact of their own local programmes was needed in order to assist PCTs in making decisions about their future. It was decided to concentrate on evaluating the impact of information skills training activity only, given the multifarious nature of general outreach activities which include leaflet drops, posters, talks and attendance at meetings, many of which by their very nature defy the use of a consistent approach and robust methodology.

It was decided to take a standard approach to evaluation of training across Thames Valley to provide local outreach programmes with reliable benchmarks against which they could assess their own performance. The results from each patch could then be combined to provide Thames Valley-wide data, even though it was recognised that this would mask the variations between trainers. The evaluation would therefore be able to contribute to the wider evaluation of the effectiveness of the Thames Valley Library Strategy in broadening access to services. This included collecting quantitative data, such as trends in the numbers of service users and percentages of users within each organization registered to access electronic resources who were actively using them.

Literature search

A literature search was carried out using lisa and medline along with hand searching. The main search terms used were: impact, effect, evaluate; outreach, train, learn, skill, instruct; library, search, information, literature, seek. An article by Robinson and Lawson1 on the impact of information skills training within primary care found that most attendees felt that training had a high impact on their work, their usage of resources having increased since attending. Frequency of use, confidence in using, and awareness of resources all increased as a result of training. Garg and Turtle2 performed a critical appraisal of the literature on the effect of training on search and retrieval skills, finding some evidence of positive impact on medical students, but no clear evidence about the level of effect, whether it lasts, and the effects of teaching on fully qualified health-care practitioners. Brettle3 did a systematic review of information skills training and found very limited evidence that training improves search skills. Two studies did indicate a positive, albeit subjectively assessed, impact on patient care of information skills training. However, of the 16 studies identified, only four were of practising clinicians, with most being based in US medical schools.4,5 Several NHS report6–9 found that information skills training had a high impact on trainees’ work and led to changes in clinical practice.

Many studies about the value of training programmes in using electronic resources were therefore found, but there was limited published evidence about whether training within the NHS impacts on people's confidence with information literacy skills or changes their behaviour in the use of information.

Methods

As there was little previous published work from which to derive an appropriate methodology, the Thames Valley Outreach Librarians Group developed its own. It was based on the premise that, for an intervention to have an impact, some change in the perceptions or behaviour of the subjects would need to be demonstrated. The uptake of training in the use of electronic health information sources was entirely voluntary so no structured selection of the subjects took place, but all were practising professionals working in the NHS.

The time constraints of busy health-care personnel also had to be taken into account, so any feedback from trainees would need to be easy and convenient for them to provide. It was decided to use three questionnaires:

  • a short questionnaire, completed at the start of a training session, to establish the trainees’ baseline use of electronic resources and their information literacy skills and confidence levels;
  • a post-session questionnaire, completed at the end of the session, to establish the attendees’ immediate perceptions of the effectiveness of the training;
  • a follow-up questionnaire, sent 6 weeks after the session, asking essentially the same questions as the pre-session questionnaire. By asking the same questions, the impact of the training intervention could be assessed.

The Group realised that this self-assessment methodology might not provide as objective an approach as an external evaluation. However, it was adopted as the most practical approach given the short time frame within which an evaluation was needed and the absence of funds for a dedicated researcher. The questionnaires were used by trainers across the Thames Valley at a range of training sessions, from one-to-one sessions in the workplace, to group workshops in an IT training room, and regardless of the resources described during the training. Results from each major training programme in Thames Valley were integrated into a standard spreadsheet for analysis.

Results

A total of 487 questionnaires were analysed, covering the period August 2004 to July 2005, and 223 trainees also returned 6-week follow-up questionnaires, a response rate of 46%. The high follow-up response rate can in part be attributed to the telephone follow-ups performed by one trainer, while others relied on return by post. In addition to analysing the combined results, they were analysed by county and also by trainee profession. As these analyses did not show any particular variations from the combined results, only the combined results are presented here. Ninety-five per cent confidence intervals have been calculated and are indicated (CI) in brackets after the results described below.

Previous training in information skills had been taken by 205 of the 487 trainees (42%; CI 204.78–205.21). At the end of the session 332 (68%) said they would like to receive further training (CI 332.23–332.63), and 483 (99%) said they would recommend the training session to colleagues (CI 483.92–483.99).

Trainees were asked whether and in what ways they would use what they had learned in the session (Fig. 1); 464 (95%) said they would make use of what they learned (CI 463.86–464.04). Most trainees ticked several of the boxes [an average of 3.3 out of six options per person (CI 3.30–3.39)], indicating that the learning had wide applicability. The most commonly listed ‘other’ use was ‘teaching/training’.

Figure 1.

Use of training

Figures 2–4 show results from the 223 trainees who completed the follow-up questionnaire.

Figure 2.

Use of specific resources before and after training

Figure 3.

Confidence with specific resources before and after training

Figure 4.

Increase/decrease in use of techniques

The pre-session questionnaire asked trainees how often they had used electronic resources such as the National electronic Library for Health (NeLH), clinical databases and e-journals; the 6-week follow-up questionnaire asked the same question. Usage of all resources increased substantially, NeLH being the most used resource after training with use increasing the most (Fig. 2). Mean usage of all resources on a 0–5 scale increased from 0.87 (CI 0.77–0.98) to 1.93 (CI 1.73–2.14).

Trainees’ assessment of their confidence in using the electronic resources was assessed pre-session and after 6 weeks. Confidence in using all resources showed substantial improvement, with NeLH again showing the highest level of confidence and greatest increase (Fig. 3). Mean levels of trainee confidence across all resources on a 0–5 scale increased from 0.83 (CI 0.72–0.93) to 2.26 (CI 2.09–2.43).

In the 6-week follow-up questionnaire, trainees were also asked whether they used certain search techniques more or less since taking the training. The questions were:

Do you make use of the following techniques?

  • Use Google as your starting point.
  • Clarifying the different subjects/topics when starting your search.
  • Deciding on search terms to use in advance of your search.
  • Deciding on the most appropriate information source(s) to answer your questions.
  • Modify search terms during a search.
  • Use AND or OR (Boolean logic) to combine terms or results.

Results (Fig. 4) indicate that over two-thirds of trainees increased their use of these techniques, except for Google, which decreased considerably.

Discussion

The outreach programme was effective in reaching previously untrained staff, 58% having had no previous information skills training, although this may be an underestimate as a result of some confusion of information skills training with IT skills training. At the end of their session most trainees (68%) expressed a wish to have further training. There could be several reasons for this. It may be because trainees had been introduced to a wide range of new resources during the training, some of which they wanted to explore in more detail (e.g. clinical databases, Cochrane Library), or it may be because they wanted to consolidate their skills and gain more confidence.

One of the acknowledged problems of information skills training, particularly in the primary care setting, is to persuade health-care professionals that an awareness of evidence-based resources will be valuable to their work. As part of the NHS Knowledge and Skills Framework (NHS KSF)10 which defines and describes the knowledge and skills NHS staff need to apply in their work in order to deliver quality services, non-medical NHS staff are expected to undertake continuing personal development, and this is a strong incentive to use library and information services. Information literacy skills training can support staff in achieving a number of the competencies of NHS KSF, including Knowledge and Information Resources (IK3), Service Improvement (Core Dimension 4) and Quality (Core Dimension 5). While 90% of trainees indicated that they would use the resources for their own professional development, the heightened expectations of the skills and resources introduced during the training, and their potential impact across all areas of practice, are clearly demonstrated. Overall opinions of the usefulness of the courses were very favourable. Attendees at the sessions were largely self-selecting and keen to improve their knowledge and skills in finding information. It is not possible to determine if compulsory attendance would have altered the way the courses were received or their impact.

Usage of information resources, and especially confidence in using them, were both indicated as being much higher 6 weeks following the training. This trend was tempered by several respondents who noted that they had had little time to practise what they had learned. Use of search techniques was perceived by trainees to be greatly increased in the 6-week follow-up questionnaire.

Many people default to using Google and similar search engines because they do not know of other, more appropriate, tools available to them. Therefore, one of the primary aims of this training, in addition to teaching searching techniques, was to increase trainees’ awareness of the wealth of resources available to them. This is especially important in health care where many key resources are licensed to the NHS and not therefore found using general search engines like Google. The fact that Google usage decreased, as shown in Fig. 4, is therefore viewed as a major success.

There were many comments on the training. The most common are exemplified by the following quotes:

‘I will need to practise my new-found skills.’

‘There was a lot of information to absorb.’

‘This has clarified things and given me more confidence.’

‘Very useful and informative.’

‘Would like a follow-up session once I've had a chance to practise.’

‘Feels like I suddenly put on my glasses.’

‘You've opened a door!’

The questionnaires were designed to measure the impact of training on trainees’ perceived improvement in search skills. ‘Before and after’ questionnaires were used because they are quick and easy to administer and analyse across a large geographic area and several different trainers. Other methods, including follow-up interview and critical incident analysis, were considered but rejected because of their demands on staff time. However, a recent literature review by Lacey-Bryant and Gray11 highlighted the value of critical incident technique to measure the impact of information and library services on patient care, in particular in the community setting, so it is now hoped to develop a critical incident methodology to take forward the research described in this paper. During the study it was recognised that there was an opportunity to ask about changes to patient care and clinical practice resulting from the training, so the following additional question has since been added to the follow-up questionnaire: ‘Has your practice/patient care changed as a result of your training? If so, please give an example’.

The authors believe that the methodology described in this paper could be applied to evaluate the impact of other information skills training, both in the NHS and in other sectors. However, the limits of this self-evaluation methodology in producing objective assessment should always be borne in mind. The methodology could also be applied to compare the effectiveness of different trainers and even training methods, although in both cases, variables such as training session type and length would need to be taken into account or eliminated.

Conclusions

Early indications from this research are that training health-care personnel in the use of electronic resources impacts positively on their information literacy skills, which they expect to employ to support clinical practice and personal development. Because of the short time frame in which a report was needed for PCTs, it was not possible to assess whether the effect of the training has lasted over time, although this could be done by running the same survey after a longer time period has elapsed.

Given the small scale of this study and the self-assessment nature of the methodology, more research is recommended, possibly using a larger cohort, an objective assessment methodology and focusing on specific types of training, rather than covering a variety of sessions as in the present study.

Copies of the questionnaires used may be obtained from Anne Gray, Outreach Librarian, Staff Library, Milton Keynes General Hospital, Standing Way, Eaglestone, Milton Keynes MK6 5LD, UK, e-mail: outreach.librarian@mkpct.nhs.uk.

Acknowledgements

The authors would like to thank David Streatfield of Information Management Associates who facilitated the initial discussion activity for this evaluation project.

Key Messages

Implications for Policy

  • •  Training health-care personnel in the use of electronic knowledge resources impacts positively on their information literacy skills and therefore such programmes contribute to increasing the skills of the health-care workforce in support of the evidence-based approach to practice. Training programmes should be supported at strategic level.
  • • Information literacy skills training can support health-care personnel in achieving Dimension IK3 (Knowledge and Information Resources) and Core Dimensions 4 (Service Improvement) and 5 (Quality) of the NHS Knowledge and Skills Framework.

Implications for Practice

  • • Training health-care personnel positively impacts on their confidence and competence in information literacy. Training sessions should be provided as a core service by health-care libraries.

Ancillary