Healthcare librarians and the delivery of critical appraisal training: attitudes, level of involvement and support

Authors


Michelle Maden-Jenkins, Clinical Information Specialist, Aintree Library and Information Resource Centre, Edge Hill University, University Hospital Aintree, Lower Lane Fazakerley, Liverpool L9 7AL, UK. E-mail: madenm@edgehill.ac.uk

Abstract

Background:  Although healthcare librarians are undertaking training in critical appraisal skills, what is not so clear is the impact of the training on the understanding and dissemination of these skills.

Objectives:  This study aims to examine the attitudes of healthcare librarians towards delivering critical appraisal training and their level of involvement.

Methods:  A questionnaire survey of 57 library services across 48 NHS Trust Library Services in north-west England followed up with 21 semi-structured interviews.

Results:  Seventy-three per cent of respondents felt that they ought to be involved in delivering critical appraisal training, however less than a third (29%) are actually involved. Librarians are involved in critical appraisal facilitation at various levels.

Conclusions:  Debate continues over the extent of librarian involvement in delivering critical appraisal training. As long as healthcare librarians recognise their own capabilities and identify the boundaries within which they feel comfortable then there is no reason why they should not be involved in delivering critical appraisal training.

Key Messages

Implications for Practice

  • • Critical appraisal training for healthcare librarians should be more in-depth and range from introductory sessions to more in-depth sessions on research methodology and statistics.
  • • Training sessions should be provided on a regular basis to capture new staff entering the profession and provide refresher sessions for existing staff.

Implications for Policy

  • • Healthcare librarians can be involved in delivering critical appraisal training at a level suited to their level of knowledge and skills.
  • • If required, librarians should look to collaborate with others to deliver critical appraisal training.
  • • Ways in which teaching statistics to librarians should be explored in terms of the level and context.
  • • Examine ways in which a resource for supporting librarians with critical appraisal could be established.

Introduction

Two interlinking factors have contributed to the increasing involvement of healthcare librarians in critical appraisal. Initially, it was a reactive response to changes within the end-user environment the rise of evidence-based practice (EBP) and a need for librarians to develop new skills in order to support these changes. From this devolved a more proactive approach with the rise of EBP within the information profession as a whole.1

Although it is evident that librarians are taking up training in critical appraisal,2,3 what is not so clear is the impact of the training on the understanding and dissemination of these skills both in relation to professional practice and the cascading of the skills through the delivery of critical appraisal training to end-users. On the one hand, personal reflections of librarians being trained themselves in critical appraisal indicate an improvement in critical appraisal skills and understanding of statistics,3 whereas on the other hand, in an evaluation of a critical appraisal workshop Booth and Brice,2 found that several participants had difficulty in understanding statistics.

In terms of the dissemination of critical appraisal skills, Brettle4 models the roles played by health information professionals (HIPs) in EBP. At the basic supportive level, HIPs are involved in the teaching of critical appraisal (capacity builders), although the level of involvement itself is not defined. At the intermediate level, there is the role of critical appraiser, either the HIP is appraising articles to produce evidence bulletins for health professionals (or critical appraised summaries of literature searches), or they are appraising the library literature. At the advanced level, critical appraisal becomes purely an active role to inform the evidence base and extends the role played by the HIP to researcher.

This study focuses specifically on one of the contexts in which healthcare librarians may find themselves being involved in critical appraisal, the dissemination of critical appraisal skills through the training of healthcare professionals. The literature gives the impression that healthcare librarians ought to be involved in critical appraisal, but there is no clear consensus concerning either the extent of the involvement of the healthcare librarian in critical appraisal training or how librarians themselves feel about their involvement in the process.

This study examines the attitudes of healthcare librarians towards involvement in critical appraisal skills training, the extent to which they are involved in the delivery of critical appraisal training and their support needs. Another study to be submitted to Health Information Libraries Journal explores the barriers faced by UK healthcare librarians in delivering critical appraisal skills training. The full study is reported elsewhere.5

Methods

As the study aimed to investigate both the level of involvement and the opinions of librarians towards critical appraisal, a mixed method approach was adopted.

Sample

Questionnaire.  A covering letter and questionnaire were emailed to all healthcare librarians in the north-west with responsibility for end-user training.6 The questionnaire sample consisted of 57 library services across 48 NHS Trust Library Services and 9 HE Library Services. The questionnaire was emailed directly to librarians identified as having direct responsibility for end-user training. Two reminder emails with the attached questionnaire were sent out.

Interviews.  Interviewees were librarians who had responded to the questionnaire and had agreed to be interviewed.

Data collection

Questionnaire.  The questionnaire was designed to elicit both quantitative and qualitative data and was piloted prior to distribution amongst five healthcare librarians and the study supervisor. No amendments were required (see Supporting Information).

Interviews.  Interviews were semi-structured, following a number of themes and areas identified from responses to the questionnaire (see Supporting Information). Telephone interviews were recorded via speakerphone on two dictaphones and transcribed in order to maintain reliability.

Confidentiality and anonymity of replies were assured in the covering letter and verbal consent was obtained prior to interview. Ethical approval was obtained from Liverpool John Moores University before the survey commenced.

Data analysis

Quantitative data from the questionnaire and semi-structured interviews were analysed using spss version 14 (SAS Institute, Cary, NC, USA). Qualititative data were analysed using the thematic analysis constant comparison method.7

Response rate

Questionnaire.  A total of 55 questionnaires were returned. In seven organisations, more than one librarian had responsibility for end-user training (cross-site library services) and therefore returned more than one questionnaire per organisation. The overall response rate per library service was 48/57 (84%).

Telephone interviews.  A total of 21 librarians were interviewed. Seven interviewees were not involved at all in the delivery of critical appraisal, six were intending to get involved and eight were already involved.

Respondent characteristics

Responses were received from 7 (13%) Clinical Librarians, 7 (13%) Outreach Librarians, 25 (45%) Library Managers, 16 (29%) other (e.g. Librarian, Learning Advisor, Knowledge Support Librarian). Thirty-one respondents (56%) work in an Acute Trust, 10 (18%) in Primary Care, 5 (9%) in a Mental Health Trust, 5 (9%) in Higher Education and 4 (7%) in other types of organisations. The majority (42; 76%) had been in post more than 2 years, 7 (13%) for 1–2 years, 2 (4%) for 6 months–1 year and 4 (7%) for <6 months.

Table 1 displays the length of time in post and librarian involvement in critical appraisal. Table 2 displays the organisation type and librarian involvement in the delivery of critical appraisal training.

Table 1.   Length of time in post and involvement in the delivery of critical appraisal training
Time in postDelivering critical appraisal training
YesNo
  1. *One respondent failed to answer these questions.

<6 months04
6 months–1 year02
1–2 years25
>2 years1427
Total*1638
Table 2.   Organisation type and involvement in the delivery of critical appraisal training
OrganisationDelivering critical appraisal training
YesNo
  1. *One respondent failed to answer these questions.

NHS Acute Trust1119
NHS Mental Health Trust14
NHS Primary Care Trust19
Higher Education14
Unknown22
Total*1638

Results

Healthcare librarian attitudes towards critical appraisal

All respondents (55; 100%) felt that healthcare librarians need to be familiar with the resources (e.g. checklists, preappraised articles, etc.) to support critical appraisal. When asked whether they thought healthcare librarians require the skills themselves to be able to appraise critically 47 (85%) said yes 8 (15%) said they did not know. Forty-three (78%) thought healthcare librarians ought to be involved in the delivery of critical appraisal training, 12 (22%) said they did not know.

The analysis identified several reasons why librarians ought/ought not to be involved which were grouped into six themes: Role expectations, Professional development, Service impact, External factors, Credibility and Communication.

Role expectations.  Above all the majority of librarians felt that they ought to be involved because they see it as part of their existing role. In particular, the main reason cited was the desire to meet the needs of end-users.

‘As champions of evidence based medicine/practice, healthcare librarians should have an understanding of the processes and be involved in the delivery of training.’

Many librarians also saw it as a natural extension of their existing role in relation to the skills they already possess.

‘Critical appraisal is sort of like the next step, you read the paper and make sense of it and I think it’s sort of almost a natural extension of what we’re already doing.’

Many also believed that critical appraisal training ought not to be delivered in isolation but in conjunction with literature searching and therefore librarians are well placed to deliver it. Indeed, many saw literature searching as being part of the critical appraisal process and vice versa. Several librarians went further indicating that critical appraisal skills help to inform an effective search.

‘If we can’t train people to discriminate between reliable and unreliable evidence then how useful are their literature searching skills going to be?’

A small number made a link between the wider information literacy framework and critical appraisal while others mentioned the role of the Clinical Librarian. Two were Clinical Librarians themselves and saw the delivery of critical appraisal training as natural to their role, whereas a third person, a Library Manager felt the responsibility for delivering such training lay with the Clinical Librarian.

‘I see librarians as providing a supporting role to Clinical Librarians or specialists in the field in terms of the delivery of the training.’

Professional development.  Librarians also regard involvement in critical appraisal training as an opportunity to inform the development of their own role and skills. Some also felt that involvement in critical appraisal training would help them to start questioning their own practice and help them become more evidence based within their own profession.

‘…[involvement with critical appraisal training] will help us represent, understand and ask questions for developing our practice.’

Service impact.  A number of respondents perceived that being proactive and getting involved in critical appraisal training would lead to a more effective library service and thus have an impact on the value of the library service.

‘…to be able to provide an effective information service to users we need to demonstrate that our skills lie in more than just retrieving the literature.’

One librarian felt that involvement would aid healthcare practitioners to make informed decisions on patient care and therefore, indirectly impact on patient care outcomes.

‘The library would be seen as a vital service in the provision of information to support patient care.’

Credibility.  Interestingly, a couple of librarians indicated that involvement in critical appraisal training would enhance their own status and increase the credibility of the librarian within a clinical/academic team.

‘If you are involved in critical appraisal they [healthcare professionals] are more likely to accept that you know what they are talking about.’

Communication.  A small number of librarians felt that involvement in teaching critical appraisal would improve communication with healthcare professionals by adding a more objective viewpoint as they would be teaching from a non-technical perspective.

‘Where participants have little confidence it can be very persuasive to have someone in the room who is not a clinician or statistician but who can explain concepts in a non-threatening way without patronising.’

External factors.  Finally, respondents also reported benefits for healthcare professionals, e.g. by providing critical appraisal training when other departments within their organisation were either not providing it at all or were providing training for researchers only.

Unsure of involvement

For a number of librarians who were unsure about getting involved in delivering the training, it is the fact that there may be others more experienced in critical appraisal than themselves and who are better placed to deliver the training.

‘I am not sure whether healthcare librarians MUST be involved in delivering critical appraisal training or not. I think critical appraisal training should be delivered within the organisation, by the most appropriate people … I don’t think that librarians need to be solely responsible for delivering the training, but I think there is a role here.’

One librarian also added that staffing level, individual skills and knowledge of the librarian also affected the decision to get involved in training.

Caveats to involvement

Some librarians mentioned caveats to being involved with the delivery of critical appraisal training. The most common caveat related to the training of librarians themselves.

‘Librarians do need extra training to be able to deliver the training successfully.’

Some stated that they were happy to be involved provided it was at a basic level. For others, collaboration was seen as the best way to get involved in the delivery of training,

‘I think in practice that most existing librarians would be better off delivering critical appraisal in tandem with a member of research or medical staff who has a sound grasp of the more technical aspects of research.’

To a lesser extent, other caveats related to the attitude, knowledge and skills of the librarian to be able to deliver critical appraisal training. For example in instances where,

‘…the librarian is happy enough, confident and knowledgeable enough to deliver the training.’

Ought not to be involved

Only two librarians felt that they ought not to be involved citing a lack of skills, knowledge and confidence, with one questioning the appropriateness of librarian involvement.

‘Should a librarian with a first degree in e.g. Art, Sociology, Media Studies be showing a consultant how to interpret a complex meta-analysis. I personally think not.’

Extent of librarian involvement in critical appraisal training

Sixteen (29%) respondents are involved in delivering critical appraisal at their organisation. Of the 39 (71%) respondents who are not involved in delivering critical appraisal at their organisation, 19 (35%) said critical appraisal was delivered by others at their organisation, 12 (22%) said no critical appraisal training was offered at their organisation and 7 (13%) did not know whether critical appraisal was delivered at their organisation (one respondent failed to answer this question).

Fifteen (94%) of the 16 involved in critical appraisal training deliver sessions in collaboration with colleagues. Only one respondent delivers the training alone. Just over half (9/16) of those involved collaborate with other librarians, six collaborate with a statistician, three with colleagues from the Research and Development department, one with a clinician, one with Clinical Effectiveness and three with other academics (N.B. librarians may collaborate with more than one colleague).

Subject areas covered in critical appraisal training

Figure 1 highlights the areas of critical appraisal that librarians and their colleagues are involved in delivering. Other aspects covered in critical appraisal training include sessions on ‘why we do critical appraisal’, peer review, formulating a clinical question and developing a search strategy.

Figure 1.

 Topics covered in critical appraisal training

Figure 2 outlines the study designs covered by librarians in their critical appraisal sessions. Diagnostic and screening tests were also covered by one librarian in collaboration with a colleague.

Figure 2.

 Study designs covered in critical appraisal training

Level of librarian involvement in delivering critical appraisal training

The interviews explored further the extent to which librarians ought to be involved in the delivery of critical appraisal training. The majority of those interviewed felt that healthcare librarians should collaborate with others rather than attempt to deliver the training on their own.

‘They shouldn’t lead it I don’t think, but I think that they should have a part to play along with people that can give more information, more in-depth statistical parts of it.’

There was no consensus on the depth of librarian involvement in the delivery of critical appraisal training. Responses ranged from simply signposting healthcare professionals to critical appraisal resources or delivering basic skills (with this view coming mostly but not entirely from those not currently involved) to being one of the main delivers of it (according to the majority of librarians already involved).

‘I personally feel that the librarians role is kind of in the background. I think as far as they need to go is to provide toolkits, to provide signposts for people.

I think it should be an intrinsic part of what they do, if critical appraisal training is going to be done within the Trust.’

The overwhelming view from all interviewees was that librarians ought to know about statistics but that they only need a basic level of understanding.

‘I think you definitely need some kind of understanding or you would trip up, you can’t mess up that bit. I don’t think we need too in-depth knowledge but we certainly need an understanding to be able to talk about it.’

Two librarians already involved in delivering training discussed the statistical aspect further.

‘I think there is a difference between understanding which you do need and the ability to interpret.

You don’t need to know how to calculate them or anything like that you just need to know what they mean.’

A couple of librarians expressed concern over statistics.

‘I suppose in an ideal world they would but I’m not sure how realistic that is. I mean I’ve never been good at statistics ever and I’m not going to get good at it just to do this.’

Whereas another did not feel it was appropriate for librarians to deliver statistical training.

‘I have attended quite a bit of training on it myself and support this with ongoing self-study but I still feel it is not appropriate for Librarians to deliver quantitative critical appraisal training.’

Training and support needs

Librarians were asked to rate their level of understanding of various terms relating to the concept of critical appraisal, research methodologies, statistics/measures of effects and bias.

Knowledge of critical appraisal terminology

Critical appraisal.  Most librarians have some understanding of the general concepts used in critical appraisal (see Fig. 3).

Figure 3.

 Librarian understanding of critical appraisal terminology

Research methodololgies.  More librarians felt that they fully understood the various research methodologies than general critical appraisal concepts (see Fig. 4).

Figure 4.

 Librarian understanding of research methodologies

Statistics/measures of effect.  There was a shift in the level of understanding of statistical concepts and measures of effect with the majority of librarians having some understanding of these concepts (see Fig. 5).

Figure 5.

 Librarian understanding of statistics/measures of effect

Bias.  The bias related concepts were ones that librarians least understood (see Fig. 6).

Figure 6.

 Librarian understanding of bias terminology

Training needs

Forty-seven (86%) respondents had attended some form of training in critical appraisal. Sixteen had attended more than one training course, 28 respondents had attended a regional workshop, 12 had attended the CASP Train the Trainers workshop, seven had attended other CASP training, one had attended the BMA training, and 16 had attended other training [including the Oxford Centre for Evidence Based Medicine (EBM) Teaching EBM, Clinical Librarian Conference session, own Trust workshops].

Figure 7 outlines the future training librarians would like to receive in critical appraisal.

Figure 7.

 Future critical appraisal training suggestions

Other training requested include ‘train the trainer’– how do you teach critical appraisal, refresher sessions, applying the knowledge in own practice.

Critical appraisal resources

Figure 8 outlines the types of information librarians would like to have in a critical appraisal resource.

Figure 8.

 Critical appraisal resource suggestions

Other resources that were suggested included references to website links/information resources handout, key texts, teaching plans, group work exercises, literature searching PICO (Patient, Intervention, Comparison, Outcome; a technique used in EBM to help healthcare professionals breakdown a clinical question into search terms) examples, a ‘review’ or ‘appraisal’ of resources currently available, further training opportunities and advanced literature searching tools (e.g. methodological search filters).

Discussion

This study set out to investigate the attitudes of UK NHS and HE healthcare librarians in the delivery of critical appraisal training, to quantify the level of their involvement in critical appraisal training and to identify training and support needs for increasing librarian involvement in critical appraisal training.

The consensus among healthcare librarians is that they feel that they ought to be familiar with critical appraisal resources and to a slightly lesser extent, they feel that they require the skills to be able to appraise critically research. Although just over three-quarters of those surveyed felt healthcare librarians ought to be involved in the delivery of critical appraisal training, this did not translate into practice with less than a third of respondents actually involved.

Attitudes towards critical appraisal

The overwhelming view from healthcare librarians in relation to their involvement in critical appraisal training concurs with earlier observations of supporting the culture of EBP,8 of being seen as a natural extension of their existing skills and of increasing their own credibility amongst the healthcare profession.9

However, this study also highlights new areas of opportunity for librarians both at a personal and service level. Certainly, an appreciation of the levels of evidence hierarchy and research methodology/critical appraisal terminology would assist in their role as ‘evidence filterers’.4 Librarians also felt that involvement in critical appraisal training added value to traditional library services. According to Weightman and Williamson,10 one of the key measures of the benefits of library services is a demonstration of their impact on patient care. In providing healthcare professionals with the skills to appraise the literature librarians can indirectly contribute to more efficient clinical decision-making relating to patient care.

The majority believe that there is a role for librarians in delivering training and that they are happy to be involved, but there was no agreement on just how far they should be involved. While the Clinical Librarian was seen by some as having an advantage in delivering critical appraisal training due to the nature of their work in undertaking literature searches and close working proximity to clinicians, most librarians did not regard the delivery of critical appraisal training as being specifically the domain of the Clinical Librarian, a fact also borne out by the results which demonstrate that critical appraisal training is delivered by a variety of job roles across different sectors.

There was also no consensus on whether librarians should be the main lead in delivering training and the level of involvement expected from them. Respondents views often reflected whether or not they were already involved in delivering critical appraisal training. There was more agreement on whether librarians ought to be involved in delivering statistical content. The majority believed that some understanding was required but only a basic level in order to inform their own understanding more so than to teach it. If librarians were to be involved to any extent in delivering statistics, then the emphasis should be on the interpretation of statistics rather than the calculation.

Very few respondents felt that librarians ought not to be involved in critical appraisal training because of a lack of statistical knowledge. In addition, the results demonstrate that librarians can be involved in critical appraisal training without the need for delivering statistical content and it is recognised that not every librarian will be able to, or want to deliver statistical content. For those who want to be involved in statistics, ways in which these librarians can be supported through collaboration and training need to be explored.

Level of librarian involvement

This study highlights that while librarians are involved in the delivery of critical appraisal training, collaboration with other colleagues is the most common approach to delivering the training. In doing so librarians can share the workload and focus on the areas in which they feel most comfortable with delivering. Most librarians are involved in delivering basic sessions (e.g. signposting to critical appraisal resources) and increasingly collaborate with colleagues on sessions involving appraisal of an article, statistics and research methodology. Statistical aspects are mostly covered by a colleague with little involvement from librarians. While librarians are collaborating with people from outside the library and information profession, more than half deliver sessions in conjunction with other librarians.

Librarians are more involved in delivering critical appraisal training on the ‘higher levels of evidence’ (e.g. systematic reviews) than other study designs. In particular, systematic reviews is the most popular training session and this is delivered by more librarians on their own than any other research methodology and was the only study design taught by all respondents either on their own or in collaboration. This could be due to the fact that it is possible to deliver critical appraisal of systematic reviews without any statistical content and that there is more emphasis on appraising the search process, an area with which librarians are more familiar.

Training and support needs

The results indicate that librarians are keen to learn critical appraisal skills, findings which are consistent with other studies.2,3,11 A number of factors identified in this study can help to inform the content and structure of future critical appraisal training sessions for librarians as well as other areas of support. Firstly, the extent of librarians’ knowledge of critical appraisal terminology varied greatly and this suggests that a number of training sessions should be offered at different levels ranging from a basic introduction to critical appraisal to more intermediate sessions on research methodologies and statistics.

Supporting this is the fact that healthcare librarians were also more likely to be involved in delivering training once they had been in post for more than 2 years which suggests that many may enter the profession with no prior experience of critical appraisal. In addition, 16/47 librarians had attended more than one critical appraisal training session indicating that a number of librarians will have already been introduced to basic critical appraisal skills. This also raises questions on how many training sessions are considered enough to confidently and competently deliver critical appraisal training and how specific should the training be.

Above all statistics was regarded as the main training need, suggesting that librarians are keen to gain at least a basic understanding of the topic area. The challenge here is the way in which this topic area is explained to participants focusing on the interpretation rather than the calculation. Booth,12 however, heeds the following warning about training librarians in critical appraisal concepts.

‘Library and information practitioners respond to the challenge of critical appraisal in their time-honoured fashion by seeking to learn more about research design, by becoming more familiar with statistics … This quest, this appetite for knowledge, is potentially insatiable … the more they learn the more they recognise that there is much more yet to learn.’ (p. 52)

He goes on to suggest that librarians therefore should equip themselves with strategies to deal with situations in which the librarian does not know the answer to a question posed during the training session.12 As highlighted in this study, one way could be to collaborate with colleagues to deliver critical appraisal training, another is to prepare in advance possible scenarios for dealing with difficult questions.

Finally, the sample chosen may be exposed to more critical appraisal training than other regions. The survey did not seek to test respondents levels of understanding of critical appraisal terminology, merely self-report, nor did it investigate librarian involvement in critical appraisal outside of training, e.g. as part of individual professional practice.

Conclusions

This study demonstrates that healthcare librarians can be actively involved in critical appraisal training at a level suited to the ability of the individual librarian. This sense of control allows librarians to deliver aspects of critical appraisal training that they themselves feel secure and confident with. As long as healthcare librarians recognise their own capabilities and identify the boundaries within which they feel comfortable, then there is no reason why they should not be involved in the delivery of critical appraisal skills training.

This study needs to be viewed in context with the barriers faced by healthcare librarians in delivering critical appraisal (in a future issue of Health Information and Libraries Journal) in order to understand the wider issues faced by librarians, to identify approaches to overcoming the barriers and increase librarian involvement in delivering critical appraisal skills training.

Acknowledgements

The survey was conducted as part of a Masters dissertation5 funded by Edge Hill University and the North West Heath Care Libraries Unit.

Ancillary