SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. Background
  4. Methodology
  5. Results
  6. Discussion
  7. Some implications for information professionals
  8. Conclusion
  9. Key Messages
  10. References
  11. References of included studies
  12. Appendix 1: eric sample search strategy

Objectives:  To systematically review the UK published literature on e-learning in the health workplace and to apply the findings to one of the most prolific UK e-learning initiatives in the health sector—the National Library for Health Facilitated Online Learning Interactive Opportunity (FOLIO) Programme.

Methods:  Sensitive searches were conducted across assia, Australian Education Index, British Education Index, cinahl, CSA Abstracts, Dissertation Abstracts, Emerald, eric, ibss, Index to Theses, lisa, medline, PsycInfo and Social Science Citation Index. Additional citations were identified from reference lists of included studies and of relevant reviews; citation tracking and contact with experts. Twenty-nine studies met the inclusion criteria and were coded and analysed using thematic analysis as described by Miles & Huberman (Qualitative Data Analysis: A Sourcebook of New Methods. Newbury Park, CA: Sage, 1984).

Results:  Five broad themes were identified from the 29 included studies: (i) peer communication; (ii) flexibility; (iii) support; (iv) knowledge validation; and (v) course presentation and design. These broad themes were supported by a total of eleven sub-themes. Components from the FOLIO Programme were analysed and existing and proposed developments were mapped against each sub-theme. This provides a valuable framework for ongoing course development.

Conclusion:  Librarians involved in delivering and supporting e-learning can benefit from applying the findings from the systematic review to existing programmes, exemplified by the FOLIO Programme. The resultant framework can also be used in developing new e-learning programmes.


Background

  1. Top of page
  2. Abstract
  3. Background
  4. Methodology
  5. Results
  6. Discussion
  7. Some implications for information professionals
  8. Conclusion
  9. Key Messages
  10. References
  11. References of included studies
  12. Appendix 1: eric sample search strategy

Within the health service, the emerging agenda for modernization and technological change places a significant imperative upon flexible, tailored and timely methods of delivery for learning for the health workforce.1 Workplace-based e-learning (WBEL) is one such delivery method which provides the potential not only to widen access to previously underserved communities of health professionals but also to overcome practical and organizational barriers for those previously accustomed to pursuing continuing professional development through face-to-face interaction.

Health information professionals are interested in WBEL for at least two principal reasons:

  • 1
    As part of the learning delivery team, health librarians in both academic and service settings have the potential to contribute their knowledge, skills and expertise to make the student experience more rewarding and productive.
  • 2
    As individuals, working within a profession at the forefront of technological and organizational change, health librarians have a considerable need to avail themselves of mechanisms for skills update and enhancement, both vocational and academic in nature.

Although e-learning is not universally suited to all preferred learning styles,2 there is increasing recognition that it can provide one potential route by which to overturn the twin barriers of lack of time and geographical isolation encountered by many learners based in the health service. Indeed, experience within the health service seems to substantiate findings from elsewhere, namely that an optimal model of delivery will include both face-to-face and e-learning components.3

Clearly, an important consideration for those commissioning e-learning, alongside concerns with effectiveness and economy, is acceptability of the delivery mode. Indeed, if there is little to separate face-to-face and e-learning modes of delivery with regard to impact and cost, as many suspect, then the quality of the student experience becomes paramount.

In 2007, the Higher Education Academy, a UK academic body charged with enabling students to enjoy the best possible learning experience, commissioned five 1-year projects on specific themes relating to e-learning in higher education. Outcomes from the successful projects, from a field of 80 submissions, were intended to inform Higher Education policy and practice and to contribute to debate on improving the student learning experience.

One of the successful project proposals came from a team of information specialists and systematic review methodologists based at the School of Health and Related Research at the University of Sheffield. Interest in their mooted topic, Enhancing the student experience of workplace-based e-learning: a systematic review and best practice framework, stemmed from extended involvement by three project team members (AB, DP and AS) in delivery of the Facilitated Online Learning Interactive Opportunity (FOLIO) Programme for the National Library for Health Librarian Development Programme. The FOLIO Programme is the largest and most successful example of WBEL amongst the UK librarian community, offered to all health librarians and information specialists serving the National Health Service (NHS) community, regardless of whether they are based in NHS, academic, charity or professional organizations.4–6 The FOLIO team anticipated synergies as their own understanding and experience of delivering WBEL might inform the review and learning gained from the review itself might help to shape and direct future delivery of WBEL courses.

Furthermore, as many of the team were either information specialists involved in supporting systematic review activities (DP and AS) or systematic review methodologists (AB and CC), they believed that this particular project would provide a unique opportunity to develop their experience of review methods within a subject area in which they were familiar as content experts. Given that qualitative research was likely to constitute the largest proportion of evidence on the student experience, it was also likely to provide experience of developing methods appropriate to the recently emerging growth area of qualitative evidence synthesis. Indeed, participating in this project has enabled team members to explore methods relating to citation searching on Google Scholar and citation pearl growing across a wide range of databases.7

Finally this project provided the Information Resources Section at ScHARR, in conjunction with a former colleague (CC), with another opportunity to conduct a systematic review where all those participating brought skills acquired from within the information science discipline. A previous systematic review, reported in this journal,8 had raised awareness that it was desirable to continue to build up additional capacity of this nature. Doing so also gave the Information Resources intern (RW) an opportunity to follow through the entire systematic review process prior to commencement of a formal library qualification.

The emphasis of this review article will be on the application of the findings of the systematic review to inform the team's own ongoing e-learning practice and to provide generalizable lessons for health librarians involved in e-learning, either as co-developers and deliverers or as recipients. As such, it does not present the full detail of methods to be reported in both the full report (imminently to be made available via the Higher Education Academy website, http://www.heacademy.ac.uk/) and other associated peer-reviewed articles. Nevertheless, to increase confidence in the review findings it is necessary at least to report the methods to a minimum level of detail as required by the quality of reporting of meta-analyses (QUOROM) systematic review guidelines.9 Furthermore, it should be noted that, while the scope of interest, from the point of view of the commissioners, the Higher Education Academy, is in WBEL regardless of sector, this review focuses only on that subset of studies reporting courses delivered within the health sector. These studies constitute about three-quarters of all included studies and are considered those of most relevance to the readership of this journal. Additionally, findings selectively reported here focus on issues of specific relevance and interest to information professionals rather than the more comprehensive treatment associated with the full review.

Methodology

  1. Top of page
  2. Abstract
  3. Background
  4. Methodology
  5. Results
  6. Discussion
  7. Some implications for information professionals
  8. Conclusion
  9. Key Messages
  10. References
  11. References of included studies
  12. Appendix 1: eric sample search strategy

The review question

The review aims to address the question: Which WBEL techniques are most effective in enhancing the student learning experience? E-learning was defined as ‘Learning facilitated and supported through the use of information and communications technology [which] can cover a spectrum of activities from the use of technology to support learning as part of a “blended” approach (a combination of traditional and e-learning approaches), to learning that is delivered entirely online’.10

The review only considered courses delivered by UK Higher Education bodies for two reasons. Firstly, the commissioner of the research was the UK Higher Education Academy and, secondly, findings would have increased applicability for the intended audience of the review: UK Higher Education Institutions and practitioners (including librarians) involved in delivering online courses to work-based student populations. The full inclusion and exclusion criteria are given in Table 1.

Table 1.  Inclusion and exclusion criteria for review
Inclusion criteriaExclusion criteria
  • *

    Super JANET was the fibreoptic upgrade of the JANET private British government-funded computer network dedicated to education and research.

Study sample in formal employment (full- or part-time) undertaking a course that is delivered either purely or primarily by e-learning E-learning should be only/major component of the courseStudy sample including participants in formal employment and not in formal employment where discrete outcomes for the participants in formal employment are not reported separately
Courses delivered by UK Higher Education or Further Education Institution, or UK professional body (e.g. a Royal College)Study sample are sandwich students
Techniques being evaluated should be specified. ‘Techniques’ are either technologies (e.g. discussion groups and email) or pedagogic means or approaches to delivering an online course, (e.g. regular assessment, group working, use of scenarios and case studies)Courses are primarily face-to-face courses supplemented by an e-learning component [e.g. accessing lecture handouts through a virtual learning environment (VLE)]
Student experience should be recorded as an outcome (e.g. satisfaction, enjoyment, usefulness, attitude to job, what is valued, perspectives, engagement, etc.) 
Other limitations applied: English language publications only; 1992 onwards only (a date that saw increased use of the Internet by Higher Education following roll-out of Super JANET*) 

Searching

The team constructed a search strategy to identify studies relevant to the review. This combined free text and, where available, database thesaurus terms, representing e-learning (such as ‘online learning’, ‘virtual learning’) and work generally (such as ‘employment’ or ‘work’). No specific attempt was made to identify those working within the health sector. The search was made deliberately sensitive to capture all potential studies (for a sample search strategy, see Appendix 1). Two information specialists (DP and AS) then conducted searches across the following databases for published and unpublished studies: assia, Australian Education Index, British Education Index, cinahl, CSA Abstracts (Social Services, Sociology, Information Technology Case Studies), Dissertation Abstracts, Emerald, eric, ibss, Index to Theses, lisa, medline, psycinfo and Social Science Citation Index. Supplementary methods were used to identify additional citations: reference lists of all included studies and of relevant reviews were screened; citation tracking was performed on all included studies and subject experts consulted for additional citations.

Selection

All citations were downloaded to a Reference Manager database and duplicates removed (RW). Four members of the project team (CC, AB, DP and AS) screened a sample of 100 titles and abstracts using the inclusion and exclusion criteria listed above and satisfactory inter-rater reliability was achieved. Titles and abstracts of all citations were divided equally between the four reviewers (CC, AB, DP and AS), who screened them for relevance. Where a decision about inclusion could not be made, citations were checked by a second reviewer and disagreements resolved by discussion or the full paper retrieved to make a definitive judgement. Full papers of all relevant and potentially relevant citations were retrieved, divided equally among the reviewers, screened against the inclusion criteria with each decision verified by a second reviewer.

Data abstraction

The resulting list of included studies were extracted using a form developed specifically for this review and piloted on a sample of two papers by three reviewers (CC, DP and AS), and their consistency checked by the fourth reviewer (AB). Study characteristics included details of the academic course provider, the sample population, the level of the training provided and methods of data collection.

Validity assessment

Standard checklists of quality assessment criteria, for different types of study design, were used to appraise the quality of the included studies. Where appropriate, case study11 and survey or questionnaire critical appraisal checklists12 were used to assess the quality of a specific study design in more detail. If neither checklist was suitable, the quantitative and qualitative checklists from the Alberta Heritage Foundation for Medical Research13 were used. This process was undertaken to afford a holistic view of the quality of respective studies, while appreciating that the quality assessment of qualitative study designs may not always be appropriate.14 In accordance with the Cochrane Collaboration guidelines,15 studies were placed into three quality bandings, good, medium and low, reflecting an increasing likelihood of bias.

Data synthesis

Data used for the review comprised of either verbatim quotations from study participants or findings reported by study authors and clearly supported by study data. Synthesis was performed according to the method of thematic analysis described by Miles and Huberman.16 Each finding was coded with a study identifier, the population served by the course; the relative quality of the study (low, medium or good); the course type, i.e. whether or not the course was accredited, and the technique being evaluated by the finding. Students’ responses to these specific e-learning techniques, defined broadly as either technologies (e.g. discussion groups and email) or pedagogic means or approaches to delivering an online course were identified from each study and classified into themes. Themes were not predefined but were generated from the data. The aim was to produce a small list of themes that explained all of the data, if possible, in a new way. Coding was initially performed by one author (CC) through an iterative process in which themes were revisited, revised and modified as familiarity with the studies and their findings, individually and collectively, increased. Resultant themes were considered and discussed by all of the authors (CC, DP, AS and AB) and a refined and mutually agreed framework was drawn-up to describe and explain the e-learning experience of the work-based student.

Results

  1. Top of page
  2. Abstract
  3. Background
  4. Methodology
  5. Results
  6. Discussion
  7. Some implications for information professionals
  8. Conclusion
  9. Key Messages
  10. References
  11. References of included studies
  12. Appendix 1: eric sample search strategy

A total of 3476 references were retrieved from the search of electronic databases. Abstracts, where available, were read and full text obtained for 107 papers that appeared to be relevant or where closer inspection was required. Nineteen of these studies were found to be relevant and conducted in a health context. Four further health studies were identified from unstructured searching (e.g. non-systematic searches performed for other purposes) (w1, w4, w6, w22) and three additional, relevant health studies were found in the references of the included papers (w10, w11, w19). Three health studies were identified by citation searching of the included studies (w8, w14, w26). For the full results of the screening process and sources of the included studies, see the QUOROM flowchart in Fig. 1.

image

Figure 1. Quality of reporting of meta-analyses (QUOROM) flow diagram

Download figure to PowerPoint

Twenty-nine studies were included in the health subset of the review. The relatively small number of studies found by reference and citation tracking may be because of the novelty of the topic and the recent publication of many of the references. No health-related studies were identified from contact with experts (although other studies included in the full review were identified in this way). This perhaps reflects the fact that experts were targeted from beyond the health care domain to complement the strengths already present within the review team.

Characteristics of included studies

Of the 29 included studies, 12 specifically included nurses, nine included doctors [including two specific additional mentions of general practitioners (GPs)]. Librarians and health information professionals were specifically mentioned in three studies. Managers and health scientists each received two mentions, one singly and one in combination with other staff categories. Apart from these, only occupational therapists and midwives received more than one specific mention. Single mentions were received by speech therapists, physiotherapists, podiatrists, radiographers, health visitors and dieticians, as well as for ‘other professionals allied to medicine’. Four studies did not mention specific professions using instead such terms as ‘healthcare workers’, ‘professionals from health, education, pharmacy and social work backgrounds’ or ‘qualified health and social care practitioners’.

Twenty-one different UK academic course providers were represented among the 29 studies. Three providers featured in three of the published reports (University of Bristol; University of Hull and University of Sheffield). However it must be acknowledged that the latter may reflect a bias in the review team's involvement in specific published accounts. Three institutions figured twice in reports [University of Manchester (including one joint initiative); University of Southampton and University of Stirling]. The remaining institutions received only a single mention each.

The research also covered different levels and types of courses: within the health subset, the majority (n = 19) were continuing professional development (CPD) courses. Five were at Masters level and four were at Bachelors/Diploma level, one was a nurse Conversion Course and one was an Information Skills module. Brief details of all twenty-nine studies are given in Table 2.

Table 2.  Summary table of included studies
Author(s)YearSource*Course providerSampleLevelData collectionQuality
  • *

    Sources: C, citation searches; D, database searches; L, lead from included studies; U, unstructured searches (including follow-up of suggestions from experts).

  • CPD, continuing professional development; GP, general practitioner; NR, not reported; PGD, postgraduate; VLE, virtual learning environment.

Allan & Lewis (w1)2006UUniversity of HullHealth care workers (n = 16)CPD training programmeQuestionnaires, discussion boards, learning logs, VLE tracking systemGood
Anthony & Duffy (w2)2003DDe Montfort UniversityNurses (n = NR)CPD courseSurvey, discussion boards, interviewLow
Anthony (w3)2006DNRDoctors, nurses, midwives, speech and occupational therapists, physiotherapists, radiographers (n = NR)Single skills courses, BSc, PGD coursesSurveyLow
Bacigalupo et al. (w4)2003UUniversity of SheffieldHealth care professionals and managers (n = NR)MSc Health InformaticsQuestionnaires, group discussionLow
Bahn et al. (w5)2001DUniversity of HullNurses (n = NR)Conversion courseNRLow
Booth et al. (w6)2005DUniversity of SheffieldHealth librarians (n = NR)CPD courseSurveyLow
Brosnan & Robin (w7)2003DUniversity of StirlingProfessionals from health, education, pharmacy and social work backgrounds (n = 16)CPD courseTracking facilities, WebCT, telephone interviews, discussion boardsLow
Bury et al. (w8)2006CEdge Hill UniversityHealth information professionals, pre-registration nurses (n = NR)Information skillsQuestionnaire, group discussionLow
Cahill et al. (w9)2003LUniversity of BristolDoctors (n = 18)CPD courseQuestionnaire, interviews, analysis of discussion boardsMedium
Chadda (w10)2000DUniversity College LondonNHS Staff (n = 8)MSc in primary health careInterviewLow
Clarke et al. (w11)2005DUniversity of HullNHS training managers and e-learning championsCPD courseSurveyLow
Conole et al. (w12)2002LUniversity of SouthamptonDoctors (n = 20)CPD courseQuestionnaire, analysis of discussion boardsGood
Gretsy et al. (w13)2007DUniversity of PlymouthPodiatrist, occupational therapist, nurses, midwives, doctors, health visitors, GPs, dietician, other professionals allied to medicine (n = 193)CPD courseSurveyMedium
Hall et al. (w14)2004CUniversity of GreenwichHealth scientists (n = 28)CPD courseSurveyGood
Hare et al. (w15)2006DUniversity of DerbyNurses (n = 34)CPD courseQuestionnaireLow
Hurst (w16)2004DCity UniversityNurses (n = NR)CPD courseQuestionnaireLow
Innes et al. (w17)2006DUniversity of StirlingNurses (n = 25)CPD courseQuestionnaireLow
Irving et al. (w18)2007DNursing Learning, CharityNurses (n = 1564)CPD courseSurveyLow
Jenkins et al. (w19)2001LUniversity of BristolDoctors (n = 14)CPD courseQuestionnaireLow
Kinghorn (w20)2005DUniversity of NewcastleNurses (n = 29)MSc moduleWeb-based questionnaire, analysis of discussion boardsLow
Larsen & Jenkins (w21)1995DDepartment for Work and PensionsDoctors (n = 20)CPD courseInterviewsGood
Morgan et al. (w22)2006UUniversity of SouthamptonQualified health and social care practitioners (n = 25)BSc public healthQuestionnaires, focus group discussions, discussion boardsGood
Sandars et al. (w23)2007DUniversity of Leeds, University of ManchesterDoctors, GPs, public health specialists (n = 73)CPD courseSemi-structured telephone interviews, analysis of discussion boardsMedium
Stephenson & Saxton (w24)2005DUfi Learndirect in partnership with Derby University and othersManagement and administration, education and training, technical/IT, self-employed, health and social care, engineering (n = NR)UG diplomas/degrees, PGD diplomas/MastersSurvey, interviewsGood
Sutton et al. (w25)2005UUniversity of SheffieldHealth librarians who support staff in the NHS (n = NR)CPD courseSurveyLow
Thorley et al. (w26)2007CUniversity of ManchesterDoctors (n = 213)CPD courseQuestionnaireMedium
Treharne & McClelland (w27)2004DCentre for Health Leadership WalesNHS managers and leaders (n = NR)CPD courseQuestionnaires, focus groups and internal evaluationLow
Whittington et al. (w28)2004DUniversity of BristolDoctors, nurses, midwives, health scientists (n = 20)MScWeb-based questionnaireMedium
Wilkinson et al. (w29)2004DKings College LondonNurses (n = 39)Bachelor (and one diploma) level modulesQuestionnaire, group interviewsGood

Methodological quality

The majority of studies were limited methodologically. Major limitations were an absence of specific theoretical underpinnings, or of adequate details of the sample population, or clear or explicit forms of data collection and analysis. However, seven evaluations were found to be of good methodological quality (w1, w12, w14, w21, w22, w24, w29), together with five studies of medium quality (w9, w13, w23, w26, w28), being more robust than the remaining majority of published studies.

Data synthesis

The indicated method of data synthesis was thematic analysis, given that themes were to be grounded in the data and not based on a pre-existing framework. Five key themes emerged from the data and in turn generated a larger number of sub-themes reflecting the experience of work-based students engaged in online learning delivered by UK Higher Education institutions (see Fig. 2). The five broad themes are: (i) peer communication; (ii) flexibility; (iii) support; (iv) knowledge validation; and (v) course presentation and design. Details of these themes and their sub-themes, and the techniques that have been shown to enhance student experience in these areas, are described (Table 3). This synthesis offers a new model for designing online learning for this population.

image

Figure 2. Framework of student experience of work-based e-learning

Download figure to PowerPoint

Table 3.  Illustrative findings relating to themes and sub-themes
Theme(s)Sub-themeIllustrative finding (source)*
  • *

    Coding in parentheses following each finding indicates (study no./study quality/whether accredited/specific technology or approach).

  • The above findings are illustrative only. Table 3 does not attempt to represent findings from each included study. Some studies are more data-rich than others.

Presentation and designLearner controlThe biggest personal attraction of the Learning through Work (LtW) programme (12 of the 16) was the extent to which participants could personally design the space, content and pace of the programme to fit the personal challenges and responsibilities of the workplace (w24/good/accredited/PERSONALISED LEARNING)
Presentation and designApplicabilityRelevance of learning activities has been increased by relating them to a case study based on real-life problems and scenarios in the NHS. We believe that this is having a positive impact on the overall learning experience (w4/low/accredited/SCENARIOS)
Presentation and designAttractivenessThe section of the website that responders rated the most interesting was that containing the interactive case studies. A total of 42/46 (91%) said that they had looked at this part of the site. The majority, 36/42 (78%), of users rated this section ‘very interesting’ or ‘interesting’, while only 2/42 (5%), rated this section as ‘not very interesting’ (w26/medium/ non-accredited/INTERACTIVE CASE STUDIES)
Presentation and designUsabilityThe majority (12/18) of students felt that these web lectures were really only useful if they had a soundtrack explaining the slides (w28/medium/accredited/WEB LECTURES)
FlexibilityOffline workingIn addition, almost all students printed out their ... module, for ease of access and reading in the short periods of time that were available, typically travelling to and from work (w14/good/accredited/PRINTABLE RESOURCES)
Flexibility/Peer communicationAsynchronous engagementIt depends on how you learn ... if left to your own devices, your work could quite easily slip ... having [scheduled group discussions] every week focused your mind on where you should be. There seemed in some of these groups to be a practical difficulty in getting people together at the one time so that a chat room could work (w14/good/accredited/DISCUSSION BOARDS, SCHEDULED DISCUSSIONS)
Peer communicationLearner interactionThe majority (83%) felt that the multidisciplinary mix of students was a positive aspect of the course as it aided their learning and they could learn from their peers and see various other points of view (w28/medium/accredited/DISCUSSION BOARDS)
Peer communication/ SupportPeer supportAll students either agreed or were neutral to the suggestion that meeting other students at the start of the course encouraged subsequent electronic communication (w28/medium/accredited/FACE-TO-FACE INDUCTION)
SupportModerated learningThe. group had high praise for their tutor. They were especially complementary about her speed of response, seemingly at all times of the day or night, and they were grateful of the supportive nature of her response (w14/good/accredited/EMAIL)
Support/Knowledge validationFormal supportMy emails were responded to within 24 h. [The lecturer] answered my questions and provided me with the information I wanted.... Students enjoyed being able to contact the lecturer via email. This communication was viewed as prompt and helpful (w29/good/accredited/EMAIL)
Knowledge validationAssessmentParticipants also reported that they found the multiple-choice questions (MCQs) useful, as a way of consolidating and reflecting on what they had covered in the module: I find the MCQs helpful. When you come up with a wrong answer you feel forced to go back and review the text—good for me (w12/good/non-accredited/DISCUSSION BOARDS)

Discussion

  1. Top of page
  2. Abstract
  3. Background
  4. Methodology
  5. Results
  6. Discussion
  7. Some implications for information professionals
  8. Conclusion
  9. Key Messages
  10. References
  11. References of included studies
  12. Appendix 1: eric sample search strategy

Before considering how findings of this systematic review apply to an existing e-learning programme, it is necessary briefly to characterize the FOLIO Programme. The FOLIO Programme is described more fully elsewhere4–6 but constitutes a continuing professional development programme of almost 30 separate online courses (http://cpdfolio.pbwiki.com/). Courses are commissioned by the National Library for Health, delivered free at the point of delivery and accessible to any information professional whose role involves supporting staff working within the NHS. These wide eligibility criteria allow participation across a wide range of settings, including from those working in academic libraries, health charities and professional associations, as well as those contracted to the NHS. FOLIO courses are delivered electronically and are typical of WBEL in being designed to enable participants to undertake training in their own workplace whilst learning alongside their day-to-day work.

The FOLIO Programme is primarily delivered via low technology methods consisting of 30 or more email messages, administered through a JISCmail electronic discussion list. The course team constructs a rich and varied experience for participants using such components as icebreaker exercises, group and buddy interactions, case studies, briefings, guided reading, interactive PowerPoint presentations, quizzes and voting, competitions, guest telephone lectures, self-reflection, practical exercises and reflective course summaries. A typical FOLIO course involves 40–50 participants supported by a small course team and a guest tutor. FOLIO courses are not formally accredited although the team is seeking to extend Chartered Institute of Library and Information Professionals (CILIP) recognition for their face-to-face courses to their e-learning counterparts. Participants’ achievements are recognized by a certificate awarded upon successful completion and submission of a portfolio at the conclusion of each course. Portfolios are graded against specific explicit criteria and written feedback provided.

Applying the framework to an existing e-learning programme

How can the findings of the systematic review be applied to an existing e-learning programme? The following discussion considers the application of findings from the systematic review framework to practical issues of course design and delivery exemplified by the FOLIO Programme. Using the sub-themes identified from the systematic review and examining each in turn reveals the following:

Learner control.  Flexibility is a mantra of e-learning, especially when one adds workplace-focused considerations to the mix. What the systematic review revealed was that student demand for flexibility is manifest in a greater variety of forms even than those commonly envisaged. Such flexibility, particularly embodied in the sub-theme of ‘learner control’, is seen in demand for different entry points and learning trajectories through a course, self-regulation of the pace of working and adaptation to local and personal circumstances. For example, the FOLIO Programme has evolved from a fairly rigid 6-week course with little room for manoeuvre to a much more flexible 10-week format (8 weeks of learning and 2 weeks of ‘catch up’). A self-directed path is now offered as an alternative to the default group-supported pathway in recognition of difficulties in synchronous working. Email messages and associated tasks are now distributed more sparingly across an 8-week core with ‘catch-up days’, reading weeks and removal of task dependencies (where students rely on completion of a task by fellow participants before being able to progress to a subsequent task). Such changes have allowed participants to adapt to their personal circumstances, including annual leave, conference attendance and unanticipated sickness absence. A 2-week interval, following completion of the formal course content, provides participants with an opportunity to complete any outstanding exercises and to finalize their portfolio. A 4-week extension, available upon presentation of a brief note of mitigating circumstances, allows participants to reconfigure their individual learning path.

How can health librarians involved in delivering and supporting e-learning courses benefit from the findings of this review? Even although e-learning courses place much store on their facility to be flexible and customizable, it is clear that participant demand is increasingly requiring that course providers go even further in accommodating flexible work and study patterns. For this reason, the FOLIO team is considering plans to offer shorter 1- to 2-week sub-modules selected from a pre-designated timetable. Such sub-modules would enable participants to pick and mix, either within a broader suite of sub-modules or across courses. Award of a certificate of achievement might require completion of a defined number of weeks—a completely customizable experience. Course providers would similarly do well to consider whether individual course structural constraints are determined by non-negotiable deadlines (such as assessment timescales) or whether more flexibility can be factored in to enhance the student experience.

Applicability.  Another finding from the review is the importance, especially in a work-based learning context, of providing realistic contextualization for learning. This can be achieved through a variety of mechanisms, including identification of a work-based mentor, use of guest speakers, production of problem-based scenarios and use of real case studies. Bringing together the theoretical and experiential components of learning provides an opportunity to make important connections and adds value to the course by emphasizing the applicability of what has been learnt. The FOLIO team utilizes several such methods to good success. Realistic scenarios can be constructed to include suitably anonymized elements drawn from real-life situations. Guest speakers or tutors can be drawn from a practitioner environment and asked to talk from personal experience. For example, in the FOLIO Understanding the Business of Clinical Care course, participants were encouraged to identify a local clinician and to explore their information needs. Similarly, for the FOLIO Breaking Out of the Box course, participants identified a potential organizational role for which they might equip themselves and were asked to interview the current occupant of that post.

Findings from the review should stimulate librarians involved in developing and delivering workplace-based e-learning courses to reflect upon the value of increased involvement of work-based colleagues as peer support or in a more formal mentor role. Although such a mechanism places identification of a mentor as an added pre-requisite in designing a course, potential benefits, for both participant and mentor, in terms of improved contextualization and more ready applicability of learning, are apparent.

Attractiveness.  Attractiveness of materials and the hosting Virtual Learning Environment, while not mission critical for the student experience, consistently figured in the review as an important supporting feature. In particular, students resented superficial attempts to simply deliver textual materials via a virtual classroom. When, as with the FOLIO team, there is dependence on low-technology methods of delivery, appearance cannot be a paramount consideration. Nevertheless, our experience affirms that any effort to make materials more attractive by health librarians developing or delivering e-learning is certainly welcomed by participants. Attempts to make the course less dependent on textual formats have led to a more interesting and varied choice of formats for the same types of content. For example, framing briefing content within a PowerPoint presentation, drawing on televisual themes and formats (e.g. a chat show, The Apprentice, etcetera), can make essentially the same content more visually stimulating. Librarians delivering e-learning will find it helpful to consider a similar range of approaches, including such alternatives as interviews or podcast lectures, to provide an audio alternative to written materials.

Attractiveness of materials and hosting environment remains a challenge for all e-learning courses. For the FOLIO Programme migration from an inflexible content management system to hosting within a more flexible wiki format has undoubtedly helped. Indeed, although they remain basic, course wikis are continually subject to ongoing development by the wiki providers.

Usability.  Unlike full-time learners, work-based e-learners are itinerant users of learning technologies. A considerable source of frustration, which manifests itself in the systematic review findings, is where operation of technology or lack of clarity of instructions acts as an impediment to effective use of a hard-won window of learning opportunity. Even low-tech technologies with a low level of difficulty can still prove challenging for the occasional or novice user. Adoption of wiki technology by the FOLIO Programme has led some participants to require additional assistance in how to login and how to post messages. A mandatory requirement to login and complete an evaluation form is found similarly challenging by a handful of participants. While it might be argued that mastering such skills is in itself a useful accomplishment, and could prove a valuable competency from an e-learning course, an e-learning provider should seek to remove any impediments to smooth completion of the course. Those experiencing technical difficulties should not be made to feel uncomfortable. Alternative formats may be provided or the moderator may step in to facilitate completion of a technical requirement.

Above all, written instructions should be clear and well signposted. An early innovation for the FOLIO team was to summarize lengthy email messages that require action with a ‘WHAT YOU NEED TO DO’ itemization of what is required. Similarly, ‘signposting’ on a portfolio template where an exercise needs to be completed and inserted can help the participant to stay familiar with course deadlines and deliverables, especially when only accessing materials on a weekly basis.

Usability remains an important concern and the need to provide clear instructions is accentuated for those for whom English is not a first language. For example, the FOLIO team will be delivering a blended learning FOLIAGE course on clinical question answering to Dutch librarians in Spring 2009. Clearly, it is essential to take on board the systematic review findings on usability by identifying critical readers to review task instructions before release of materials.

Offline working.   One paradox identified by the systematic review is that the increasing availability of electronic network-based formats of course materials does not preclude a demand for more portable formats such as CD-ROMS and paper. Clearly, much workplace-based learning is opportunistic and this is facilitated by ready availability of course materials. This includes enabling users to have equal access from both home- and work-based learning contexts. For example, participants may request enrolment under both private and work-based email addresses to facilitate evening working or to mitigate the effects of part-time working or home-based annual leave. Even where a course is, in theory, completely electronic, participants print materials as hard copy and store them in a course folder. This was noticeable for the UK version of the blended learning FOLIAGE course where participants brought their own course folders to face-to-face training events.

Increasingly, course providers will have a wider variety of potential formats for offline learning. Possibilities might include, for example, provision of a course USB stick containing all course materials or of MP3 lecture files for use on portable MP3 players. In recognition of findings from the systematic review, course providers should monitor carefully ongoing developments in portable educational technologies, including paper!

Asynchronous engagement.  Closely allied to the ‘any place’ imperative identified for offline working, the systematic review recognized that WBEL, in particular, places a requirement for asynchronous ‘any time’ learning. It is clearly not feasible to require work-based participants to have to engage in a large number of synchronous interactions. Indeed, for many, such a requirement is contrary to the very features that made e-learning attractive in the first place. For example, the FOLIO team discovered that even guest lectures repeated once in the morning and once in the afternoon were not suitably flexible for some participants and audio-recordings had to be provided as a matter of routine. Although, in theory, drop-in chat or telephone sessions may provide additional access and support for learner tasks, uptake of these by work-based learners may be poor when compared with the facility to ask questions by email at a time convenient to the learner.

As with the previous sub-theme, course providers need to balance awareness of the future potential of new media with a detailed knowledge of the circumstances of course participants and what they require. This emphasizes the importance for all course providers of having a clear picture of where their course wants to be in the future. Above all, the allure of new technologies should not be at the expense of pragmatic real-life learning. It is much more important to get the overall learning dynamic right and then to look at how best to translate this into delivery within an appropriate e-learning format.

Learner interaction.  The systematic review confirmed a tension between the desirability of experiencing WBEL as a shared learning experience with colleagues and the pressure to deliver a portfolio of tasks by a prescribed deadline. Indeed, the team has had to modify its own expectations from the default that group-supported learning is an essential ingredient of work-based learning to acknowledgement that, at least for some, provided that a commensurate degree of experience and reflection can still be achieved, self-directed learning is a preferential alternative. Ultimately, the decision should be based on learner preferences and the challenge faced by those developing e-learning is to design satisfactorily equivalent tasks for those who want to learn with others and those who learn best alone.

This is not to say that self-directed learners need be entirely separated from group learning approaches. For example, FOLIO courses use wikis (collaborative websites for group discussion and debate where participants can add or modify their own content), sometimes placing a formal requirement to post for or against a particular issue. Even for those who consent to the group-supported route—typically 80–90% of participants—the FOLIO team has had to minimize the number of tasks of a ‘me too’ format (thus requiring each member of a buddy group to contribute a unique perspective or task to a group exercise) and to reduce the number of task dependencies.

Peer support.   Closely related to the need for educational interaction is the imperative, identified from the systematic review, to provide mechanisms for peer support as an adjunct to formal support from the course team. Many studies from the systematic review suggest that mechanisms for peer support are an essential component of an e-learning course environment and yet there is little evidence of widespread use of such provisions by students. At this stage it is not clear whether it is simply sufficient to create some feature for peer support (e.g. a student-only coffee lounge) and to signal its existence or whether the course team has some formal responsibility for encouraging its utilization and effectiveness. A related issue is whether peer support should be conceived as a parallel but separate process from formal support (as with the students-only example) or whether formal and peer support should be interwoven (as in instances where staff and students meet in a shared course café).

The FOLIO team has recently faced requests to set up student-only discussion facilities. In harmony with findings from the literature, uptake has been very modest. Clearly, however, some degree of socialization is optimal. Experience from the blended learning FOLIAGE course seems to confirm that peer support increases once participants have met face to face. Librarians involved in supporting or delivering future e-learning programmes may find it useful to explore the facility of social networking sites in providing for aspects of socialization.

Moderated learning.  While e-learning theory has much to say about the optimal role of the facilitator, findings from the systematic review emphasize that this is a difficult balancing act to achieve. Students want to feel that the facilitator is available and interested in the activities that they have been asked to achieve. At the same time, evidence suggests that the participation (if not presence) of a facilitator can be inhibiting to student participation.17 This equates to silence experienced in a face-to-face setting when a roving facilitator joins a particular small group. While it is helpful from an administrative viewpoint to be able to monitor the progress of each group, this is at the expense of deference by participants to the facilitator in their midst. FOLIO buddy groups no longer have a course facilitator, having become entirely self-facilitated. This provides useful experience for the volunteer peer facilitator as well as ensuring a less-inhibited context for communication. Where checkpoints with a course team are needed, these can be manufactured by requiring that volunteer facilitators post an update to the course tutor, to a wiki or discussion group.

Clearly, advanced facilitation skills must be a priority for librarians wishing to develop their own skills in delivering e-learning. Following feedback in response to the review findings, the FOLIO team has targeted a need to advance from generally held skills in summarizing to the more sophisticated e-facilitation technique of ‘weaving’ (i.e. weaving involves joining different arguments or highlighting an interesting point in order to take the discussion forward).17

Formal support.  Formal support is identified by the systematic review as a key factor in the student experience. Several positive comments related to promptness of response, availability of support (especially outside normal working hours) and clarifying of instructions. The FOLIO team finds it helpful to have a formal query answering protocol, target response times and templates for additional clarification and instructions in response to frequently asked questions. An important mechanism for this has been the shared email (folio@sheffield.ac.uk) so that all four teaching staff (plus administrative support) receive all email queries and, equally importantly, are aware of whether a query has been answered. Those providing e-learning courses will find it beneficial to employ a similar approach, thereby enabling knowledge sharing within the team in terms of standardized and consistent responses and collective knowledge and precedent established over time.

Course providers will also find it helpful to consider issues around access to formal support—it is rarely feasible to provide a true 24/7 learning environment. The FOLIO team faces new challenges for formal support with the recent commencement of e-learning courses (FOLIOz) delivered to members of the Australian Library and Information Association (ALIA). These courses follow an identical format to the FOLIO Programme described above but are more generic to allow participation by library staff from all sectors and settings within Australia. Despite time differences, the team has evolved a method of delivery whereby messages are sent out in the late afternoon (UK), allowing Australian participants to respond with questions overnight. A designated FOLIO team member can consider all queries first thing in the UK morning and then respond, or consult more widely within the team, as appropriate. Those working in the evening in Australia may thus receive an immediate response while those accessing from work the following (Australian) morning receive the psychological reassurance of an answer upon opening their mailbox. Libraraians involved in delivering or supporting e-learning will find it helpful to develop query answering protocols and a library of stock responses for frequently asked questions.

Assessment.  The systematic review emphasizes that a variety of types of assessment, both formative and summative, is essential to evaluate participant achievement and, equally importantly, to keep learners interested and motivated. A portfolio-based approach, such as used by FOLIO and also by those health CPD programmes that require revalidation or accreditation, is particularly suited to multiple varied forms of assessment. Further research is needed to inform the appropriate selection of specific assessment techniques for specific types of learning objective. For example, the review team believe that one possible explanation for divergent opinions on the usefulness of quizzes relates to whether they are used to evaluate procedural and competency-based knowledge (e.g. how to complete a sickness certificate) or simply used to check understanding of definitions and broad concepts.

The FOLIO team primarily uses quizzes to check understanding of concepts. Feedback on this single learning component has proved untypically unfavourable. While some form of reflection and checking of understanding is desirable, course providers may find this better achieved through asking participants to apply broad principles to their own work-based practice. On the basis of the systematic review and their own experience, the FOLIO team plans to review the content and format of quiz exercises.

Some implications for information professionals

  1. Top of page
  2. Abstract
  3. Background
  4. Methodology
  5. Results
  6. Discussion
  7. Some implications for information professionals
  8. Conclusion
  9. Key Messages
  10. References
  11. References of included studies
  12. Appendix 1: eric sample search strategy

Information professionals have an increasing role to play within WBEL, both as deliverers and developers of e-learning and as recipients. Although the findings from the systematic review are exemplified with reference to a specific e-learning programme, albeit one with which over 1000 librarians in the UK and worldwide are familiar, many of the lessons are readily generalizable to other modes and contexts of e-learning.

As deliverers and developers of e-learning, librarians should be aware of the need for increasing flexibility in format and delivery. Clearly, while an e-learning equivalent of a face-to-face information skills training course will provide advantages in terms of improved accessibility and availability, such benefits will be limited if, for example, the deliverers then impose a requirement to engage in synchronous group work. Procedural decisions such as deadlines for completion and assessment must be justified in terms of user benefit, not simply administrative convenience. The facility to contextualize learning in terms of realistic scenarios or, preferably real-life problems, should determine both the content and the context for learning exercises. Support should be prompt and easily accessible and provide participants with the opportunity to verify understanding of concepts and instructions. Artificial separations of sources for teacher, librarian and technical support should be minimized as far as possible.

As recipients of WBEL, librarians should seek to use findings from this systematic review to enhance the quality of their own learner experience. For example, if an e-learning course makes no formal provision for contextualization, a librarian might take the time to identify a work-based mentor or at least manufacture the opportunity to discuss what is learnt with a manager or colleague. Similarly, an early step in orientation within an e-learning course should be to explore ‘tolerances’ and task-dependencies raised by the course curriculum and timetable. For example, are specific milestones ‘hard’ or ‘soft’ deadlines? Can reading weeks be used to manage slippage within the course timeline, perhaps to negotiate a way around work-related deadlines and commitments?

Conclusion

  1. Top of page
  2. Abstract
  3. Background
  4. Methodology
  5. Results
  6. Discussion
  7. Some implications for information professionals
  8. Conclusion
  9. Key Messages
  10. References
  11. References of included studies
  12. Appendix 1: eric sample search strategy

The Higher Education Academy anticipates benefits for the academic community from the systematic review once the framework has been applied to a handful of case studies identified purposively from the UK higher education sector across a range of sectors and course qualifications. Interviews for these case studies have taken place and will figure prominently in the final report to be published early in 2009. This will provide empirical validation for the review findings. However, for the review team, most of whom are currently active in delivering e-learning programmes, such a research–dissemination–implementation pipeline is unnecessarily protracted. In combining researcher–practitioner perspectives and employing techniques of reflection-in-practice (derived from current experiences) and reflection-on-practice (stimulated by the review),18 the FOLIO team reaps immediate dividends from the review process. Given that systematic reviews are criticized for being divorced from the ‘real world’ of implementation and innovation, especially within library and information practice,19 it is hoped that this review provides one possible model for bridging the well-reported research–practice gap.20

Key Messages

  1. Top of page
  2. Abstract
  3. Background
  4. Methodology
  5. Results
  6. Discussion
  7. Some implications for information professionals
  8. Conclusion
  9. Key Messages
  10. References
  11. References of included studies
  12. Appendix 1: eric sample search strategy

Implications for Policy

  • • 
    The framework published here, derived from a systematic review of qualitative research, provides a useful tool for evaluation of current and future workplace-based e-learning (WBEL) courses.
  • • 
    Important considerations for librarians involved in designing and developing WBEL include learner control, applicability, formal support and innovative methods of assessment.
  • • 
    Further investigation should examine the extent to which particular features are differentially considered important by specific groups of workplace-based health e-learners.
  • • 
    Directly relating findings from a systematic review to current practice is one way to yield immediate dividends from evidence-based practice.

Implications for Practice

  • • 
    The published framework can assist health librarians involved in developing and delivering WBEL to anticipate key issues of importance to the student experience.
  • • 
    Exemplars identified from this systematic review provide a valuable library of existing practice in relation to work-based e-learning.
  • • 
    Librarians who are e-learners themselves should seek to enhance their own learner experience by positively seeking opportunities for learner control and contextualization.

References

  1. Top of page
  2. Abstract
  3. Background
  4. Methodology
  5. Results
  6. Discussion
  7. Some implications for information professionals
  8. Conclusion
  9. Key Messages
  10. References
  11. References of included studies
  12. Appendix 1: eric sample search strategy
  • 1
    Peacock, D., Walton, G. & Booth, A. The role of library and information services in supporting learning. In: Walton, G. & Booth, A. (eds). Exploiting Knowledge in Health Services. London: Facet Publishing, 2004: 14769.
  • 2
    Ally, M. & Fahy, P. Information Sessions: Using Students’ Learning Styles to Provide Support in Distance Education. Paper presented at the 18th Annual Conference on Distance Teaching and Learning, University of Madison, Wisconsin, 14–16 August 2002.
  • 3
    Sharpe, R., Benfield, G., Roberts, G. & Francis, R. The Undergraduate Experience of Blended E-Learning: A Review of UK Literature and Practice. York: Higher Education Academy, 2006. Available from: http://www.heacademy.ac.uk/resources/detail/ourwork/research/Undergraduate_Experience (accessed 2 December 2008).
  • 4
    Sutton, A., Booth, A., Ayiku, L. & O’Rourke, A. e-FOLIO: using e-learning to learn about e-learning. Health Information and Libraries Journal 2005, 22(Suppl. 2), 848.
  • 5
    Booth, A., Ayiku, L., Sutton, A. & O’Rourke, A. Fulfilling a yearning for e-learning?: fun, collaborative courses for NHS support staff. Library + Information Update. 2005, 4, 279.
  • 6
    Booth, A., Sutton, A., Wilkinson, A. & Papaioannou, D. Break out with e-learning. CILIP Library and Information Update 2007, 6, 424.
  • 7
    Papaioannou, D., Sutton, A., Carroll, C., Wong, R. & Booth, A. Searching for Systematic Reviews: a Comparison of Methods. Poster at the Health Libraries Group Meeting, Cardiff, Wales, 21–22 July 2008. Available from: http://www.cilip.org.uk/NR/rdonlyres/FE713E74-AFE7-4F89-A584-4C7F516FD7F3/0/Systematicreviewsearching.ppt (accessed 18 November 2008).
  • 8
    Beverley, C. A., Booth, A. & Bath, P. A. The role of the information specialist in the systematic review process: a health information case study. Health Information and Libraries Journal 2003, 20, 6574.
  • 9
    Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D. & Stroup, D. F. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of reporting of meta-analyses. Lancet 1999, 354, 1896900.
  • 10
    Beetham, H. Review: Developing e-Learning Models for the JISC Practitioner Communities 2004. Available from: http://www.jisc.ac.uk (accessed 2 December 2008).
  • 11
    Atkins, C. & Sampson, J. Critical Appraisal Guidelines for Single Case Study Research. Proceedings of the Xth European Conference on Information Systems (ECIS), Gdansk, Poland, 6–8 June 2002. Available from: http://is2.lse.ac.uk/asp/aspecis/20020011.pdf (accessed 18 November 2008).
  • 12
    Boynton, P. Hands-on guide to questionnaire research: selecting, designing, and developing your questionnaire. British Medical Journal 2004, 328, 1312.
  • 13
    Kmet, L. M., Lee, R. C. & Cook, L. S. Standard Quality Assessment Criteria for Evaluating Primary Research Papers from a Variety of Fields. 2004. Available from: http://www.ihe.ca/documents/hta/HTA-FR13.pdf (accessed 18 November 2008).
  • 14
    Dixon-Woods, M., Shaw, R. L., Agarwal, S. & Smith, J. A. The problem of appraising qualitative research. Quality and Safety in Health Care 2004, 13, 2235.
  • 15
    Higgins, J. P. T. & Altman, D. G. Chapter 8: Assessing risk of bias in included studies. In: Higgins, J. P. T. & Green, S. (eds). Cochrane Handbook for Systematic Reviews of Interventions, Version 5.0.1 (updated September 2008). The Cochrane Collaboration, 2008. Available from: http://www.cochrane-handbook.org (accessed 18 November 2008).
  • 16
    Miles, M. & Huberman, A. Qualitative Data Analysis: A Sourcebook of New Methods. Newbury Park, CA: Sage, 1984.
  • 17
    Salmon, G. E-Moderating: The Key to Teaching and Learning Online, 2nd edn. London: Routledge-Falmer, 2004.
  • 18
    Grant, M. J. The role of reflection in the library and information sector: a systematic review. Health Information and Libraries Journal 2007, 24, 15566.
  • 19
    Law, M. The Systematic Review: A Potential Tool for Research-Grounded Library Management. Proceedings of the Canadian Association for Information Science/ L’association Canadienne des Sciences d l’Information (CAIS/ACSI) Annual Conference, London, Ontario, 2–4 June 2005. Available at: http://www.cais-acsi.ca/proceedings/2005/law_2005.pdf (accessed 2 December 2008).
  • 20
    Booth, A. Bridging the research-practice gap? The role of evidence-based librarianship. New Review of Information and Library Research 2003, 9, 324.

References of included studies

  1. Top of page
  2. Abstract
  3. Background
  4. Methodology
  5. Results
  6. Discussion
  7. Some implications for information professionals
  8. Conclusion
  9. Key Messages
  10. References
  11. References of included studies
  12. Appendix 1: eric sample search strategy
  • w1
    Allan, B. & Lewis, D. Virtual learning communities as a vehicle for workforce development: a case study. Journal of Workplace Learning 2006, 18, 36783.
  • w2
    Anthony, D. & Duffy, K. An evaluation of a tissue viability online course. ITIN 2003, 15, 208.
  • w3
    Anthony, D. Online courses in nursing and midwifery: comparisons with allied healthcare professions. British Journal of Healthcare Computing and Information Management 2006, 20, 2830.
  • w4
    Bacigalupo, R., Bath, P. A., Booth, A., Eaglestone, B. M., Levy, P. & Procter, P. Studying health informatics from a distance: the impact of a multimedia case study. Health Informatics Journal 2003, 9, 515.
  • w5
    Bahn, D., Needham, Y. & Marsh, D. Using the Web to develop an EN conversion course. Nursing Standard 2001, 15, 3740.
  • w6
    Booth, A., Ayiku, L., Sutton, A. & O’Rourke, A. (Lead author: Ayiku) Fulfilling a yearning for e-learning?: fun, collaborative courses for NHS support staff. Library + Information Update 2005, 4, 279.
  • w7
    Brosnan, K. & Robin, C. B. Web-based continuing professional development—a learning architecture approach. Journal of Workplace Learning 2003, 15, 2433.
  • w8
    Bury, R., Martin, L. & Roberts, S. Achieving change through mutual development: supported online learning and the evolving roles of health and information professionals. Health Information and Libraries Journal 2006, 23(Suppl. 1), 2231.
  • w9
    Cahill, D. J., Cook, J. & Jenkins, J. How useful are worldwide web discussion boards and email in delivering a case study course in reproductive medicine. In: Ghaoui, C. (ed.). Usability Evaluation of Online Learning Programs. London: Information Science Publishing, 2003: 36070.
  • w10
    Chadda, D. Cyberschool days. Health Service Journal 2000, 24, 56.
  • w11
    Clarke, A., Lewis, D., Cole, I. & Ringrose, L. A strategic approach to developing e-learning capability for health care. Health Information and Libraries Journal 2005, 22(Suppl. 2), 3341.
  • w12
    Conole, G., Hall, M. & Smith, S. An evaluation of an online course for medical practitioners. Educational Technology and Society 2002, 5, 6675.
  • w13
    Gresty, K., Skirton, H., & Evenden, A. Addressing the issue of e-learning and online genetics for health professionals. Nursing and Health Sciences 2007, 9, 1422.
  • w14
    Hall, N., Harvey, P., Meerabeau, L. & Muggleston, D. An Evaluation of Online Training in the NHS Workplace. London: University of Greenwich, 2004.
  • w15
    Hare, C., Davies, C. & Shepherd, M. Safer medicine administration through the use of e-learning. Nursing Times 2006, 102, 257.
  • w16
    Hurst, J. Evaluating staff and student experiences of multidisciplinary continuous professional development via distance-learning, EDTNA/ERCA 2004, 31, 1603.
  • w17
    Innes, A., Mackay, K., & McCabe, L. Dementia Studies Online. Reflections on the opportunities and drawbacks of e-learning. Journal of Vocational Education and Training 2006, 58, 30317.
  • w18
    Irving, M. J., Irving, R. J. & Sutherland, S. Graseby MS16A and MS26 syringe drivers: reported effectiveness of an online learning programme. International Journal of Palliative Nursing 2007, 13, 5662.
  • w19
    Jenkins, J., Cook, J., Edwards, J., Draycott, T. & Cahill, D. A pilot internet training programme in reproductive medicine. British Journal of Obstetrics and Gynaecology 2001, 108, 1146.
  • w20
    Kinghorn, S. Electronic learning. Delivering multiprofessional web-based psychosocial education: the lessons learnt. International Journal of Palliative Nursing 2005, 11, 4327.
  • w21
    Larsen, T. & Jenkins, L. Evaluation of Online Learning Module about Sickness Certification for General Practitioners. Leeds: Department for Work and Pensions, 2005: 304.
  • w22
    Morgan, J., Rawlinson, M. & Weaver, M. Facilitating online reflective learning for health and social care professionals. Open Learning 2006, 21, 16776.
  • w23
    Sandars, J., Langlois, M. & Waterman, H. Online collaborative learning for healthcare continuing professional development: a cross-case analysis of three case studies. Medical Teacher 2007, 29, 1, E9E17.
  • w24
    Stephenson, J. & Saxton, J. Using the internet to gain personalized degrees from learning through work: some experience from UfI. Industry and Higher Education 2005, 19, 24958.
  • w25
    Sutton, A., Booth, A., Ayiku, L. & O’Rourke, A. e-FOLIO: using e-learning to learn about e-learning. Health Information and Libraries Journal 2005, 22(Suppl. 2), 848.
  • w26
    Thorley, K., Turner, S., Hussey, L., Zarin, N. & Agius, R. CPD for GPs using the THOR-GP website. Occupational Medicine 2007, 57, 57580.
  • w27
    Treharne, R. & McClelland, S. Findings from a venture into elearning in NHS Wales. British Journal of Healthcare Computing and Information Management 2004, 21, 335.
  • w28
    Whittington, K., Cook, J., Barratt, C. & Jenkins, J. Can the Internet widen participation in reproductive medicine education for professionals? Human Reproduction 2004, 19, 18005.
  • w29
    Wilkinson, A., Forbes, A., Bloomfield, J. & Fincham, G. C. An exploration of four web-based open and flexible learning modules in post-registration nurse education. International Journal of Nursing Studies 2004, 41, 41124.

Appendix 1: eric sample search strategy

  1. Top of page
  2. Abstract
  3. Background
  4. Methodology
  5. Results
  6. Discussion
  7. Some implications for information professionals
  8. Conclusion
  9. Key Messages
  10. References
  11. References of included studies
  12. Appendix 1: eric sample search strategy
  1. This strategy was adapted to each database in turn.

1e-learn*
2elearn*
3web based
4web-based
5virtual learn*
6Vle
7Corporate e-learning
8online learn*
9on line learn*
10on-line learn*
11online course*
12on line course*
13on-line course*
14online professional development
15on-line professional development
16on line professional development
17computer based learn*
18computer-based learn*
19technology based learn*
20technology-based learn*
21or/1–20
22de = ‘online courses’
23work place
24workplace
25‘at work’
26employ*
27workforce
28work force
29staff
30office
31de = ‘employees’
32or/22–31
3321 and 32