A systematic review of the effectiveness of training in emergency obstetric care in low-resource environments

Authors


Dr L van Lonkhuijzen, Department of Obstetrics and Gynaecology, University Medical Center Groningen, PO Box 30001, 9700 RB Groningen, the Netherlands. Email luc@vanlonkhuijzen.com

Abstract

Please cite this paper as: van Lonkhuijzen L, Dijkman A, van Roosmalen J, Zeeman G, Scherpbier A. A systematic review of the effectiveness of training in emergency obstetric care in low-resource environments. BJOG 2010;117:777–787.

Background  Training of healthcare workers can play an important role in improving quality of care, and reducing maternal and perinatal mortality and morbidity.

Objectives  To assess the effectiveness of training programmes aimed at improving emergency obstetric care in low-resource environments.

Search strategy  We searched Pubmed, Embase, Popline and selected websites, and manually searched bibliographies of selected articles. Language was not an exclusion criterion.

Selection criteria  All papers describing postgraduate training programmes aimed at improving emergency obstetric care in low-resource environments were included.

Data collection and analysis  Two reviewers independently extracted the data and classified these according to the level of the measured effects (reaction of participants, improved knowledge and skills, changes in behaviour and outcomes in practice). Any disagreements were resolved by discussion with a third author until agreement was reached.

Main results  A total of 38 papers were selected. Training programmes vary considerably in length, content and design. The evaluation of effects is often hampered by inadequate study design and the use of non-validated measuring instruments. Most papers describe positive reactions, increased knowledge and skills, and improved behaviour after training. Outcome is assessed less frequently, and positive effects are not always demonstrated. Measures that can contribute to a positive effect of training programmes include hands-on practise, team approaches and follow-up on training efforts.

Author’s conclusions  Training programmes may improve quality of care, but strong evidence is lacking. Policymakers need to include evaluation and reporting of effects in project budgets for new training programmes.

Introduction

Estimates from 2005 show that annually 535 900 women die during pregnancy, childbirth or in the postpartum period, mainly in sub-Saharan Africa and Asia.1 One of the key elements in reducing maternal mortality is to ensure that during pregnancy, childbirth and the immediate postpartum period, all women have access to skilled care, delivered by trained health professionals, such as midwifes, nonphysician clinicians, doctors or nurses.2,3 However, the levels of knowledge and skills of birth attendants in low-resource countries often do not meet the required standards. Training may help to improve the quality of care.4–6

This review assesses the effects of postgraduate training programmes to improve the skills and knowledge of trained healthcare professionals who are involved in maternal and perinatal care. We categorised the results based on four levels, described by Kirkpatrick, at which the effectiveness of training can be measured.7

  • 1 How do the trainees react to the training: do they like it; do they think it is useful?
  • 2 Does the training improve knowledge, and does it improve technical skills (e.g. do people know the indications for a vacuum extraction, and do they demonstrate the technical ability to perform one)?
  • 3 Does the training also lead to the use of this knowledge and skills in practice (e.g. does training lead to an increase in the number of vacuum deliveries)?
  • 4 Does the training improve the outcome (e.g. an increase in the number of vacuum deliveries performed for fetal distress, or a reduced number of neonates born with a low Apgar score)?

We also aimed to derive lessons from the reviewed papers that may help policymakers in deciding how to effectively train health workers in low-resource environments.

Methods

We searched the Pubmed, Embase, African Journals Online and Popline databases up to 31 December 2007, using combinations of the following keywords: ‘train*’, ‘education’, ‘staff development’, ‘educational measurement’, ‘patient simulation’, ‘in-service training’; combined with ‘obstetrics’ and ‘pregnancy’. In addition, we searched selected websites and used the ‘Web of Science’, specifically looking for articles that referred to the papers included in the review. Bibliographies of included papers were also manually searched. (See Appendix S1 for the complete search strategy). This process helped to identify unpublished evaluation reports and relevant articles.

We selected papers (defined as any written document, including peer-reviewed articles, evaluation reports, theses and conference abstracts) describing courses for professional birth attendants, as defined by the World Health Organization (WHO) (Box 1), in low-resource environments. We excluded papers describing medical curricula or specialist training programmes. Training of traditional birth attendants was also outside the scope of the review. Editorials, opinions and reviews were excluded unless they included direct results of training.

Training in low-resource countries is often supported by governmental or non-governmental organisations, and is evaluated as part of the funding process. Although we tried to retrieve such evaluation reports, many are not available in the public domain, and therefore could not be included in this review.

Two authors (LvL and AD, both gynaecologists with experience in training for obstetric emergencies) screened the titles and abstracts of the papers resulting from the initial search to determine if they met the selection criteria. If there were doubts, the full paper was retrieved. The selected papers were classified by two reviewers (LvL and AD), independently, using two complementary systems. First, the studies were categorised according to Kirkpatrick’s four levels at which the effect of training can be measured (Box 2).7 Secondly, the study designs evaluating the effects of training were identified and classified using the level of evidence system commonly used in preparing evidence-based practice guidelines.8 In this system, randomised controlled trials are considered to produce stronger evidence for the relationship between exposure and effect than a case–control study, for instance (Box 3). In two cases of uncertainty in the classification, a third author specialised in the evaluation of education (AS) was consulted, and the classification was discussed until agreement was reached. Data with regard to training specifics such as length, training content and participants were extracted using a standardised data extraction form.

Results

The literature search resulted in 2789 papers. Screening of titles and abstracts yielded 62 papers that were retrieved for further examination: 27 of these met the inclusion criteria. Twenty of these were identified from the literature search, and seven were from selected websites (Figure 1). A further eight papers were identified through hand searching of bibliographies of selected papers. Three papers were identified outside the search: one was written by one of the authors, one was an evaluation report identified by coincidence in the bibliography of a report on maternal mortality in Malawi, and one was an abstract from a conference attended by one of the authors. Thus, a total of 38 papers describing a course and/or evaluating its effects were included in the systematic review (Figure 1). For one of the included peer-reviewed articles, it was uncertain whether the trainees met the definition of ‘skilled attendant’ (Box 1).

Figure 1.

 Search flow profile.

We classified the studies by the duration of the course.

  • • Short course: 1 week or less.9–15
  • • Courses of longer duration: 1–8 weeks.16–34
  • • Self-directed learning: variable timeline.34–46

These three types are described using the Kirkpatrick system. Studies with evidence Kirkpatrick levels 1–3 are summarised in Table 1.

Table 1.   Characteristics and outcome of papers with Kirkpatrick evidence levels 1–3 (only papers with evidence level 4 are not included)
Authors and yearCountriesNo. traineesApproach for assessing effectsArticle typeKirkpatrick levelDescription of effect
1234
ReactionLearningBehaviourOutcome
Evidence Level
Short courses
Managing Obstetric Emergencies and Trauma (MOET)
Johanson et al. 200213Bangladesh9Pre- and post-training test scoresPeer-reviewed articleIIIIII  Knowledge improved. Uptake of MOET principles by one teacher
RCOG-LSTM Life Saving Skills an Essential Obstetric Care and Neonatal Care
van Lonkhuijzen et al. 200814Tanzania31Pre- and post-training test scoresPeer-reviewed articleIVIII  Knowledge improved, enthusiastic participants
Essential Obstetric Care Manual for Health Care Providers in Kenya
Warren and Liambila 200415Kenya142Pre- and post-test using several measurement toolsReport IIIIIIIIIKnowledge, use of partograph, infection prevention and use of cervical dilatation all improved. Women were treated with more respect
Courses of longer duration
WHO Midwifery Training Modules
O’Heir 199718Ethiopia, Lesotho, Nepal, Fiji, Mozambique142Pre- and post-training test scoresPeer-reviewed articleIIIIII  Participants and teachers were content with the course. Students showed improved knowledge
American College of Nurse-Midwives Life Saving Skills
Sloan et al. 200521Vietnam24Quasi-experimental, comparing three groupsPeer-reviewed article  IIb More life-threatening conditions identified, but low levels of essential management
ACNM Life Saving Skills, Continuing Education and Peer Review
Jonas 200022Indonesia80Post-training test comparing three trained and untrained groupsReport III  Trained midwives had better knowledge and skills as compared with untrained midwives, but did not reach acceptable levels
McDermott et al. 200123Indonesia108Post-training test comparing three trained and untrained groupsPeer-reviewed article IIIIV Improved knowledge and skills, with best results in intensive programme, no difference in infection prevention and use of partograph
Malawi Safe Motherhood Project Life Saving Skills 
Kroeger et al. 200324Malawi79Post-training test of knowledgeReport IIIIV Mixed results on post-test knowledge and skills. Incidental observation of clinical practice was poor
Mothercare
Jonas 200025Guatemala22Post-training skills and knowledge testReport IIIIII No difference in knowledge between groups, but skills improved with training and improved teamwork
Jonas 200026Bolivia17Post-training skills and knowledge testReport IIIIII No difference in knowledge. Skills improved, but still insufficient after training
Other
O’Rourke 199528GuatemalaPre- and post-training patient outcomePeer-reviewed article   IIIIncreased patient satisfaction
Orero et al. 200329KenyaPre- and post-training patient outcomeAbstract  IIIIIIIncreased referral rate and institutional deliveries. Increased use of infection prevention
Naziri et al. 200331Afghanistan23Post-training testAbstract III  All participants passed a knowledge and skills test
Carlough, McCall 200533Nepal104Post-training skills test comparing with and without trainingPeer-reviewed article III  Trained Maternal Child Health workers scored better on skills tests
Osei et al. 200534Ghana60Quasi-experimental, comparing two training methodsReport IIbIIb Knowledge declined, and more then 50% failed to reach 70% score
Self-directed learning
Perinatal Education Program
Woods and Theron 199537South Africa114Uncontrolled before-and-after studyPeer-reviewed article III  Knowledge improved after training
Theron 199938South Africa93Controlled trialPeer-reviewed article IIa  Attitude towards work improved in study town; no change in control town
Theron 199939South Africa93Controlled trialPeer-reviewed article IIa  Knowledge improved in intervention group; no change in control group
Theron 200040South Africa73Controlled trialPeer-reviewed article IIa  Skills improved in study town
Theron 199941South Africa93Controlled trialPeer-reviewed article IIa  Knowledge improved when completing antenatal card and partograph
Theron 199942South Africa93Controlled trialPeer-reviewed article  IIa Use of antenatal card showed improvement on some points; no change in others. Partograph use improved in some, but deteriorated in others
le Roux et al. 199843South Africa8Semi-controlled before-and-after studyPeer-reviewed article  IIb No effect of the course was found in clinical parameters such as measuring uterine growth, pelvimetry and measuring dilatation
Other         
Gulmezoglu et al. 200744Mexico and Thailand465Cluster randomised trialPeer-reviewed article IbIb Awareness and use of the Reproductive Health Library increased. No change in target practices (social support, selective use of episiotomy, etc.)
Osei et al. 200534Ghana60Quasi-experimental, comparing two training methodsReport IIbIIb Knowledge improved, but remained low for most participants; no change in observed antenatal care practices

Short courses

The Advanced Life Support in Obstetrics, the Managing Obstetric Emergencies and Trauma, and the Royal College of Obstetricians and Gynaecologists (RCOG)/Liverpool School of Tropical Medicine (LSTM) Life Saving Skills Essential Obstetric Care and Neonatal Care courses are all based on the experience from similar life support courses, such as the Advanced Trauma Life Support and the Advanced Cardiac Life Support.9–14,47 These 2-day courses include short classes, alternating between theoretical and practical sessions using simulation, to train for emergency situations such as shoulder dystocia, severe bleeding, severe pre-eclampsia and eclampsia, breech delivery and instrumental vaginal delivery, and neonatal resuscitation.

The Essential Obstetric Care Manual for Health Care Providers in Kenya course lasts 5 days, and includes practical sessions in a district hospital combined with theoretical classes outside the hospital. Apart from covering the management of uncomplicated labour and delivery, and complications and the use of the partograph, the course also emphasises the need for an improvement in communication skills. Training efforts are supported by follow-up supervisory visits and on-the-job training.15

Effect

Most short courses invoke positive reactions from participants, and are successfully implemented in many countries.12–15 Participants generally show improved knowledge on written examination.13–15 To evaluate the Kenyan 5-day course, a baseline survey was compared with an endline survey after 3 years, and showed changes in behaviour (Kirkpatrick level 3), with increased ascertainment of cervical dilatation to establish the onset of labour, increased use of the partograph, and improved infection prevention measures. The quality of monitoring improved, with blood pressure being taken more frequently, and a larger proportion of healthcare providers reported that they checked uterine contractions and monitored temperature. More women reported having been treated with respect at the health facilities. The endline survey also showed changes in outcome (Kirkpatrick level 4): the number of women with prolonged labour declined, and fewer women were recorded to have had fits.15

Courses of longer duration

Nineteen of the 38 papers describe a total of 13 different training programmes, taking between 1 and 8 weeks.16–34 Four programmes are described in more than one paper (Table 1). Most of these programmes are broader in scope than the short courses, and aim at overall quality improvement, covering topics beyond life-saving skills, such as antenatal care, uncomplicated labour and delivery, infection prevention, team cooperation, communication, prevention of mother to child HIV transmission, and promotion of breastfeeding. Nine of the 13 programmes reported the use of clinical practice to teach skills. In Indonesia the American College of Nurse-Midwives life-saving skills course is augmented by peer review and continuing education.23 In the WHO midwifery training modules, new teaching methods such as the use of drama, learning games and group work are used.17,18

Effect

Participants’ reactions (Kirkpatrick level 1) were assessed in several studies: course content and new teaching methods used in the WHO modules were appreciated by the teachers in several countries in which the modules were tested.17,18 The practical approach to teaching skills and knowledge on obstetric emergencies in one course resulted in such good feedback that this teaching method was also introduced to pre-service midwifery training.19,24 In the ten studies in which knowledge and skills were assessed (Kirkpatrick level 2), six studies showed little or no improvement, for instance in knowledge about antenatal care, labour and delivery, the use of the partograph, and managing obstetric emergencies.24–26,32–34 Four studies demonstrated improved skills and knowledge.18,20,22,23 In Indonesia one study compared different approaches to skill training. One group of midwives who were trained intensively had the opportunity to practise and use their skills by attending at least 15 births, supported by peer review and continuing education. These midwives scored better on a skills evaluation compared with a group of midwives who had been trained with less patient exposure and without follow-up support. No differences were found in infection prevention practices and use of the partograph.23 Some papers report a change in the behaviour of participants (Kirkpatrick level 3), including the successful use of the partograph, more frequent performances of manual removal of placenta and the bimanual uterine compression technique, and an increased attention to cleanliness and aseptic techniques.19,20 Other positive behavioural changes noted were a greater awareness of deficiencies in care, as observed by the hospital management,25 improved identification of life-threatening conditions, differences in the management of life-threatening conditions,21,24 and the correct application of newly acquired skills, such as managing postpartum haemorrhage, severe anaemia, retained placenta and pre-eclampsia.27 Improved teamwork and communication were also observed.26,29 One paper reported no improvement of routine antenatal care practice following a 3-week residential course.34

Few papers reported an improvement in outcomes (Kirkpatrick level 4). In one paper, the number of appropriate referrals increased dramatically (>200%), patient satisfaction improved, and the time from admission to initiation of treatment declined. However, no change in caesarean section rate or perinatal outcome was demonstrated.28

Self-directed learning

Thirteen papers described self-directed learning programmes offering no scheduled course, but a set of learning materials that students mastered individually at times convenient for themselves.34–46 This concept was applied in multiple ways: the Perinatal Education Programme provided continuing medical education to midwives working in rural areas. It used a maternal and newborn care manual consisting of theory units and skills workshops covering common and important problems faced by rural midwives and doctors.35–43 Participants mastered the theory usually in self-selected groups, whereas skills workshops were conducted with the help of a local resource person identified by the students who can demonstrate the skill – usually a doctor.35,36 Students were supported by supervisory visits. In Mexico and Thailand, the Reproductive Health Library, an annually updated electronic publication aimed at disseminating evidence-based practices, was introduced through a series of workshops encouraging staff to access the database by themselves.44 In Tanzania, a distance-learning approach was piloted, aimed at medical officers working in rural stations. Students kept a diary to record and plan their own learning, and were supported by visits from tutors.45 In Ghana, the Self Paced Learning programme relied on self study for the theoretical part, and students travelled to a designated training site to practise skills with models and to be assessed.34

Effect

The reaction of participants (Kirkpatrick level 1) was not measured. The effect of self-directed learning programmes on knowledge and skills (Kirkpatrick level 2) was assessed in several studies. The Perinatal Education Program, and the Reproductive Health Library and Self Paced Learning programmes all resulted in improved knowledge. The Perinatal Education Program also resulted in improved practical skills on a number of antenatal and intrapartum procedures.37–41

Change in behaviour (Kirkpatrick level 3) was less evident. After the Perinatal Education Program the use of the antenatal card improved in some aspects, but not all, and no difference was found in referral rates and clinical parameters such as syphilis testing, blood grouping and pelvimetry.42,43 The general introduction of the Perinatal Education Program brought teaching and practice of nurses and doctors together in a standardised approach to perinatal care in South Africa.35 After the introduction of the Reproductive Health Library, a review of patient charts showed no change in the use of social support, use of magnesium sulphate and corticosteroids for fetal lung maturity, selective use of episiotomy, use of vacuum extraction or the use of antibiotics with caesarean sections.44

Discussion

Many of the studies included in the review used nonvalidated measuring instruments to evaluate the effects of programmes, or they only measured effects at Kirkpatrick levels 1 or 2. Most papers describe positive reactions, increased knowledge and skills, and improved behaviour after training. Kirkpatrick level 4 was assessed in a minority of the studies, and these did not uniformly report positive effects.

Nevertheless, some lessons can be derived from the combined experiences in the different countries and settings from the papers included in the review. Care should be taken when applying these lessons, as the evaluations were not rigorous.

Practising skills

The fact that all programmes include some form of skills training suggests that practising skills is considered an important aspect of training. However, only one paper compared different approaches to skill training and found a beneficial effect of increased clinical practice.23 The longer duration courses more often used clinical practice instead of simulation training, as compared with the short courses and self-directed learning programmes. This could be related to the limited time frame of the short courses and the design, which uses a quick succession of theory and practise. There is no evidence in superiority of one training method over another. The choice of method for transferring skills may therefore have depended on logistical aspects of the course design.

Team approach

In western European countries, ineffective teamwork and communication have been proven to contribute to maternal and perinatal mortality.48 There is no reason to believe that teamwork is less important in low-resource settings.49 Participants who train as a team will be able to teach and learn from each other. This aspect possibly played a role in the success of the Perinatal Education Program programme in South Africa.35 Good teamwork may contribute to mutual respect between healthcare workers, and also between midwives and traditional birth attendants, as was the case in Guatemala, where it was associated with an increased numbers of referrals.28

Follow-up

Supervisory visits possibly helped sustain the results of self-paced learning.50 In Indonesia, peer review and continuing education after skill training led to better scores on skill performance, compared with the same course without follow-up.23

Several issues were observed by the authors of different papers as contributing factors to the success of a training course: for example, the effectiveness of training is also determined by the healthcare settings in which professionals work. Are the necessary resources available, and will the women in need be able to reach the facility? Some authors argue that if this enabling environment is not available, training will only be effective as part of a wider safe motherhood programme.15,21,24 Discussing the course content with stakeholders such as government agencies, professional associations, and staff and management of teaching hospitals may contribute to the success of a course.15,18,27,45

Limitations of the studies included in the review relate to the evidence levels that are presented, and the methods used to gather that evidence. Of the 38 included papers, 15 failed to assess the outcomes of the programme in question (Evidence level IV). The five studies that evaluated the effect on outcome (Kirkpatrick level 4) did so using an observational design, or by merely assuming such outcomes without proper evaluation. Furthermore, reporting bias in favour of positive results cannot be excluded, particularly in non-peer-reviewed reports from non-governmental organisations, which usually serve to justify a programme and its associated costs.

Several recommendations can be made to improve the reporting and evaluation of courses across different Kirkpatrick levels. Although a clinician might want to focus on Kirkpatrick level 4, the other three levels are equally important to determine the success of a course.7 The reaction of participants (Kirkpatrick level 1) can usually be measured by administering a questionnaire after a course. To test a change in knowledge (Kirkpatrick level 2), a counterbalance design is required (both halves of a group are given different knowledge pre-tests, and after the training the tests are switched between the groups).51 Measuring behavioural change, in other words whether and how participants actually use their newly acquired knowledge and skills in practice (Kirkpatrick level 3), is challenging. Clinical behaviour, such as team cooperation, can best be measured using a reliable and valid assessment instrument, such as the Clinical Teamwork Scale.52 Alternatively, proxy measures such as patient reports, clinician self-reports or medical record reviews might be used, but the evidence for the use of these methods is limited.53 Ultimately, the objective of training in safe motherhood is improved outcomes (Kirkpatrick level 4) for pregnant women and newborns. Measuring the effect of training on behaviour, and on the outcome of obstetric emergencies, is hampered by the low frequency of emergencies. Changes in both behaviour and outcome are best assessed using cluster randomised trials, of which the study evaluating the effect of the Reproductive Health Library is an example.44 However, experience with this trial also teaches us that designing and implementing such a trial is time consuming and challenging.54

Reports on training programmes should ideally include a description of the course content and the teaching modalities, the cost involved, how sustainability is assured, how trainers are prepared for their task, and whether the training is part of a larger safe motherhood programme.

Conclusion

The articles examined in this review have limitations that hamper their usefulness in evaluating the effects of postgraduate educational interventions to improve obstetric care in low-resource environments. Failure of most studies to underpin the results with adequate evidence precludes valid pronouncements on the effectiveness of the courses described.

Although the introduction of the Reproductive Health Library and the Perinatal Education Program led to an improvement in knowledge and skills, no positive effects on behaviour were reported, and patient outcomes were not evaluated. It is the responsibility of organisations that initiate and fund training programmes to make evaluation an integral part of programmes, and to ensure that the results, assessed by a proper peer-reviewed process, are made available to those who stand to benefit the most from a successful programme. Large parts of the world are behind schedule in reaching the fourth and fifth Millennium Development Goals. Improving knowledge and skills through training can contribute to the attainment of these goals. In order to do so successfully, sound research is needed to provide reliable evidence to support the implementation of effective training programmes.

Box 1. Definition of a skilled birth attendant by the WHO, International Confederation of Midwives, and International Federation of Gynaecology and Obstetrics.2

A skilled attendant is an accredited health professional – such as a midwife, doctor, or nurse – who has been educated and trained to proficiency in the skills needed to manage normal (uncomplicated) pregnancies, childbirth, and the immediate postnatal period, and in the identification, management, and referral of complications in women and newborns.

Box 2. Effect of training: Kirkpatrick’s levels of training evaluation.7

Level 1 Reaction: how well trainees liked the programme, as assessed by an evaluation form at the end of the course.

Level 2 Learning: what principles, facts and techniques were learned? What attitudes were changed? Improvement is usually assessed by a written exam at the end of the course. When assessing practical skills, a skills demonstration may be used on a simulated case. Clinical decision making might be assessed by written case scenarios.

Level 3 Behaviour: what changes in job behaviour resulted from the programme? Were certain practices used?

Level 4 Outcome: what are the tangible results of the programme in terms of patient outcome, reduced cost, improved quality etc.?

Box 3. Oxford levels of evidence.8

Ia Evidence from meta-analysis of randomised controlled trials

Ib Evidence from at least one randomised controlled trial

IIa Evidence from at least one controlled study without randomisation

IIb Evidence from at least one other type of quasi-experimental study

III Evidence from nonexperimental descriptive studies, such as comparative studies, correlation studies and case–control studies

IV Evidence from expert committee reports, or opinions and/or clinical experience of respected authorities

Disclosure of interests

LvL, AD, JvR and GZ are all involved in training programmes aimed at improving obstetric care in the Netherlands and/or Tanzania.

Contribution to authorship

LvL and AD searched databases, selected papers, and extracted and analysed the data; GZ, JvR and AS gave additional advice on data extraction and analysis. LvL and AD wrote the first draft. All authors contributed to, and approved, the final version.

Details of ethics approval

Ethics approval was not needed.

Funding

No funding was received.

Acknowledgements

We thank S. van der Werf and A. C. W. Volkers for their assistance with developing search strategies and executing the searches, and M. Gorsira for her careful reading of the manuscript.

Ancillary