Evaluation of an online education program for healthcare professionals on best practice management of perinatal deaths: IMPROVE eLearning

The IMPROVE (IMproving Perinatal Mortality Review and Outcomes Via Education) eLearning, developed by the Stillbirth Centre of Research Excellence in partnership with the Perinatal Society of Australia and New Zealand was launched in December 2019. Based on the successful face‐to‐face program, the eLearning aims to increase availability and accessibility of high‐quality online education to healthcare professionals providing care for families around the time of perinatal death, to improve the delivery of respectful and supportive clinical care and increase best practice investigation of perinatal deaths.


INTRODUCTION
Stillbirths and neonatal deaths (perinatal deaths) are devastating events affecting more than 4.3 million families worldwide every year and are associated with profound and long-lasting consequences for parents and healthcare providers. 1,2In Australia, downward trends have been noted in late gestation stillbirth in recent perinatal statistics. 3,4Yet, Australia continues to experience higher stillbirth rates compared to the best-performing countries, and persistent inequities between Australian jurisdictions indicate room for improvement. 3,57][8] Advancing accountability and uncovering root-cause and associated factors to deaths are key to achieving future prevention. 9,10e IMPROVE face-to-face workshop (IMproving Perinatal Mortality Review and Outcomes Via Education) 11 was developed to upskill and assist health care professionals in implementing the Perinatal Society of Australia and New Zealand (PSANZ) / Stillbirth

Centre of Research Excellence (Stillbirth CRE) Clinical Practice
Guideline for Care Around Stillbirth and Neonatal Death. 12,13It was based on evidence-based education principles to address gaps in practice on all key aspects of care following perinatal death, including respectful and supportive bereavement care, counselling and communication to autopsy consent, and investigations and perinatal mortality audit.The target audience are healthcare professionals providing care for families around the time of perinatal death.
After an initial pilot phase commencing in 2008, and subsequent updates, the IMPROVE program has been nationally available since 2010 in a face-to-face workshop format. 11Using the SCORPIO (Structured, Clinical, Objective Referenced, Problemoriented, Integrated and Organised) approach, small groups undertake participant-centred, multi-professional teaching while rotating in a total of six skills stations. 11,13SCORPIO combines a mix of interactive and didactic teaching embedded in a framework based on psychological evidence that maximises adult learning. 14e IMPROVE face-to-face workshop has been evaluated across all Australian jurisdictions and internationally. 13Across all settings, the workshop has shown significant increases in knowledge and confidence, and high degree of participant satisfaction. 13e Stillbirth CRE, in partnership with PSANZ has adapted the content of the successful face-to-face workshop as an online program, called 'IMPROVE eLearning'.The eLearning program delivers the same content in an engaging electronic platform and has been endorsed by all relevant professional organisations.The IMPROVE eLearning was introduced in December 2019 and has been freely available and promoted widely through professional networks and partnering with Departments of Health in Australia.
It is accessed via the same platform as the National Safer Baby Bundle (SBB) Stillbirth Prevention Program eLearning (https:// learn.stillbirth cre.org.au/) and promoted as a fundamental aspect of implementing the SBB. 15The SBB and the IMPROVE were designed to complement each other on the continuum of care.
The IMPROVE eLearning comprises an introductory chapter, followed by six learning chapters and one final formative assessment (Table S1).Each chapter takes approximately 20 min to complete.
This study aims to evaluate participants' reported learning outcomes (change in knowledge and confidence) and overall acceptability of the program.

Study design
This study utilised data from a survey built into the IMPROVE eLearning platform.Each participant who registered to complete the IMPROVE eLearning was required to answer questions to assess existing knowledge before undertaking the program (pre-eLearning), and after completion (post-eLearning) to receive a certificate of completion and be eligible for Continuing Professional Development points from their respective professional colleges.
A waiver of the need for ethical review was obtained from the Chair of the Mater Misericordiae Ltd Human Research Ethics Committee (EXMT/MML/69391).Participant consent was obtained upon registration to the IMPROVE eLearning.

Study setting/participants
We included participants residing in Australia who self-reported their profession as working directly with families around the time of perinatal death (eg midwives, student midwives, nurses, obstetricians, obstetric registrars, medical officers, neonatologists, pathologists, general practitioners, social workers, Aboriginal and Torres Strait Islander health practitioners and health workers), and who completed the IMPROVE eLearning program between December/2019 to November/2021.We excluded the following participants: international participants not residing in Australia, participants who did not specify a profession involved in care around the time of perinatal death (eg allied-health professionals), and participants with missing survey responses.

Outcomes
Primary outcomes were participants reporting change in knowledge and confidence pre-and post-eLearning and participants' responses regarding perceived relevance, acceptability, engagement, satisfaction, and likelihood to change aspects of their clinical practice following IMPROVE eLearning completion.

Data collection
Demographic characteristics were collected via the eLearning module's registration page and participants' registration details (excluding personal identifiers) were linked to their evaluation responses.
The survey consisted of 13 identical questions pre-and postcompletion of the eLearning program.The first seven questions asked participants to rate their perceived level of knowledge and the next six questions asked about their level of confidence to provide the care needed for families who experienced a perinatal death.
An additional pre-eLearning question inquired about completion of previous education for best practice care after perinatal death.The post-eLearning survey included additional questions designed to assess perceived relevance, acceptability, and engagement with the IMPROVE eLearning, satisfaction, and likelihood to change aspects of their clinical practice.Participants who disclosed previous completion of the face-to-face IMPROVE workshop were asked about their level of agreement with the statement: 'I gained the same knowledge from the IMPROVE eLearning program as I did from the face-to-face IMPROVE workshop'.A final non-mandatory question asked if participants felt they would benefit from further education in a face-to-face format (survey questions in Table S2).

Data analysis
Statistical analysis was performed using SPSS version 27.All variables were treated as categorical and were described using frequencies.Likert items for knowledge, confidence, relevance, acceptability, and engagement were dichotomised as 'did not agree' (strongly disagree/disagree/neutral) and 'agreed' (agree/ strongly agree).Items for satisfaction and likelihood to change practice were dichotomised as '1-3' and '4-5' on a scale where 1 = not likely at all and 5 = extremely likely.The McNemar's test was used to assess differences in perceived knowledge or confidence between pre-and posteducation surveys.Descriptive statistics (pre-and posteducation) and Pearson's χ 2 (pre-module) were used to examine any differences and perceived change in knowledge or confidence by profession.Subgroup analysis by profession was undertaken for categories with more than ten respondents.
Associations between profession and responses to relevance, acceptability, engagement, satisfaction, and intention to change practice were assessed using Fisher's exact test.A P-value <0.05 was considered statistically significant.

RESULTS
There were 1339 participants included for analyses, from a total of 1475 participants who completed the IMPROVE eLearning between December 2019 and November 2021.Of these, 135 were excluded (45 international, eight allied-health professionals and 83 did not specify a profession).Midwives were the most common profession (80.2%; n = 1074), followed by student midwives (

Knowledge and confidence pre-and post-eLearning
Responses to knowledge (n = 1339) and confidence (n = 1325) preand post-eLearning linked to each of the six learning chapters are shown in Figure 1.A total of 14 participants (ten midwives, two student midwives, one nurse and one obstetric registrar) had incomplete responses to confidence statements and were excluded from the analysis.Pre-eLearning, participants felt least knowledgeable (21.9%) and confident (21.6%) regarding perinatal mortality audit and classification pre-eLearning.An important improvement was seen post-eLearning, although this remained the topic in which participants had the lowest levels for knowledge (89.2%) and confidence (84.2%).Participants felt most knowledgeable about autopsy and placental examination both pre-and post-eLearning (there were two questions related to knowledge for this chapter; 47.3-62.7%and 94.2-94.7% of participants 'agreed' with the knowledge statements pre-and post-eLearning, respectively).Participants felt most confident for the chapter on examination of babies who die in the perinatal period both pre-and post-eLearning (72.5% pre-and 95.2% post-eLearning).There was significant difference in perceived knowledge and confidence between pre-and post-eLearning surveys for all chapters and overall (all P < 0.05, Tables S3 and S4).

Acceptability and relevance of the IMPROVE eLearning
Responses to perceived relevance, acceptability, engagement, satisfaction, and likelihood to change aspects of their clinical practice are presented in Table 1.A total of 1328 responses were analysed (ten midwives and one student midwife had incomplete responses and were excluded from analysis).The IMPROVE eLearning was considered relevant and helpful by 94.7% of participants.Its acceptability was very high, with over 93.8% of participants agreeing with each of the three acceptability statements.Overall, 90.9% of respondents found the online education engaging and 95.0% felt the eLearning encouraged reflection upon knowledge and skills.Most respondents were also likely or extremely likely to recommend the IMPROVE eLearning to work colleagues (87.3%).Regarding the likelihood to change any aspect of their clinical practice as a result of the online education, 80.7% of participants considered it likely/extremely likely.
Previous completion of the face-to-face IMPROVE workshop was reported by 49 (3.6%) participants.Among those, 26 (53.1%) strongly agreed/agreed that they gained the same knowledge from eLearning when compared to the face-to-face IMPROVE.In a non-mandatory post-eLearning question, 74.6% (n = 473) out of 634 respondents indicated they would benefit from further education in a face-to-face format.

Differences by profession
Differences in responses for level of knowledge and confidence pre-and post-eLearning between the different professional groups are shown on Figures 2 and 3, and Tables S3 and S4.
There were statistically significant differences in level of knowledge and confidence between professions in the pre-eLearning responses (P < 0.03 for confidence in perinatal mortality audit and classification, P < 0.01 for all other statements).A higher proportion of obstetricians/gynaecologists 'agreed' with each of the seven pre-eLearning knowledge (52.6-84.2%)and six confidence (44.7-76.3%)statements when compared to the other professional groups.The lowest knowledge and confidence rates pre-module were reported by general practitioners (13.3% and 6.7%), nurses (13.9% and 14.3%) and student midwives (13% and 11.5%).
There was no statistically significant difference in responses to the IMPROVE eLearning regarding overall relevance, acceptability, engagement, encouragement to reflection, likelihood to recommend the eLearning and intention to change practice posteducation between professions (Table 1).

DISCUSSION
This study shows that the IMPROVE eLearning program was able to improve self-reported knowledge and confidence in care and management of perinatal deaths.The eLearning was wellreceived by healthcare professionals across a range of professions.The high reported likelihood to change aspects of clinical practice post-eLearning is encouraging and points toward potential improvements in care and management of perinatal deaths.

IMPROVE eLearning was an engaging online education
Did not agree Results presented as number (%).'Did not agree': strongly disagree/disagree/neutral. 'Agreed': agree, strongly agree.† Total valid responses N = 675.‡ On a scale of 1-5 where 1 = not likely at all and 5 = extremely likely.§ Fisher's exact test.Similar to the results from the face-to-face IMPROVE workshop evaluation, 13 eLearning participants reported to be least performing routine perinatal death audits; however, only 60% fed back practice recommendations to staff. 17This is improved from earlier survey data in 2010 but still indicated room for improvement. 18 In Australia, higher stillbirth rates are observed for Aboriginal and Torres Strait Islander women, and women living in remote areas. 20Developing and implementing a national evidence-based, culturally safe stillbirth education program available for all healthcare professionals is underscored by the Australian National Stillbirth Action and Implementation Plan. 21The IMPROVE eLearning has successfully reached multiprofessional clinicians from all Australian states and territories.
Although numerically midwives' participation was predominant, involvement was proportional to the representation of the workforce (around 3% of midwives and 2% of obstetricians/gynaecologists registered in Australia have participated).

F I G U R E 1
Agreement (%) with knowledge and confidence statements by learning chapter pre-and post-education.'Agreed': agree/strongly agree.'Knowledge' (N = 1339): knowledge statement: 'My knowledge on each of the following topics is sufficient to be able to provide the care needed to families who have experienced a perinatal death'.'Confidence' (N = 1325): confidence statement: 'I feel confident about using the latest clinical best practice in improving perinatal mortality review and outcomes relating to…'.† Two knowledge statements for Chapter 2 in the survey, mean result is shown in this chart.Difference between pre-and post-workshop as assessed by McNemar's test, all P < 0.001.participants) and four workshops conducted internationally (136 participants).Both eLearning and face-to-face formats showed significant increases in self-reported confidence and knowledge and high overall participant satisfaction.In line with other online education programs such as the SBB eLearning, 16 the IMPROVE eLearning had high acceptability, and most participants reported they were extremely likely to change aspects of their clinical practice as a result of completing the eLearning.These results are promising given the eLearning online delivery format can increase the availability and accessibility of education.The eLearning option is particularly important when considering the time-and resource-intensive nature of the face-to-face workshop format (eg availability of educators; financial support) and restrictions to in-person education opportunities during the COVID-19 pandemic.Only a minority of healthcare professionals had previously attended the face-to-face workshop, which is likely due to constraints in attending an in-person education session.Nonetheless, most participants indicated they would benefit from further face-to-face education.Although this study was not designed to compare the efficacy of both education formats, the face-to-face format appeared to be perceived as superior.Plans are currently in place to increase the availability of the face-to-face format, with states like Queensland, Tasmania and Australian Capital Territory updating and funding more workshops.It is, therefore, important to offer continued availability and accessibility of both face-to-face and eLearning formats to upscale the reach online, while enhancing hands-on skills and addressing local needs through in-person workshops where resourcing allows.Further studies are needed to assess and compare the efficacy of both education formats and inform enhancements.
knowledgeable and confident in perinatal mortality audit and classification.Likewise, this was the area showing the greatest improvement post-education in both IMPROVE formats.A 2018 survey of Australian maternity hospitals demonstrated that the majority of responding hospitals (92%) had a committee F I G U R E 2 Percentage of agreement with the statements by profession.'My knowledge on each of the following topics is sufficient to be able to provide the care needed to families who have experienced a perinatal death': the Perinatal Society of Australia and New Zealand (PSANZ)/Stillbirth Centre of Research Excellence respectful and supportive bereavement care framework; the relevant information to provide parents to enable informed choice about perinatal autopsy; the core investigations following a stillbirth; when placental histopathology should be undertaken; how to transport placentae to the pathology department; how to examine a baby who has died using the PSANZ checklist; the recommended steps involved in a high-quality mortality audit.Difference in level of knowledge between professions in the pre-education responses assessed by Pearson's χ 2 test, all P < 0.001.

F I G U R E 3
Percentage of agreement with the statements by profession.'I feel confident about the using latest clinical best practice in improving perinatal mortality review and outcomes regarding…': providing respective and supportive care to families who have experienced a perinatal death; talking to families about the option of a perinatal autopsy for their baby; explaining to bereaved parents the alternative investigations when autopsy is declined; performing examination of the placenta and cord immediately following the birth; accurately measuring a baby's head circumference, length and weight and plot these measurements on centile charts; using the Perinatal Society of Australia and New Zealand (PSANZ) contributing factors tool.Difference in level of confidence between professions in the pre-education responses assessed by Pearson's χ 2 test; P < 0.028 for 'Using the PSANZ contributing factors tool', all other statements P < 0.001.This study has a series of limitations.There lack of data on hospital settings (eg rural/regional) and care for priority populations.Further, there was inequality in representation from different Australian states.The improvement in knowledge and confidence were collected immediately after the online education completion, and intent to change clinical practice was selfreported.Future studies could include hospital setting data and use strategies to increase engagement of multidisciplinary teams and an equal participation from all Australian states.Furthermore, long-term follow-up could assess sustained clinical improvement after the eLearning (eg increase on autopsy rates and adequate investigations linked to the IMPROVE program uptake).The IMPROVE eLearning program supported knowledge acquisition and increased self-reported confidence regarding perinatal bereavement care among healthcare professionals.It was acceptable and engaging, and participants were likely to change some aspect of their clinical practice as a result.Additionally, the IMPROVE eLearning has no geographical constraints, is not affected by restrictions to in-person education and can be undertaken at a time and in an environment of the participant's choosing.While ongoing research is needed to monitor long-term outcomes of this online education format, the IMPROVE eLearning program may present a scalable education option to improve care and management of perinatal deaths and achieve prevention targets nationally and internationally.

TABLE 1 Responses
to overall relevance, acceptability, engagement, encouragement to reflection, likelihood to recommend the eLearning and intention to change practice post-education: overall and by profession