Impact of audit of routine second-trimester cardiac images using a novel image-scoring method

Authors


Abstract

Objective

To assess the impact of using an objective scoring method to audit cardiac images obtained as part of the routine 21–23-week anomaly scan.

Methods

A prospective audit and re-audit (6 months later) were conducted on cardiac images obtained by sonographers during the routine anomaly scan. A new image-scoring method was devised based on expected features in the four-chamber and outflow tract views. For each patient, scores were awarded for documentation and quality of individual views. These were called ‘Documentation Scores’ and ‘View Scores’ and were added to give a ‘Patient Score’ which represented the quality of screening provided by the sonographer for that particular patient (maximum score, 15). In order to assess the overall performance of sonographers, an ‘Audit Score’ was calculated for each by averaging his or her Patient Scores. In addition, to assess each sonographer's performance in relation to particular aspects of the various views, each was given their own ‘Sonographer View Scores’, derived from image documentation and details of four-chamber view (magnification, valve offset and septum) and left and right outflow tract views. All images were scored by two reviewers, jointly in the primary audit and independently in the re-audit. The scores from primary and re-audit were compared to assess the impact of feedback from the primary audit.

Results

Eight sonographers participated in the study. The median Audit Score increased significantly (P < 0.01), from 10.8 (range, 9.8–12.4) in the primary audit to 12.4 (range, 10.4–13.6) in the re-audit. Scores allocated by the two reviewers in the re-audit were not significantly different (P = 0.08).

Conclusion

Objective scoring of fetal heart images is feasible and has a positive impact on the quality of cardiac images acquired at the time of the routine anomaly scan. This audit tool has the potential to be applied in every obstetric scanning unit and may improve the effectiveness of screening for congenital heart defects. Copyright © 2009 ISUOG. Published by John Wiley & Sons, Ltd.

Introduction

Congenital heart disease (CHD) is a leading cause of infant mortality, with an incidence estimated at about 4–13 per 1000 live births1–3. These anomalies are frequently missed on prenatal ultrasonography4, 5. With improvements in the quality of ultrasound equipment and training programs, detection rates have improved, but they still vary widely from center to center6–8. The training of operators performing routine screening of the fetal heart to include ‘basic’ and ‘extended basic’ cardiac views is designed to maximize the detection of heart anomalies during a second-trimester scan8–10. Following training, there is a need, however, to maintain the standards expected of operators and thus the units in which they work.

Ongoing quality assurance is in place to ensure that sonographers maintain the standards required in the measurement of fetal nuchal translucency thickness in the 11–14-week scan11–14. However, as yet, there has been no such attempt to assess the quality of cardiac images obtained at the time of screening. The aims of this study were to assess the feasibility of using an image-scoring method to audit the quality of cardiac images and to assess the impact of such objective scoring on the images obtained 6 months later.

Methods

This was a prospective study carried out in a teaching hospital. In our institution, all women are offered an ultrasound examination at 11 to 13 + 6 weeks of gestation for pregnancy dating and screening for fetal chromosomal abnormalities by nuchal translucency measurement. A routine fetal anomaly scan is then offered at 21–23 weeks of gestation, when screening for CHD is performed. At the time of the audit, this was done by obtaining a four-chamber view and, whenever possible, views of the outflow tracts. The audit cycle was completed with feedback given to every sonographer and re-audit was planned for 6 months later.

Primary audit

Participants included sonographers performing routine screening. During a pre-defined audit week, they were asked to submit a set of thermal images (STI) of the views obtained routinely for cardiac screening, for every fetal anomaly scan performed. No specifications were given regarding the number of images to be submitted or which features were to be demonstrated. In addition to the routine views using gray-scale ultrasound, some sonographers also submitted images using color Doppler. As this would not be part of ‘routine screening’, we only included the gray-scale images for the purposes of this study.

A new image-scoring method was devised to evaluate images of the standard views used for routine cardiac screening at the 21–23-week scan (Figure 1). Scores were awarded both for documentation of the cardiac views (‘Documentation Score’) and for the quality of information in each view (‘View Score’).

Figure 1.

(a) Four-chamber view showing adequate magnification (Score 2) with good demonstration of atrioventricular valve offset (Score 2) and intact ventricular septum (Score 2). (b) View of the left ventricular outflow tract clearly demonstrating the septoaortic continuity and the vessel arising from the left ventricle (Score 2). (c) Three vessels view showing all three vessels (pulmonary artery, aorta and superior vena cava), their relative sizes and position (Score 2).

The Documentation Score ranged from 0 to 4, with 1 point being given for the imaging of each of four different ‘cardiac views’, namely, situs view, four-chamber view and right (RVOT) and left (LVOT) ventricular outflow tract views (Table 1; Score = 1 for image(s) provided and Score = 0 for image not provided). The Documentation Score for a patient therefore could not be increased simply by providing multiple images of one cardiac view. Images depicting the fetal situs or fetal laterality, i.e. a cross-section of the fetal abdomen showing the position of the fetal stomach, were scored as part of the documentation as they were indicative of the sonographer's attempt to ascertain the fetal left side. The View Score ranged from 0 to 11 and comprised a more detailed analysis of the four-chamber and outflow tract views, scored for the quality of their information as described in Table 1. The three vessels view (3VV) was considered as equivalent to the RVOT view and these two views are used interchangeably for the purposes of this paper. The sum of the Documentation and View Scores for every STI produced a ‘Patient Score’, which gave an idea of the quality of screening provided by the sonographer for a particular patient (maximum Patient Score = 15).

Table 1. Criteria for score-based objective evaluation of routine second-trimester cardiac images
Score typeScore
Documentation Score
 Situs1
 Four-chamber view (one or more images)1
 Left ventricular outflow tract view1
 Right ventricular outflow tract and/or three vessels view1
View Scores
Four-chamber view
 Magnification 
  Heart image fills more than one third of display screen (only heart seen – no chest wall)3
  Heart image fills one third of the display screen (heart and entire chest seen, but nothing around the chest)2
  Heart image fills less than one third of the display screen (entire chest wall and some area around)1
  Heart and chest with a lot of surrounding area0
 Atrioventricular valve offset 
  Offset seen clearly2
  Offset demonstrated poorly1
  Offset not seen at all0
 Interventricular septum 
  Clear, well-defined septum (apex to crux)2
  Septum unclear with areas suggestive of drop-out1
  Septum defined poorly with definite drop-out0
Left ventricular outflow tract view
  Septoaortic continuity demonstrated clearly with outflow tract2
  Septoaortic continuity and outflow tract seen with difficulty1
  Poor or no demonstration of septoaortic continuity or outflow tract0
Right ventricular outflow tract and/or three vessels view
  Clear and well demonstrated2
  Unclear but just demonstrated1
  Poor or no demonstration0
Patient Score15 (maximum)

In order to assess the overall performance of each sonographer, an ‘Audit Score’ was calculated by averaging his or her Patient Scores. In addition, to assess each sonographer's performance in relation to particular aspects of the various views, six ‘Sonographer View Scores’ were derived, one each for documentation of views, four-chamber details (magnification, atrioventricular valve offset and septum), LVOT and RVOT/3VV. These Sonographer View Scores were averages of the individual View Scores obtained by a particular sonographer across all of his or her patients. The various scores are summarized in Table 2.

Table 2. Definition and significance of the scores used in the objective assessment of routine second-trimester cardiac images
ScoreDefinition
Documentation ScoreIndicates which views were submitted for audit by a particular sonographer per patient (maximum = 4, regardless of number of images for a particular view)
View ScoreIndicates anatomical details demonstrated in each particular view submitted per patient by a particular sonographer (maximum = 11)
Patient ScoreIndicates quality of screening performed by a particular sonographer on a particular patient. Derived from sum of Documentation and View Scores (maximum = 15).
Audit ScoreProvides overview of overall performance of a particular sonographer. Derived by averaging all Patient Scores obtained by a sonographer. Allows comparison between sonographers and assessment of change in each sonographer's performance after subsequent re-audit.
Sonographer View ScoreProvides in-depth assessment of performance of a particular sonographer in documenting and demonstrating each view. Derived by averaging separately Documentation Scores and View Scores for individual views obtained by a sonographer across all his/her patients. Allows detailed individual feedback on performance.

Each STI was scored by two reviewers jointly (S.S. and A.A.); the sonographers were unaware of the scoring method. Each sonographer was given detailed individual feedback on their performance, based on these scores. They were given insight into what information was expected in each view. In addition, they were encouraged to observe S.S. or J.S.C. carrying out fetal echocardiography.

Re-audit

Re-audit was carried out 6 months after the primary audit without the sonographers' knowledge, during a week chosen arbitrarily. For each sonographer, gray-scale cardiac images from five consecutive anomaly scans were retrieved from the electronic database used for archiving images in the unit and a set of digital images (SDI) was compiled on a computer. The scoring was done using the same scoring system as for the primary audit but this time by two independent observers (S.S. and K.C.) so as to assess interobserver variability. The average of the two observers' scores was used for comparison between primary and re-audit. In addition, information relating to the maternal body mass index (BMI) and the machine used (Philips ATL HDI 5000, Philips Healthcare, Best, The Netherlands; Aloka Prosound4000/SSD5500, Aloka Co Ltd, Tokyo, Japan) was collected in order to examine whether these factors influenced the scores. The latter was particularly relevant, as the sonographers did not use the same machine all the time.

The scores from primary and re-audit were compared to assess the impact of feedback from the primary audit. Audit Scores and Patient Scores were compared to provide an overview of the sonographers' overall performance. In addition, Sonographer View Scores were compared to assess the impact on specific views. Patient scores were correlated with BMI and the ultrasound machines used. Interobserver variability was assessed by comparing the View Scores awarded independently by the two observers in the re-audit.

Statistics

The Mann–Whitney U-test was used to compare Patient Scores between the primary audit and the re-audit. The Wilcoxon signed-rank test was used to compare the Audit and Sonographer View Scores between the primary and the re-audit. Pearson's correlation coefficient was used to assess the strength of the relationship between the Patient Scores and BMI. The Mann–Whitney U-test was used to assess the strength of the relationship between the Patient Scores and the ultrasound machine used. Interobserver variability was assessed by applying Friedman's tests for non-parametric data on the re-audit View Scores15, 16.

Results

Eight sonographers from the primary audit were available for re-audit and only data from these eight sonographers were included in this study. On average, nine STIs (range, 4–18) were scored per sonographer in the primary audit and five SDIs were scored in the re-audit. The median Patient Score at the first audit was 11 (range, 7–14). This showed a significant improvement at the re-audit to 12.5 (range, 6–14.5); (P < 0.01). The median Audit Score at the primary audit was 10.8 (range, 9.8–12.4). This also showed a significant improvement at the re-audit, increasing to 12.4 (range, 10.4–13.6; P < 0.01) (Figure 2).

Figure 2.

Individual sonographers' Audit Scores in primary audit (□) and re-audit 6 months later (equation image) of routine second-trimester cardiac images.

The Sonographer View Scores, which were used to provide detailed feedback, allowed analysis of the strengths and weaknesses of every individual participant, providing information about the areas that needed focusing on and, when required, training. The impact of this feedback on all views can be seen in Figure 3. More detailed analysis with a breakdown of these scores for every view is given in Figure 4.

Figure 3.

Average Sonographer View Scores in primary audit (□) and re-audit 6 months later (equation image) of routine second-trimester cardiac images. Maximum scores possible are indicated (Max). Sonographer View Scores were derived from image Documentation Scores and from details of four-chamber view (4CV) (magnification (mag), valve offset and septum) and left and right outflow tract views (LVOT and RVOT/3VV).

Figure 4.

Individual Sonographer View Scores in primary audit (□) and re-audit 6 months later (equation image) of routine second-trimester cardiac images along with corresponding P-values for comparison between primary and re-audits. (a) Documentation Scores; (b–d) details of four-chamber view; (e) left ventricular outflow tract (LVOT); (f) three vessels view. NS, not significant.

The Patient Scores from the re-audit showed an apparent negative correlation with the maternal BMI, but this was not statistically significant (r = − 0.2162; 95% CI, − 0.491 to 0.097). There was no statistically significant difference between Patient Scores obtained from the two sets of ultrasound machines (Philips ATL HDI 5000 and Aloka Prosound4000/SSD5500) (P = 0.496).

Friedman's test showed no statistically significant difference between the scores allocated by each of the two reviewers in the re-audit (P = 0.08).

Discussion

We have presented an objective image-scoring method for auditing cardiac images in a fair and constructive manner. This audit tool is the first of its kind and may help in standardizing the cardiac images obtained during screening for CHD at the time of the routine anomaly scan. Implementation of this audit led to a significant improvement in the sonographers' median Audit Score. This improvement was due to an increase in the median Patient Score, reflecting better documentation and higher quality of images during screening.

Feasibility of conducting an audit

It is evident from this audit that it is possible to assess objectively the quality of still images obtained for routine screening for CHD. It is well accepted that the fetal heart anatomy is best viewed in dynamic scanning mode and cannot always be well-represented with still images. However, storing dynamic images during routine screening is not always practical, whereas many obstetric ultrasound departments may already have facilities for printing thermal images. Therefore, examination of still images may at present be the only practical option for quality assessment. Furthermore, the process of obtaining still images of good quality means that good quality dynamic imaging is undertaken. This audit proved to be both constructive and feasible. With the feedback provided, sonographers gained a better understanding of what they were expected to demonstrate in each view as well as the standard of images required. This audit complements the guidelines issued by the International Society of Ultrasound in Obstetrics & Gynecology Education Committee. Formal auditing of effectiveness of screening for CHD involves obtaining outcomes for each pregnancy screened. The length of infant follow-up required for such audit coupled with the relatively low incidence of the disease makes this difficult in day-to-day practice. In comparison, auditing images obtained by sonographers performing screening is easier and can be an ongoing process of audit, feedback and re-audit. This could therefore be the first step in assessing the quality of screening and a tool for improving the effectiveness of this screening10.

Audit Score

This score, obtained from all images a particular sonographer obtained when screening various patients, gave an overall assessment of his or her screening standard. Given that the sonographers were not aware of the scoring method in the primary audit, it is commendable that seven of the eight sonographers achieved 65% of the maximum score (> 10/15). The wide range in the Audit Scores (9.8–12.4) amongst sonographers in the same department highlights the varied ability that may be encountered within the same unit. The improvement in their scores in the re-audit, with 100% of them scoring above 65%, along with the significant improvement in the median score, is proof of the positive impact of this audit. This was not a short-term improvement, as is often seen in studies before and after training, because 6 months had elapsed from the primary audit. Furthermore, the improvement is likely to have been a real effect, because sonographers were not aware of the time of re-audit. We believe that this improvement is attributable to the individualized feedback that was provided following the primary audit, which essentially educated each sonographer on the important, specific aspects of the scan on which they needed to concentrate.

Documentation and View Scores

At the time of the audit, the protocol in our department indicated that screening the four-chamber view of the heart is mandatory, whereas screening the outflow tracts would be desirable. Despite this, all sonographers had screened the outflow tracts and had provided images to demonstrate them, as indicated by their scores for documentation. Mandatory screening of outflow tracts has since been included in our unit's protocol.

Ascertaining fetal laterality during routine screening was also considered desirable at the time of the primary audit and most sonographers routinely looked at visceral situs and cardiac position but did not necessarily take an image of this. Incorporating this into the documentation increased the awareness of the fetal situs and the importance of determining the relative cardiac position at the time of screening. We plan to educate sonographers further, to acquire the upper mediastinal view, which not only demonstrates the three vessels, with the aortic and ductal arches, but also demonstrates their position in relation to the fetal trachea. In future audits, documenting this view and assessing the quality of the information provided will be incorporated into the scoring system.

Analyzing the Sonographer View Scores obtained by each sonographer allowed the reviewers to break down and highlight the areas which required particular attention on an individual basis. Even in the presence of good Patient Scores, it is possible that a sonographer may be underperforming regularly in a particular view. For example, although Sonographer H scored well in both audits, there was no improvement in the overall score (Audit Score). A detailed analysis of the Sonographer View Scores, however, showed an improvement in outflow tract scores but a loss in some aspects of the four-chamber view scores, both of which were discussed individually with this sonographer. If consistent low scores are identified for a specific view, this can be addressed and efforts made to overcome the difficulty the sonographer may be facing in obtaining a particular view.

It is evident from Figure 3 that scores increased for all views except for the four-chamber magnification. The reduction in this score value between the two audits must be interpreted with caution as it is just as important to assess the relative size of the heart and chest (score closer to 2) as it is to demonstrate the intracardiac features (score closer to 3). Figure 4b shows that in both audits, six of the individual sonographers' scores for magnification were ≥ 2. We believe that magnification corresponding to a score of 2 as described in the Table 1 is perhaps adequate for screening. Alternatively, two images may be used—one to assess the cardiothoracic ratio and the second to demonstrate the cardiac features in greater detail. The vast majority of sonographers showed improvement in most View Scores (Figure 4), although statistical significance was only seen for three of the six scores analyzed. This may have been due to the relatively small numbers of sonographers in this study as well as to the relatively high score values already obtained in the primary audit.

The participants were all qualified sonographers. While their level of experience varied, all were considered to be at the level of Superintendent Grade III. The two audits were 6 months apart and during this time, no specific change was made in the unit in terms of work, supervision or further training. Opportunities to observe specialist fetal echocardiography have always been available in the unit and this was no different during these 6 months. However, as part of the feedback from the audit, sonographers were encouraged to attend more of these sessions. It would be difficult to attribute the improvement in scores to anything other than the awareness created by the audit itself, as indicated by the sonographers themselves following the feedback from the primary audit. The sonographers were all capable of achieving high scores and this audit provided them with the direction and guidelines needed for a higher level of performance. We have not included the comments from individual sonographers in this manuscript as this would make an article in itself.

BMI and equipment

It is likely that BMI plays a significant role in the quality of views obtained for cardiac screening. Although there was a negative correlation between increasing BMI and quality of views in the re-audit, this was not statistically significant, which may be due to a limited power to detect this correlation in our study. When assessing the effect of equipment, there was no correlation between scores and the machines used. This may be due to the fact that equipment in our unit is regularly upgraded, meaning that differences in quality between machines are small.

Conclusion

We have demonstrated that it is possible to audit the cardiac images obtained at the routine anomaly scan in an objective manner using a simple and universally applicable scoring method. The scoring can be modified to suit local needs and protocols. For instance, in a unit that uses only the four-chamber view for cardiac screening, the audit can be carried out using the scores for the four-chamber view alone. The unit could then introduce outflow tract views and audit these at a later date. Such an approach, on a wider scale, would undoubtedly improve overall standards in routine fetal cardiac ultrasound examinations.

This objective scoring system allows individual feedback to be provided, highlighting areas that require specific attention in the views obtained for cardiac screening. It can be used to audit sonographers and to assess trainees, as those features that are expected in the standard basic and extended cardiac views can be clearly formulated. We found that application of an audit cycle, using primary audit, feedback and re-audit, led to a significant improvement in scores obtained and that this scoring and feedback improved the process of care. Assessment of this on a larger scale needs to be carried out, to determine whether it is likely to result in an improvement in patient care, i.e. in improved detection rates for CHD.

Finally, the availability of such a cardiac scoring method may make certification in cardiac scanning a possibility for the future.

Ancillary