The Learning Curve of Resident Physicians Using Emergency Ultrasonography for Obstructive Uropathy

Abstract

Background:  Given the time, expense, and radiation exposure associated with computed tomography (CT), ultrasonography (US) is considered an alternative imaging study that could expedite patient care in patients with suspected obstructive uropathy. However, there is a paucity of literature regarding bedside US for obstructive uropathy in the emergency department (ED), and it is unknown how much experience is required for competency in such exams.

Objectives:  The objective was to assess the learning curve for the detection of obstructive uropathy of resident physicians training in ED bedside US (EUS) during a dedicated EUS elective.

Methods:  This was a prospective cohort study of residents participating in an EUS elective. Patients presenting with acute abdominal or flank pain suggestive of an obstructive uropathy were enrolled and underwent EUS prior to noncontrast CT. Physicians who had previously performed at least 10 EUS exams for obstructive uropathy recorded results on a standardized data sheet, which was subsequently compared to the results of noncontrast CT read by board-certified radiologists blinded to the results of the EUS. In addition to an unadjusted chi-square test for trend, a multivariable logistic regression analysis, adjusting for stone size and operator, was performed. Finally, generalized estimating equations were used to describe test characteristics while accounting for potential clustering between exams by operator.

Results:  Twenty-three resident physicians participated and enrolled a convenience sample of 393 patients. A total of 157 patients (40%) were diagnosed with an obstructing ureterolith, and three (1%) were diagnosed with nonobstructing ureterolithiasis. An unadjusted chi-square test for trend demonstrated a statistically significant increase in both sensitivity (χ² = 11.4, p = 0.02) and specificity (χ² = 6.4, p = 0.04) for each level of increase in number of exams. On multivariable regression analysis, when adjusting for size of stone and operator, for every five additional exams after the first 10 EUS exams, the odds ratio for a true positive for obstruction increased by 1.7 (95% confidence interval [CI] = 1.2 to 2.5, p = 0.003). After accounting for clustering of exams by operator, overall EUS sensitivity and specificity for obstructive uropathy were 82% (95% CI = 77% to 87%) and 88% (95% CI = 85% to 92%). Stratifying by number of exams, the sensitivity was 72% (95% CI = 62% to 80%) for the 11th through 20th exams, 90% (95% CI = 83% to 96%) for the 21st through 30th exams, and 95% (95% CI = 91% to 99%) for the 31st through 43rd exams. Likewise, specificity was 82% (95% CI = 75% to 89%) for the 11th through 20th exams, 90% (95% CI = 85% to 95%) for the 21st through 30th exams, and 92% (95% CI = 86% to 98%) for the 31st through 50th exams.

Conclusions:  Physicians training in EUS may be able to accurately assess for obstructive uropathy after 30 exams.

ACADEMIC EMERGENCY MEDICINE 2010; 17:1024–1027 © 2010 by the Society for Academic Emergency Medicine

Nephrolithiasis occurs in 2% to 3% of the general population, and ureterolithiasis is a common cause of acute abdominal pain in emergency department (ED) patients.^1,2 Classically, diagnosis was made by history, physical exam, and urinalysis with or without radiographic imaging,¹ but neither urinalysis nor plain film radiography can accurately diagnose ureterolithiasis.^1,2 Noncontrast computed tomography (CT) is currently recognized as the best imaging study for ureteronephrolithiasis,³ even though it involves technician time, incurs increased expense, and exposes patients to radiation.

Emergency department bedside ultrasonography (EUS) was previously described as a possible alternative to CT in the assessment of patients with suspected ureterolithiasis^1,4 with one protocol using evidence of obstruction as a surrogate for ureterolithiasis.⁴ Experienced emergency physician sonographers appear to have accuracy comparable to traditional US performed by the department of radiology for detecting obstructing ureterolithiasis,¹ but it is unknown how much experience is required to develop competency in such exams. The purpose of this study was to assess the learning curve for the detection of obstructive uropathy of resident physicians participating in a dedicated EUS elective.

Methods

Study Design

This was an institutional review board–approved prospective cohort study of resident physicians participating in a dedicated EUS elective. All patient participants provided written informed consent.

Study Setting and Population

This study was conducted at an urban, academic ED with 49,000 annual adult visits, a postgraduate year (PGY)-2 through PGY-4 emergency medicine residency and a PGY-1 through PGY-5 combined internal medicine/emergency medicine residency. The residency offers an optional, 2-week elective rotation in EUS that requires participants to complete 25 exams for each indication by the end of the rotation.

The participating physicians were resident physicians participating in an EUS elective, who had completed an introductory course on EUS during the first week of their PGY-2 year and had performed at least 10 prior EUS exams for obstructive uropathy before enrolling patients. The study physicians were blinded to all patient data until after the EUS was performed, and results were recorded on a predesigned data sheet. All EUS exams were reviewed for resident education within 72 hours by a physician who was fellowship-trained in EUS, but the results of these reviews were not used to alter the completed data sheets, as the performance of EUS for obstructive uropathy depends on the technical and interpretive skills of the operator.

We used a convenience sample of patients enrolled between December 1, 2004, and December 31, 2007. All patients presenting to the ED with acute abdominal or flank pain were eligible for participation if their treating physicians were ordering a CT for the diagnosis of ureterolithiasis and one of the participating physicians was available to perform EUS for obstructive uropathy. Patients who consented to participate underwent EUS prior to noncontrast CT for the diagnosis of obstructive uropathy. Patients were excluded if they were unable to give informed consent in English or had known nephrolithiasis or ureterolithiasis diagnosed within the past year.

Study Protocol

Participating physicians performed EUS for obstructive uropathy before CT was obtained. The results were then recorded on a predesigned data sheet. Research assistants, trained in data abstraction and blinded to the results of the EUS exams, reviewed the results of the CT scans, which were read by board-certified radiologists for subsequent comparison. The criterion standard for an obstructive uropathy was the presence of hydroureter or hydronephrosis on CT as determined by board-certified radiologists blinded to the EUS results.

EUS exams were performed with an Ultrasonix CEP (Ultasonix, Richmond, BC, Canada) using a phased array probe or Aloka SSD-1400 (Aloka, Inc., Wallingford, CT) using a curved, linear-array probe to bilaterally assess for dilation of the renal collecting system (“hydronephrosis”) or dilation of the ureter entering the pelvis (“hydroureter”). Transverse and longitudinal views were required for “normal” exams, but only a single diagnostic view was required for abnormal exams. “Hydronephrosis” was defined sonographically as the separation of the renal sinus echoes by interconnected fluid-filled areas. “Hydroureter” was defined sonographically as the fluid-filled dilation of the ureter exiting the renal pelvis. Data sheets only specified the presence or absence of these findings. No attempt was made to measure the kidneys or identify a ureteronephrolith.

Data Analysis

Data were collected in an Excel database (Microsoft Corp., Redmond, WA) and translated into a native SAS format using DBMS/Copy (Dataflux Corp., Cary, NC). Analyses were conducted using STATA version 9.2 (StataCorp, College Station, TX) and SAS version 9.1 (SAS Institute, Cary, NC).

It was predetermined to track the EUS exams per resident by experience level in increments of 10 (i.e., exams 11–20, 21–30, 31–40, etc.). Thus, the data were hierarchical such that group 1 consisted of exams 11–20, group 2 of exams 21–30, group 3 of exams 31–40, etc. Consequently, the number of residents and number of exams in each successive group was anticipated to be smaller than the preceding group. If a physician-subject had previously performed 10 EUS exams for uropathy, he or she started in group 1 and had to do 11 more exams before being classified at the group 2 experience level, while a physician-subject with 19 prior EUS exams started in group one but only needed two more exams before being classified at the group two experience level. Thus, the tracking of data by experience level was consistent across subjects, even though their experience prior to participation varied.

A chi-square test for trend was performed to assess whether there was improvement in test characteristic performance between each group, without adjusting for potential clustering by operator. Next, multivariable logistic regression analysis was performed, adjusting for stone size and operator, while also evaluating for the presence of an interaction between operator and examination level. Finally, to account for potential clustering of exams by operator, the PROC GENMOD command was used to calculate ultrasound test characteristics (e.g., sensitivity and specificity) via generalized estimating equations, using the physician as the “repeated subject,” the logit link function, and assessing the covariance matrix to describe the 95% confidence intervals (CIs).

Results

Twenty-three of 57 eligible resident physicians (40%; 11 PGY-3 and 12 PGY-4) participated in the EUS elective during the study period. A total of 499 patients had CT scans for the evaluation of obstructive uropathy when a participating physician was available, but 69 were excluded due to having the CT done before the physician could perform EUS, six refused participation, and 31 were otherwise missed. Therefore, 393 patients were enrolled for participation, had EUS prior to CT, and were included for analysis. A total of 147 patients (37%) were diagnosed with hydroureter or hydronephrosis due to ureterolithiasis, 10 (3%) were diagnosed with only hydroureter due to ureterolithiasis, and three (1%) were diagnosed with nonobstructing ureterolithiasis. The median number of exams performed per resident was 22 (interquartile range [IQR] = 16–29), and the median ureterolith size was 4 mm (IQR = 3–5 mm).

The unadjusted chi-square test for trend demonstrated an increasing trend in sensitivity (χ² = 11.4, p = 0.02) and specificity (χ² = 6.4, p = 0.04) for each level. On multivariable regression analysis, when adjusting for size of stone and operator, for every five additional exams after the first 10 EUS exams, the odds ratio for a true positive for obstruction increased by 1.7 (95% CI = 1.2 to 2.5, p = 0.003). After accounting for clustering of exams by operator, overall EUS sensitivity for obstructive uropathy was 82% (95% CI = 77% to 87%), while overall EUS specificity was 88% (95% CI = 85% to 92%). Table 1 shows sensitivity and specificity by group number and experience level. Predictive values are also shown in Table 1. To achieve 95% sensitivity for obstructive uropathy, operators needed to perform at least 30 exams, compared to a minimum of 20 exams to achieve 95% specificity for obstructive uropathy.

Table 1.   Performance Characteristics of EUS for Obstructive Uropathy Using Generalized Estimating Equation Model After Adjusting for Operator and Correlations Between Exams

Groups (n)	Obstructive Uropathy (No. of Exams)	Sensitivity	Positive Predictive Value	Specificity	Negative Predictive Value
All values are % (95% CI). EUS = ED bedside ultrasonography.
Group 1 (23)	Exams 11–20 (173)	72 (62–80)	76 (66–85)	82 (75–89)	78 (71–85)
Group 2 (17)	Exams 21–30 (146)	90 (83–96)	93 (89–97)	90 (85–95)	93 (89–97)
Group 3 (12)	Exams 31–43 (74)	95 (91–99)	86 (74–98)	92 (86–98)	97 (95–99)

Discussion

The accurate clinical diagnosis of ureterolithiasis can be difficult given the nonspecific symptoms associated with ureteroliths (e.g., nausea and vomiting), which is why radiologic confirmation is often required. Unfortunately, this may not be feasible in unstable patients or optimal in young or fertile patients, necessitating the development of reliable diagnostic alternatives.

Emergency US appears to be a valid alternative to CT as the initial imaging modality in patients with suspected ureterolithiasis. It is noninvasive, does not require any technician time or radiation exposure, and may be done quickly without removing patients from the clinical area. Our data are consistent with previous work demonstrating that a clinical algorithm relying on EUS was feasible and minimized the work-up of patients with obstructive uropathy.⁵ Furthermore, EUS offers prognostic information because high-grade obstructions and infections associated with any obstruction require urgent intervention.³ Although some clinicians might prefer to have direct visualization of the ureterolith, it seems reasonable to begin with EUS and then reconsider the risks and benefits of CT for particular, individual patients, especially because a CT-based approach can be associated with radiation exposures up to 175 mSv.⁶

Our data suggest that physicians using EUS may be able to evaluate for urinary obstruction as accurately as those reported for US by the department of radiology.^7,8 While the American College of Emergency Physicians (ACEP) recommends a 25-exam minimum training standard,⁹ the CIs and performance of our sample suggest some physicians may require more than 30. Unfortunately, it is unclear which exact factors contribute the most to the development of competency (e.g., total number of exams vs. number of abnormal exams, a dedicated elective experience vs. prolonged exposure over the course of a residency program, cumulative US experience vs. experience with one specific exam). Likewise, although the ACEP training guidelines are per indication, it is likely that competency in organ-specific exams improves in the context of a broad background of developing competence in multiple exams (e.g., EUS for trauma and obstructive uropathy overlap) and it is unclear exactly how to quantify this interplay.

Limitations

Although 69 patients (14%) were excluded due to having a CT scan done before EUS could be performed, another 31 patients were otherwise missed and may represent a selection bias wherein “difficult” patients may not have been evaluated. Specifically, the study patients were composed of a convenience sample rather than a random sample. Thus, as our logistic regression models are based on assumptions of normality and probability sampling, our results should be interpreted with this limitation in mind. There is also the possibility of disease spectrum bias. In our study sample, the prevalence of urinary obstruction was quite high (40%), which could have enhanced the sensitivity of our EUS examinations. It is unclear how EUS would perform in the general population because prior work was also done in populations with a high prevalence of obstruction.^1,9 Furthermore, development of competency may depend more on the number of abnormal exams than the absolute number of exams, but this was not assessed in our study. Thus, a future study should be performed with a random sample of patients.

Second, study physicians were not prohibited from asking patients about a prior history of ureteronephroliths and could observe the patients’ level of pain or nausea, either of which may have biased their interpretation of EUS. However, this is also the case with US done by the radiology department and is normative for EUS.

Third, we did not assess for the possible effect of experience with EUS in trauma on performance of EUS for obstructive uropathy, since ACEP training guidelines are per indication.⁹ While the focused assessment by sonography in trauma (FAST) exam does allow for imaging of the perirenal recesses, it does not focus on the renal parenchyma and is, thus, a separate exam. However, it may be that experience with EUS for trauma might lead to faster skill acquisition with EUS for obstructive uropathy. This should be assessed prospectively in the future.

Finally, only 23 of 57 eligible resident physicians participated in the EUS elective, representing an “US-interest” bias. These physicians knew that their results were going to be compared to a criterion standard, raising the potential for a Hawthorne effect. Our findings may not apply to less interested physicians who do not believe that they are being “tested” or observed, since EUS is well known to be operator dependent. Likewise, the decreased number of participants in the 31+-exam group might indicate that only those operators who found the skill easy continued to perform EUS for obstructive uropathy. Thus, a future study should be performed with a sample including all physicians using EUS for obstructive uropathy.

Conclusions

Physicians training in emergency ultrasound may be able to accurately assess for obstructive uropathy after 30 examinations.