Evaluation of three rapid diagnostic tests for cholera: does the skill level of the technician matter?
Pavani Kalluri Ram, Department of Social and Preventive Medicine, School of Public Health and Health Professions, RM. 270 Farber Hall, Buffalo, NY 14214, USA. Tel.: +1-716-829-2975, ext 738; Fax: +1-716-829-2979; E-mail: firstname.lastname@example.org
Objective To evaluate SMARTTM, MedicosTM Dip Stick and an Institut Pasteur (IP) cholera dipstick tests for accuracy and ease of use.
Method Every 50th patient presenting with diarrhoea at ICDDR,B between 1 April 2003 and 30 November 2003 was enrolled. The rapid diagnostic tests were performed by field and laboratory technicians, and sensitivity (Se), specificity (Sp), positive (PPV) and negative (NPV) predictive values calculated.
Results We isolated Vibrio cholerae O1 from 116 (38%) of 304 patients. The Se, Sp, PPV and NPV of the SMARTTM test were 58%, 95%, 84% and 84% for field technicians, and 83%, 88%, 83% and 88% for laboratory technicians. The Se, Sp, PPV and NPV of the IP dipstick test were 93%, 67%, 63% and 94% for field technicians, and 94%, 76%, 70% and 95% for laboratory technicians. The Se, Sp, PPV and NPV of the MedicosTM test were 84%, 79%, 71% and 90% for field technicians, and 88%, 80%, 72% and 92% for laboratory technicians. A high proportion of indeterminates (30%) hampered the performance of the SMARTTM test. The IP dipstick had the highest Se, irrespective of technician skill level.
Conclusion The IP dipstick is the most appropriate rapid diagnostic assay for the detection of V. cholerae O1 in locations where the skill level of personnel may be low, such as remote areas or refugee camp settings. High cost may limit the utility of any diagnostic test in the developing world.
Objectifs Evaluer la précision et facilité d'utilisation des tests aux bandelettes SMARTTM, MedicosTM et de l'Institut Pasteur (IP) dans la détection du choléra.
Méthode Chaque 50eme patient se présentant au ICDDR,B entre le 01/04/2003 et le 30/11/2003 a été recruté. Les tests rapides de diagnostic ont été appliqués par des techniciens de laboratoire et de terrain. La sensitivité (Se), la spécificité (Sp), les valeurs prédictive positive (VPP) et négative (VPN) ont été calculées.
Résultats Nous avons isolé le Vibrio cholerae O1 de 116 (38%) des 304 patients. La Se, Sp, VPP et VPN du test SMARTTMétaient de 58%, 95%, 84% et 84% pour les techniciens de terrain, 83%, 88%, 83% et 88% pour les techniciens de laboratoire. La Se, Sp, VPP et VPN du test IP étaient de 93%, 67%, 63% et 94% pour les techniciens de terrain, 94%, 76%, 70% et 95% pour les techniciens de laboratoire. La Se, Sp, VPP et VPN du test MedicosTMétaient de 84%, 79%, 71% et 90% pour les techniciens de terrain, 88%, 80%, 72% et 92% pour les techniciens de laboratoire. Une proportion élevée de résultats indéterminés (30%) ont compromis la performance du test SMARTTM. Le test aux bandelettes IP avait la sensitivité la plus élevée quelle que soit la compétence des techniciens.
Conclusion Les bandelettes IP représentent le test rapide de diagnostic le plus approprié pour la détection du V. cholerae O1 dans les zones où la compétence du personnel peut être faible, telles que les camps de réfugiés éloignés. Ce test était également le moins coûteux.
Objetivo Evaluar la exactitud y la facilidad de uso de las pruebas SMARTTM, MedicosTM Dip Stick y la tira diagnóstica para cólera del Instituto Pasteur (IP).
Método Se reclutó uno de cada 50 pacientes que se presentaron al ICDDR,B entre 1.4.2003 y 30.11.2003. El test diagnóstico rápido fue hecho por técnicos de laboratorio y de campo, y se calculó la sensibilidad (Se), la especificidad (Sp), el valor predictivo positivo (VPP) y el valor predictivo negativo (NPV)
Resultados Aislamos Vibrio cholerae 01 en 116 (38%) de los 304 pacientes. La Se, Sp, PPV y NVP de SMARTTM test fueron respectivamente 58%, 95%, 84% y 84% para técnicos de campo y, 83%, 88%, 83% y 88% para técnicos de laboratorio. La Se, Sp, PPV y NPV de las tiras diagnósticas IP fueron respectivamente 93%, 67%, 63%, and 94% para técnicos de campo, y 94%, 76%, 70%, 95% para técnicos de laboratorio. La Se, Sp, PPV, and NPV del MedicosTM test fueron 84%, 79%, 71%, 90% respectivamente para técnicos de campo, y 88%, 80%, 72%, 92% para técnicos de laboratorio. Una alta proporción de indeterminados (30%) disminuyó el desempeño del test SMARTTM. La tira diagnóstica IP dipstick tuvo la más alta sensibilidad, sin importar el nivel de experiencia del técnico.
Conclusion Las tiras diagnósticas IP es el método diagnóstico rápido más apropiado para la detección de V. cholerae 01 en lugares donde las habilidades del personal pueden ser bajas, como en lugares remotos o campos de refugiados. También fue el test más barato.
Cholera is caused by infection with cholera toxin-producing strains of Vibrio cholerae O1 or O139. Infected persons may suffer profuse dehydrating watery diarrhoea and death, a milder diarrhoeal illness, or no symptoms. In 2003, the World Health Organization reported 111 575 cholera cases and 1894 cholera deaths (WHO 2004). These numbers are believed to be an underestimate, because of underreporting and limited resources available for laboratory confirmation and routine surveillance. In areas with poor water and sanitation infrastructure, a small cluster of cholera cases can rapidly develop into a large epidemic (Anonymous 1995). Outbreaks often occur in refugee camps or remote rural areas, or during periods of civil conflict, when inadequate financial resources and geographic remoteness limit access to laboratory facilities, thereby delaying confirmation of cholera epidemics (Anonymous 1995; CDC 1997, 2003). In these settings, sensitive diagnostic tests that can be performed by field personnel with minimal laboratory infrastructure would allow for rapid confirmation of suspected cholera outbreaks and early resource mobilization for epidemic control.
Infection with toxigenic V. cholerae O1 strains is confirmed microbiologically by culture of stools or rectal swabs, and serogrouping and toxin testing of isolates. Traditional culture techniques often take ≥48 h to complete. Several diagnostic assays have been developed to detect V. cholerae O1 in stool more rapidly. We evaluated three of these assays in Dhaka, Bangladesh and examined whether test validity was related to the skill level of the technician.
Since 1979, stool samples have been routinely collected from every 50th patient who presents to the ICDDR,B: Centre for Health and Population Research. Specimens are tested in the Centre's Clinical Laboratory for viral and bacterial enteric pathogens. For this study, we enrolled all patients with diarrhoea participating in this routine surveillance. Written consent was obtained from adult patients or from the guardians of patients ≤18 years old. Consenting patients were interviewed regarding symptoms and the use of antimicrobial agents and rehydration solution after symptom onset. Stool specimens were collected as soon as possible after admission and before any antimicrobial agents were administered at the hospital. We recorded the treating doctor's assessment of illness severity, the clinical diagnosis, and the in-hospital use of antimicrobial agents and intravenous rehydration fluids.
Specimen processing and microbiological methods
Two samples of whole stool were collected from each enrolled patient. One sample was sent to the ICDDR,B Clinical Laboratory for routine microbiological evaluation, including rotavirus antigen detection and culture for Salmonella and Shigella. A second sample was divided into two subsamples; one was sent to the ICDDR,B Research Laboratory and the second was sent to a panel of field technicians. At the Research Laboratory, culture was performed by plating stool directly and after enrichment in alkaline peptone water (APW) onto taurocholate tellurite gelatine agar (TTGA), a growth medium selective for V. cholerae. Suspicious colonies were picked for serogrouping by polyvalent and type-specific antisera. Culture results from the Research Laboratory were considered the ‘gold standard’ for comparison. Technicians at the Research Laboratory also performed each of the three rapid diagnostic assays described below. These technicians had several years of clinical microbiology experience and are considered highly skilled. They were trained by a senior microbiologist on use of the rapid assays.
Each of the three field technicians (referred to locally as ‘paramedics’) received an aliquot of stool for testing with the rapid diagnostic assays. As may be the case for healthcare providers in a refugee camp clinic, the field technicians were educated to at least college-level, but did not have previous laboratory experience and were considered less skilled at laboratory techniques than their Research Laboratory counterparts. They were trained on use of the rapid assays by senior microbiologists from the Research Laboratory. All laboratory and field technicians were blinded to the clinical diagnosis, rapid assay results obtained by their colleagues and stool culture results.
The Sensitive Membrane Antigen Rapid Test (SMARTTM, New Horizons Diagnostics, Columbia, MD, USA) is a colorimetric immunoassay designed to detect the O1 antigen of V. cholerae in whole stool samples (Hasan et al. 1994). SMARTTM was reported to have 95% sensitivity (Se) and 100% specificity (Sp) for the detection of V. cholerae O1 in a small field trial (Hasan et al. 1994). A similar assay for V. cholerae O139 was reported to be 100% sensitive and 97% specific in a field trial in Bangladesh (Qadri et al. 1995).
The Institut Pasteur (IP; Paris, France) has developed dipstick assays using monoclonal antibodies for the detection of the V. cholerae O1 and O139 lipopolysaccharides (LPS). In a laboratory-based trial using whole stools conducted in Bangladesh and Madagascar, the Se of IP dipstick for the detection of V. cholerae O1 was 94%, for V. cholerae O139 it was 99%. Respectively, the Sp was 84% and 96% (Nato et al. 2003). The IP dipstick is highly sensitive and specific for V. cholerae O1 when rectal swabs were enriched with APW in a laboratory-based evaluation (Bhuiyan et al. 2003). This assay is not yet commercially available.
The MedicosTM Cholera Dip Stick (Advanced Diagnostics Inc., South Plainfield, NJ, USA) is designed to detect V. cholerae O1 in whole stools. The manufacturer has not disclosed the specific mechanism employed for V. cholerae O1 detection. In a laboratory evaluation using saline suspension of pure cultures, the MedicosTM Dip Stick had high Se and Sp (M. Bird, CDC, personal communication). We have not found published evaluations of this assay.
The three assays are distinguishable with respect to mechanism, time required for performance and cost (Table 1).
Table 1. Comparison of three rapid diagnostic assays for the detection of Vibrio cholerae O1 in stool
|Mechanism||Monoclonal–polyclonal sandwich||Monoclonal antibody||Mechanism not disclosed by manufacturer|
|Target||Antigen A of V. cholerae O1 LPS||V. cholerae O1 LPS||Unknown|
|Time for test performance in minutes (per manufacturer)||15||10||10|
|Storage||Refrigeration for long-term storage||Room temperature||Refrigeration for long-term storage|
|Cost||$14/device||Not yet commercially available||$4/dipstick|
This study was approved by the Research and Ethical Review Committees of ICDDR,B: Centre for Health and Population Research (Dhaka, Bangladesh) and the Institutional Review Board of the Centers for Disease Control and Prevention (Atlanta, GA, USA).
Using stool culture in the ICDDR,B Research Laboratory as the gold standard for comparison, we calculated the Se, Sp, and positive (PPV) and negative predictive values (NPV) of each rapid diagnostic assay. We report separately the measures of test performance for each assay, as performed by field and laboratory technicians. Samples read as ‘indeterminate’ were excluded from the initial calculation of measures of test performance. We later recalculated measures of test performance, with all ‘indeterminate’ readings recoded as ‘negative’ and then as ‘positive’. We used sas v. 8.2 to generate 95% confidence intervals (CI) for the point estimates of the measures of test performance. We also determined whether the measures of test performance were significantly different when the assays were performed by field technicians compared with laboratory technicians.
We enrolled 304 patients between 1 April and 30 November 2003. Their median age was 6 years (range: 11 days to 80 years); 40% were female. The most commonly reported symptoms were diarrhoea (100%), vomiting (88%), dizziness (82%) and leg cramps (77%). The use of rehydration therapy and antimicrobial agents before hospital presentation was common, reported by 95% and 54% of patients respectively.
The Research Laboratory identified V. cholerae in 118 (38%) of 304 stool samples. Of these, 115 yielded V. cholerae O1 alone, two yielded V. cholerae O139 alone and one yielded both V. cholerae O1 and O139. All V. cholerae O1 were biotype El Tor; 29% were serotype Inaba and 71% were serotype Ogawa. There was no difference in antimicrobial use between patients with and without confirmed V. cholerae O1 infection. Among the 188 stools negative for V. cholerae O1, other pathogens were identified by the Clinical Laboratory in 78 (40%), of which rotavirus was the most common (86%).
Clinical diagnosis and rapid assay results
A clinical diagnosis of cholera was made by treating doctors, before culture results were available, in 130 (43%) patients. The Se and Sp of clinical diagnosis were each 79%. The PPV and NPV were 70% and 86% respectively. Overall, the IP dipstick had the highest Se for detection of V. cholerae O1, with similar results in the hands of field (93%) and laboratory (94%) technicians. The SMARTTM was found to have the lowest Se for both skill levels, particularly for field technicians (58%). The SMARTTM performed by field technicians was the most specific assay (95%). For both skill levels, the SMARTTM was superior with respect to PPV, 84% for field technicians and 83% for laboratory technicians. In contrast, clinical diagnosis and the remaining assays had similar PPV, ranging from 63% to 73%. The NPV was high for all three rapid assays, with the highest NPV detected for the IP dipstick used by laboratory technicians.
We examined the 95% CIs of the measures of test performance to determine whether there were major differences between the assays according to the skill level of the technician. Field and laboratory technician performance was similar for the IP and MedicosTM Dip Sticks, with overlapping 95% CIs and no statistically significant difference in Se, Sp, PPV and NPV. The SMARTTM assay was an exception with a Se of just 58% (95% CI: 46–71%) when performed by field technicians, compared with 83% when performed by laboratory technicians (95% CI: 75–89%; P < 0.001). The Sp of SMARTTM also varied according to the skill level of the technician (P < 0.05).
The proportion of samples read as ‘indeterminate’ was much higher for the SMARTTM than for the MedicosTM and IP dipstick assays. Field technicians recorded 30% of samples as ‘indeterminate’ with the SMARTTM (Table 2). The laboratory technicians also reported a high number of indeterminate readings for the SMARTTM, in contrast to IP dipstick and the MedicosTM Dip Stick. As noted in the Methods section, the measures of test performance were initially calculated with the exclusion of indeterminate samples. A clinician seeking to make a definitive diagnosis of cholera may choose to interpret an indeterminate reading as ‘negative’. Thus, we examined the impact of recoding indeterminates as ‘negative’ on measures of SMARTTM performance (Table 3). The Se and NPV of the SMARTTM declined considerably, particularly for the field technicians, rendering the test inadequate for cholera diagnosis. When indeterminates were considered ‘positive’, the Se was increased, but the Sp and PPV of SMARTTM declined, particularly for the field technicians. Measures of test performance for the IP and MedicosTM Dip Stick assays did not vary significantly when indeterminates were recoded as ‘positive’ or as ‘negative’.
Table 2. Measures of test performance of diagnostic methods for Vibrio cholerae O1, according to the skill level of the technician*
|Clinical diagnosis||78 (70–86)||79 (73–85)||69 (60–77)||86 (80–91)||N/A||N/A|
|SMARTTM (field)*||58 (46–71)||95 (91–98)||84 (71–94)||84 (78–89)||92 (30)||19 (5–40)|
|SMARTTM (lab)*||83 (75–90)||88 (82–93)||83 (75–90)||88 (82–93)||50 (16)||21 (15–40)|
|IP (field)||93 (87–97)||67 (60–74)||63 (55–70)||94 (89–97)||5 (2)||16 (3–58)|
|IP (lab)||94 (88–98)||76 (70–82)||70 (63–77)||95 (91–98)||0||24 (5–50)|
|MedicosTM (field)||84 (77–91)||79 (73–85)||71 (63–78)||90 (84–94)||0||23 (7–54)|
|MedicosTM (lab)||88 (81–94)||80 (73–95)||72 (64–79)||92 (86–95)||1 (0)||19 (6–45)|
Table 3. Effect of recoding ‘indeterminate’ readings as ‘negative’ on test performance
|Indeterminate readings excluded from analysis|
| SMARTTM (field)*||58||95||84||84|
| SMARTTM (lab)*||83||88||83||88|
| IP (field)||93||67||63||94|
| IP (lab)||94||76||70||95|
| MedicosTM (field)||84||79||71||90|
| MedicosTM (lab)||88||80||72||92|
|Indeterminate readings coded as ‘negative’|
| SMARTTM (field)||33||96||84||70|
| SMARTTM (lab)||78||91||83||87|
| IP (field)||92||68||63||94|
| IP (lab)||94||76||70||95|
| MedicosTM (field)||84||79||71||90|
| MedicosTM (lab)||88||79||72||92|
|Indeterminate readings coded as ‘positive’|
| SMARTTM (field)||77||75||64||84|
| SMARTTM (lab)||84||68||62||88|
| IP (field)||93||66||62||94|
| IP (lab)||94||76||70||95|
| MedicosTM (field)||84||79||71||90|
| MedicosTM (lab)||88||79||71||92|
We reviewed technician records to determine the grounds for considering a sample ‘indeterminate’ by the SMARTTM test. The manufacturer's instructions state that ‘a sample should be reported as presumed positive if there is pink to red colour on the Test Spot with no or very slight development of colour on the Negative Control Spot’. Of the 92 samples read as indeterminate by the field technicians, all were found to have colours other than pink or red in the Test Spot; colours such as purple and black were reported. None of the samples was considered indeterminate because of a pink or red appearance in the Negative Control Spot. In contrast, 16 (32%) of 50 samples reported as indeterminate by the laboratory technicians had a pink colour in the Negative Control Spot.
The utility of a diagnostic assay is enhanced if it is rapid and perceived to be easy to use. Laboratory and field technicians reported a longer time for performance of each of the three diagnostic assays evaluated than was described by the respective manufacturer/developer (Table 2). Laboratory and field technicians reported that the IP and MedicosTM Dip Sticks were easy to use and easy to interpret; however, with some samples, particularly semisolid stools, the time required for soaking the dipsticks was much longer than described in the test instructions. Both laboratory and field technicians reported that the SMARTTM device was often difficult to interpret and was frustrating to use.
We present a direct comparison of three rapid diagnostic assays for the diagnosis of V. cholerae O1 and present findings according to the skill level of the technician. Our evaluation demonstrates that field technicians performed the two dipstick assays at least as well as their laboratory counterparts. We found that the IP dipstick was the most sensitive assay for the detection of V. cholerae O1, in the hands of both field and laboratory technicians. Educated non-laboratorians often function as medical care providers in remote locations and refugee camps. In these settings, a highly sensitive assay would be useful to alert medical providers to an outbreak of cholera. Cholera case-fatality rates are typically highest at the beginning of an outbreak; earlier confirmation of cholera could dramatically expedite the public health response and prevent unnecessary deaths from occurring. Earlier recognition of cholera would also lead to more rapid implementation of appropriate interventions for outbreak control, such as the distribution of bleach for drinking water treatment and promotion of handwashing with soap.
A highly specific assay would be most valuable to rule out cholera in an individual patient. The public health implications of a very specific assay for cholera are greater in countries where the disease is rare than in countries where cholera is endemic. The SMARTTM was the most specific of the three assays. However, these results come with an important caveat: using the current manufacturer's instructions, 30% of samples tested by the field technicians and 16% of samples tested by the laboratory technicians were interpreted as ‘indeterminate’. Hence, if used in the clinical setting, a high proportion of samples tested would yield an uninterpretable result that would benefit neither clinicians nor public health authorities.
Given that rehydration therapy is recommended for all patients presenting with watery diarrhoea, irrespective of aetiology, the clinical significance of a positive cholera assay is limited. In this evaluation, conducted at a diarrhoeal diseases hospital in a setting that is hyper-endemic for cholera, clinical diagnosis alone was reasonably sensitive and specific. Healthcare providers who are less accustomed to seeing patients with cholera may have less success at differentiating cholera from other diarrhoeal diseases without assistance from laboratory testing.
A suspected outbreak of cholera, whether detected by clinical diagnosis or by rapid diagnostic assays, should be confirmed with culture of stools from a sample of typical patients (CDC 1994). Culture confirmation will allow antimicrobial susceptibility testing to be performed on V. cholerae O1 isolates in order to guide treatment of severely ill patients.
The results of this study cannot be generalized to all populations as the PPV of any diagnostic test varies with the prevalence of the disease in the population. Vibrio cholerae was cultured from stools of 38% of patients enrolled in this study, reflecting a high prevalence among persons presenting to the ICDDR,B hospital. The PPV reported here is, therefore, higher than would be expected in a low-cholera prevalence area.
We conclude that the IP dipstick is the most appropriate rapid diagnostic assay for the detection of V. cholerae O1 in remote locations or refugee camp settings. It was found to be valid and easy to use by both field and laboratory technicians. It does not require refrigeration, even for long-term storage, in contrast to the MedicosTM and SMARTTM assays. At a cost of $14 and $4 per determination, respectively, when purchased from the manufacturers in 2003, the SMARTTM and MedicosTM assays are well beyond reach for medical providers in areas at risk for cholera. While the cost of the IP dipstick will not be known until it becomes commercially available, cost is the ultimate factor limiting the utility of any diagnostic test in developing countries, and should be considered early in the course of assay development. At the time of writing, transfer of the IP dipstick technology to a commercial firm is underway.
The authors express sincere gratitude to patients and families for their willingness to participate in this study. Authors thank the following field and laboratory technicians: Ashrafus Safa, Fahmida Idris, Marzia Sultana, Suraia Nasreen, Shahnaz Begum, Shamim Afroz, Zahirul Islam. Finally, authors are grateful to Mr MA Malek, Anowara Khatun, Mahtab Begum and Umme Kulsum for data management, and Ms Fatema Khatun who oversaw data collection. The study was funded by the Centers for Disease Control and Prevention, Atlanta.