SEARCH

SEARCH BY CITATION

Keywords:

  • leishmaniasis;
  • trypanosomiais;
  • diagnostics;
  • nucleic acid sequence–based amplification;
  • polymerase chain reaction;
  • oligochromatography
  • leishmaniose;
  • trypanosomiase;
  • diagnostic;
  • NASBA;
  • PCR;
  • oligochromatographie
  • Leishmaniasis;
  • tripanosomiasis;
  • diagnóstico;
  • NASBA;
  • PCR;
  • oligocromatografía

Summary

  1. Top of page
  2. Summary
  3. Introduction
  4. Materials and methods
  5. Results
  6. Discussion
  7. Acknowledgements
  8. References

Objective  To evaluate the repeatability and reproducibility of four simplified molecular assays for the diagnosis of Trypanosoma brucei spp. or Leishmania ssp. in a multicentre ring trial with seven participating laboratories.

Methods  The tests are based on PCR or NASBA amplification of the parasites nucleic acids followed by rapid read-out by oligochromatographic dipstick (PCR-OC and NASBA-OC).

Results  On purified nucleic acid specimens, the repeatability and reproducibility of the tests were Tryp-PRC-OC, 91.7% and 95.5%; Tryp-NASBA-OC, 95.8% and 100%; Leish-PCR-OC, 95.9% and 98.1%; Leish-NASBA-OC, 92.3% and 98.2%. On blood specimens spiked with parasites, the repeatability and reproducibility of the tests were Tryp-PRC-OC, 78.4% and 86.6%; Tryp-NASBA-OC, 81.5% and 89.0%; Leish-PCR-OC, 87.1% and 91.7%; Leish-NASBA-OC, 74.8% and 86.2%.

Conclusion  As repeatability and reproducibility of the tests were satisfactory, further phase II and III evaluations in clinical and population specimens from disease endemic countries are justified.

Répétabilité et reproductibilité de la PCR et du NASBA suivis de l’oligochromatographie pour le diagnostic moléculaire de Trypanosoma brucei et Leishmania

Objectif:  Evaluer la répétabilité et la reproductibilité de quatre tests moléculaires simplifiés pour le diagnostic de Trypanosoma brucei spp. ou Leishmania ssp. dans un essai multicentrique avec 7 laboratoires participants.

Méthodes:  Les tests sont basés sur l’amplification par PCR ou NASBA des acides nucléiques des parasites suivie par la lecture rapide sur bandelette oligochromatographique (PCR-OC et NASBA-OC).

Résultats:  Sur les échantillons d’acide nucléique purifié, la répétabilité et la reproductibilité des tests ont été les suivantes: Tryp-PRC-OC, 91,7% et 95,5%; Tryp-NASBA-OC, 95,8% et 100%; Leish-PCR-OC, 95,9% et 98,1%; Leish-NASBA-OC, 92,3% et 98,2%. Sur les échantillons de sang mélangés avec des parasites la répétabilité et la reproductibilité des tests ont été les suivantes: Tryp-PRC-OC, 78,4% et 86,6%; Tryp-NASBA-OC, 81,5% et 89,0%; Leish-PCR-OC, 87,1% et 91,7%; Leish-NASBA-OC, 74,8% et 86,2%.

Conclusion:  As repeatability and reproducibility of the tests were satisfactory, further phase II and III evaluations in clinical and population specimens from disease endemic countries are justified.

Conclusion:  Comme la répétabilité et la reproductibilité des tests étaient satisfaisantes, des évaluations ultérieures de phase II et III sur des échantillons cliniques provenance de populations des pays endémiques sont justifiées.

Repetibilidad y reproductibilidad de la PCR y el NASBA con oligocromatografía posterior para el diagnóstico molecular de Trypanosoma brucei y Leishmania

Objetivo:  Evaluar la repetibilidad y reproductibilidad de cuatro ensayos moleculares simplificados para el diagnóstico de Trypanosoma brucei spp. o Leishmania ssp. en un ensayo multicéntrico con la participación de 7 laboratorios.

Métodos:  Las pruebas estaban basadas en la amplificación de los ácidos nucléicos del parásito mediante PCR o NASBA, y seguidos por una lectura rápida con una tira diagnóstica oligocromatográfica (PCR-OC y NASBA-OC).

Resultados:  En muestras de ácido nucléico purificado, la repetibilidad y reproductibilidad de las pruebas fueron: Tryp-PRC-OC, 91.7% y 95.5%; Tryp-NASBA-OC, 95.8% y 100%; Leish-PCR-OC, 95.9% y 98.1%; Leish-NASBA-OC, 92.3% y 98.2%. En muestras de sangre con parásitos, la repetibilidad y reproducibilidad de las pruebas fueron: Tryp-PRC-OC, 78.4% y 86.6%; Tryp-NASBA-OC, 81.5% y 89.0%; Leish-PCR-OC, 87.1% y 91.7%; Leish-NASBA-OC, 74.8% y 86.2%.

Conclusión:  Puesto que las pruebas de repetibilidad y reproducibilidad fueron satisfactorias, se justifica el realizar más pruebas de evaluación, de fase II y III en muestras clínicas y de población provenientes de países endémicos.


Introduction

  1. Top of page
  2. Summary
  3. Introduction
  4. Materials and methods
  5. Results
  6. Discussion
  7. Acknowledgements
  8. References

Polymerase chain reaction (PCR) and nucleic acid sequence–based amplification (NASBA) are established techniques for the detection of pathogen DNA or RNA, respectively, and standardized formats of PCR and NASBA have been developed for the diagnosis of human African trypanosomiasis and leishmaniasis targeting 18S ribosomal DNA or RNA (van der Meide et al. 2005; Deborggraeve et al. 2006, 2008; van der Medier et al. 2008; Mugasa et al. 2008).

To facilitate implementation of molecular diagnostics in laboratories in disease endemic countries, there is a simple dipstick format for the detection of amplification products, i.e. oligochromatography or OC (Deborggraeve et al. 2006; Claes et al. 2007; Mugasa et al. 2009). After nucleic acid amplification with either PCR or NASBA, the amplification products are allowed to migrate on the sensitized membrane of an oligochromatographic dipstick. The detection step can be performed in 5–10 min and no other equipment than a pipette and water bath are needed. Nucleic acid amplification control, internal control and migration control are incorporated in the assays. Combining PCR or NASBA with OC has led to the development of four different diagnostic tests for either trypanosomiasis or leishmaniasis: Tryp-PCR-OC (Deborggraeve et al. 2006); Tryp-NASBA-OC (Mugasa et al. 2009); Leish-PCR-OC (Deborggraeve et al. 2008) and Leish-NASBA-OC (Mugasa et al. 2010) and all tests have shown satisfactory analytical sensitivity and specificity.

Before these tests can be subjected to large-scale phase II and III evaluations with participating laboratories in disease endemic countries, they should demonstrate sufficient repeatability and reproducibility. Determining these parameters must be part of the development flow of any diagnostic test from proof-of-principle to demonstrating their utility in any diagnostic setting, but it is rarely reported. Repeatability measures the variability between measurements when performed on identical specimens by the same executor in the same laboratory and using the same equipment. Reproducibility measures the variability when identical specimens are analysed by different executors in different laboratories. However, repeatability and reproducibility cannot be measured for tests generating qualitative data such as OC tests described above. Therefore, Langton et al. (2002) introduced two new measures, as analogues for repeatability and reproducibility of tests generating qualitative data, i.e. accordance (ACC) and concordance (CON), respectively.

Our aim was to assess the ACC and CON of the PCR-OC and NASBA-OC assays on purified nucleic acid specimens as well as on human blood spiked with parasites in a multicentre ring trial with seven participating laboratories from Africa and Europe.

Materials and methods

  1. Top of page
  2. Summary
  3. Introduction
  4. Materials and methods
  5. Results
  6. Discussion
  7. Acknowledgements
  8. References

Participating laboratories

The ring trial was performed by trained laboratory technicians in seven independent laboratories in six countries: Makerere University in Uganda, Institut National de Recherche Biomédicale in Democratic Republic (DR) of the Congo, Kenya Medical Research Institute in Kenya, University of Khartoum in Sudan, The Koninklijk Instituut voor de Tropen Biomedical Research in The Netherlands, Coris BioConcept and Institute of Tropical Medicine both in Belgium. The laboratories in Europe evaluated all four tests; the laboratories in Uganda and DRC evaluated the Tryp-PCR-OC and Tryp-NASBA-OC and the laboratories in Kenya and Sudan tested the Leish-PCR-OC and Leish-NASBA-OC. All participants were trained in test execution during a one-week workshop held at Makerere University (Kampala, Uganda).

Specimen preparation

In-vitro cultured Trypanosoma brucei gambiense (LiTat 1.3) and Leishmania donovani (MHOM/SD/68/S1) parasites were suspended in phosphate-buffered saline (PBS) at a concentration of 106 parasites per ml PBS. This suspension was used to prepare two different sets of specimens. The first set of specimens comprised serially diluted parasites in Tris-EDTA (TE) buffer containing 20 μg/ml calf thymus DNA to prevent degradation of the target DNA (dilution buffer) at concentrations of 100 000, 10 000, 1000, 100, 10 and 0 parasites per ml. The second set comprised human blood from a healthy volunteer spiked with 1000, 100, 10 and 0 T. brucei gambiense or Leishmania parasites per ml of blood. All blood specimens were prepared in triplicate. Four blood specimens from non-endemic healthy volunteers were further included in the study as negative controls.

Transportation and storage of specimens

All specimens were prepared by a single individual in a central laboratory. As cooled transport to several participants could not be guaranteed, the blood specimens were stabilized on silica before shipment. This was carried out by performing the first step in the nucleic acid extraction protocol (Boom et al. 1990). In brief, 200 μl of each specimen was mixed with 1.2 ml of L6 lysis buffer (50 mmTris–HCl, 5m GuSCN 20 mm EDTA, 0.1% Triton X100) and subsequently 40 μl of silica suspension was added and mixed for a further 5 min. The samples were centrifuged and the supernatant discarded, leaving behind a pellet of silica with bound nucleic acids. The specimen sets were shipped on dry ice to the various participating laboratories. All samples were coded and tested blindly by a trained technician in each of the seven laboratories.

Extraction of nucleic acids

The silica pellet was washed twice with 1 ml of L2 wash buffer (5 m GuSCN, 100 mm Tris–HCl [pH 6.4] supplied by the central laboratory to trial participants), twice with 1 ml of 70% ethanol, and once with 1 ml of acetone. The pellet was dried at 56 °C for 5 min after which the nucleic acids were eluted in 50 μl TE buffer during 5-min incubation at 56 °C. The eluted nucleic acids were stored at −20 °C until amplification.

Ring trial study

The Trypanosoma specimen set was analysed with the Tryp-NASBA-OC and Tryps-PCR-OC; while the Leishmania specimen set was analysed with the Leish-NASBA-OC and Leish-PCR-OC. With the specimen set and necessary test reagents, the participating laboratories also received a test report sheet, questionnaire and standard operating procedures for nucleic acid extraction as described by Boom et al. (1990), and the PCR-OC and NASBA-OC execution protocols as described by Deborggraeve et al. (2006)Deborggraeve et al. (2008) and Mugasa et al. (2009, 2010). The laboratories further purified the nucleic acids from the blood specimens and analysed the blood specimens once and the nucleic acid dilutions in PBS thrice. All specimens were tested blindly. Test results were reported to the central laboratory for statistical analysis.

Statistical analysis

ACC, defined as the average chance of finding the same result for two identical specimens analysed in the same laboratory, and CON, defined as the average chance of finding the same result for two identical specimens analysed in different laboratories, of Tryp-PCR-OC, Tryp-NASBA-OC, Leish-PCR-OC and Leish-NASBA-OC were estimated in a random framework by the formulae described by van der Voet and van der Raamsdon (2004): ACCr = (1/L) Σi (p0,i2+p1,i2) and CONr = P02+P12, in which p0,i2 and p1,i2 were defined as the squared proportion of Tryp-PCR-OC, Tryp-NASBA-OC, Leish-PCR-OC and Leish-NASBA-OC negative and positive results, respectively, for each analysis i and where P02 and P12 were defined by the equations P02 = (1/L) Σ= 1p0,i and P12 = (1/L) Σ= 1p1,i with L the number of laboratories in the trial. Around the ACC and CON estimates, 95% confidence intervals (CI) were quantified by bootstrapping (Davison & Hinkley 1997).

Results

  1. Top of page
  2. Summary
  3. Introduction
  4. Materials and methods
  5. Results
  6. Discussion
  7. Acknowledgements
  8. References

An overview of the test results at the different laboratories is presented in Tables 1 (Tryp-PCR-OC and Leish-PCR-OC) and 2 (Tryp-NASBA-OC, and Leish-NASBA-OC). The Leish-PCR/NASBA-OC results of one laboratory were excluded from the data analysis as unexplained high numbers of invalid results were observed at this laboratory, probably because of incorrect test execution. A test result is considered invalid when both the test lines as the negative internal control line remain negative.

Table 1.   Results of the Tryp and Leish PCR-OC tests on the specimen sets in the seven laboratories participating in the study
Specimenno of repetitionsno of positive Tryp-OC-PCR tests at laboratoryno of positive Leish-OC-PCR tests at laboratory
1234512367
  1. no, number; NA, nucleic acids; TE, Tris-EDTA; Inv, invalid; the DNA dilution series in TE buffer were provided as one series and tested in triplicate; the parasite dilution series in blood were provided in triplicate and tested one time.

Trypanosoma brucei gambiense NA in TE buffer
 100 000 parasites/ml333333     
 10 000 parasites/ml333333     
 1000 parasite/ml333333     
 100 parasite/ml333313     
 10 parasite/ml322000     
 0 parasite/ml300000     
Leishmania donovani NA in TE buffer
 100 000 parasites/ml3     3333Inv
 10 000 parasites/ml3     3333Inv
 1000 parasite/ml3     3332Inv
 100 parasite/ml3     2332Inv
 10 parasite/ml3     2002Inv
 0 parasite/ml3     0000Inv
Trypanosoma brucei gambiense parasites in blood
 1000 parasites/ml333333     
 100 parasites/ml333223     
 10 parasites/ml333012     
 0 parasite/ml310011     
Leishmania donovani parasites in blood
 1000 parasites/ml3     3323Inv
 100 parasites/ml3     3333Inv
 10 parasites/ml3     2322Inv
 0 parasite/ml3     0020Inv
Blood from healthy donors
 Donor 11010100000Inv
 Donor 21000100000Inv
 Donor 31000100000Inv
 Donor 41000000010Inv
Table 2.   Results of the Tryp and Leish NASBA-OC tests on the specimen sets in the seven laboratories participating in the study
Specimenno of repetitionsno of positive Tryp-NASBA-OC tests at laboratoryno of positive Leish-NASBA-OC tests at laboratory
1234512367
  1. no, number; NA, nucleic acids; TE, Tris-EDTA; Inv, invalid; the DNA dilution series in TE buffer were provided as one series and tested in triplicate; the parasite dilution series in blood were provided in triplicate and tested one time.

Trypanosoma brucei gambiense NA in TE buffer
 100 000 parasites/ml333333     
 10 000 parasites/ml333333     
 1000 parasite/ml333333     
 100 parasite/ml333333     
 10 parasite/ml333330     
 0 parasite/ml300000     
Leishmania donovani NA in TE buffer
 100 000 parasites/ml3     3332Inv
 10 000 parasites/ml3     3332Inv
 1000 parasite/ml3     3332Inv
 100 parasite/ml3     3002Inv
 10 parasite/ml3     3000Inv
 0 parasite/ml3     0000Inv
Trypanosoma brucei gambiense parasites in blood
 1000 parasites/ml333333     
 100 parasites/ml333333     
 10 parasites/ml310022     
 0 parasite/ml300020     
Leishmania donovani parasites in blood
 1000 parasites/ml3     3333Inv
 100 parasites/ml3     3123Inv
 10 parasites/ml3     3001Inv
 0 parasite/ml3     0000Inv
Blood from healthy donors
 Donor 11000000000Inv
 Donor 21000100000Inv
 Donor 31000000000Inv
 Donor 41000000000Inv

ACC and CON were calculated for the four tests on the nucleic acids dilution series in PBS and on the parasite dilution series in blood separately and are presented in Table 3.

Table 3.   Overview of the accordance and concordance of the Trypanozoon and Leishmania OligoC-TesT on DNA and blood specimens analysed during the ring trial
TestAccordance % (95% CI)Concordance % (95% CI)
  1. Accordance, intralaboratory repeatability; Concordance, interlaboratory reproducibility; 95% CI, 95% confidence interval by bootstrapping.

Dilution series of DNA
 Tryps-PCR-OC test95.5 (92.6–98.5)91.7 (87.0–97.9)
 Leish-PCR-OC test98.1 (94.4–100)95.9 (91.7–100)
Dilution series of RNA
 Tryps-NASBA-OC test10095.8 (92.0–100)
 Leish-NASBA-OC test98.2 (94.4–100)92.3 (87.0–100)
Dilution series of parasites in blood
 Tryps-PCR-OC test86.6 (75.6–97.8)78.4 (68.5–91.1)
 Leish-PCR-OC test91.7 (88.9%–97.2)87.1% (80.9–96.2)
 Tryps-NASBA-OC test89.0 (82.2–95.6)81.5 (72.4–90.7)
 Leish-NASBA-OC test86.2 (77.8–94.4)74.8 (67.0–89.2)

Discussion

  1. Top of page
  2. Summary
  3. Introduction
  4. Materials and methods
  5. Results
  6. Discussion
  7. Acknowledgements
  8. References

The evaluation of molecular diagnostics (and even diagnostics in general) is often neglected. It is important to estimate the repeatability (ACC) and reproducibility (CON) of a new test prior to further phase II and III evaluation studies (which should be performed in other laboratories than the laboratory where the test was developed). The present study reports on a multi-centre evaluation study of two molecular assays in seven laboratories on two continents. Our study’s set-up and data analysis are of interest of many researchers working on diagnostic test development. The presented work can be an example for researchers wishing to develop new diagnostics for neglected tropical diseases, and it highlights some parameters that warrant control.

Our main aim was to validate inter (ACC) and intra (CON) laboratory performance of the Tryp-PCR-OC, Tryp-NASBA-OC, Leish-PCR-OC and Leish-NASBA-OC tests. When analysing purified nucleic acid specimens, the four tests showed an ACC and CON above 95% and 90%, respectively. The ACC and CON were below 90% when blood specimens were analysed. This is not surprising as the nucleic acids had to be extracted from the blood specimens by the participating laboratories, leading to higher chances of intra and interlaboratory inconsistencies. In addition, the variation in test results was predominantly observed in the low parasite range of 1–0 parasites/ml blood. As this parasite concentration is close to the analytical sensitivity of the tests (Deborggraeve et al. 2006, 2008; Mugasa et al. 2009, 2010), slight variations in equipment and operating procedures may have a major influence on the results obtained with these low parasite concentrations.

Occasional false positive results were observed in the blood specimens from healthy European controls. Although cross-reaction of the tests with human DNA or RNA cannot be excluded, carry-over contamination during DNA extraction from the blood specimens is more likely for two reasons. First, no false positive results were observed in the specimen sets containing DNA. Secondly, there is the observation that three of the five false positive results for DNA were obtained in the same laboratory. The results of one laboratory analysing the Leish-PCR/NASBA-OC tests had to be excluded because very high numbers of invalid results were generated, most probably because of faulty test execution. Similar problems were not encountered at the other laboratories.

To our knowledge, this is the first study evaluating the ACC and CON of molecular diagnostic tests for human African trypanosomiasis and leishmaniasis on blood samples. The ACC and CON of the Tryp-PCR-OC have already been estimated in a multicentre ring trial with six participating laboratories on purified DNA specimens, reported by Claes et al. (2007). In that study, the test showed an ACC of 88.7% (95%CI: 84.4%-92.5%) and a CON of 88.1% (95%CI: 84.3%-92.3%), slightly lower than the results obtained in this study.

In conclusion, the ACC (repeatability) and CON (reproducibility) of the Tryp-PCR-OC, Leish-PCR-OC, Tryp-NASBA-OC and Leish-NASBA-OC tests were successfully estimated by (i) a multicentre ring trial with several European and African participants and (ii) data analysis of the qualitative test results in a random framework as described by van der Voet and van der Raamsdon (2004). All four tests demonstrated to be repeatable and reproducible and can undergo further phase II and III evaluations in variable laboratory settings in disease endemic countries.

Acknowledgements

  1. Top of page
  2. Summary
  3. Introduction
  4. Materials and methods
  5. Results
  6. Discussion
  7. Acknowledgements
  8. References

We thank all our partners in the TRYLEIDIAG consortium for the input towards this study and Ronald Geskus (Department of Clinical Epidemiology, Biostatistics and Bioinformatics; Academic Medical Center; University of Amsterdam; Amsterdam; The Netherlands) for performing bootstrap analyses. This work was supported by the Commission of the European Communities’ Sixth Framework Programme, priority INCO-DEV, project TRYLEIDIAG, contract 015379. Stijn Deborggraeve is a postdoctoral fellow of the Research Foundation Flanders (FWO, Belgium).

References

  1. Top of page
  2. Summary
  3. Introduction
  4. Materials and methods
  5. Results
  6. Discussion
  7. Acknowledgements
  8. References
  • Boom R, Sol C, Salimans M, Jansen C, Wertheim van Dillen P & Noordaa van der J (1990) Rapid and simple method for purification of nucleic acids. Journal of Clinical Microbiology 28, 495503.
  • Claes F, Deborggraeve S, Verloo D et al. (2007) Validation of a PCR-Oligochromatography test for detection of Trypanozoon parasites in a multicenter collaborative trial. Journal of Clinical Microbiology 45, 37853787.
  • Davison AC & Hinkley DV (1997) Bootstrap Methods and Their Application. Cambridge University Press, Cambridge.
  • Deborggraeve S, Claes F, Laurent T et al. (2006) Molecular dipstick test for diagnosis of sleeping sickness. Journal of Clinical Microbiology 44, 28842889.
  • Deborggraeve S, Laurent T, Espinosa D et al. (2008) A simplified and standardized polymerase chain reaction format for the diagnosis of leishmaniasis. Journal of Infectious Diseases 198, 15651572.
  • Langton SD, Chevennement R, Nagelkerke N & Lombard B (2002) Analysing collaborative trials for qualitative microbiological methods: accordance and concordance. International Journal of Food Microbiology 79, 175181.
  • Mugasa CM, Schoone GJ, Ekangu RA, Lubega GW, Kager PA & Schallig HDFH (2008) Detection of Trypanosoma brucei parasites in blood using real-time nucleic acid sequence based amplification. Diagnostic Microbiology and Infectious Disease 61, 440445.
  • Mugasa CM, Laurent T, Schoone GJ, Kager PA, Lubega GW & Schallig HDFH (2009) Nucleic acid sequence-based amplification with oligochromatography (NASBA-OC) for the detection of Trypanosoma brucei in clinical samples. Journal of Clinical Microbiology 47, 630635.
  • Mugasa CM, Laurent T, Schoone GJ et al. (2010) Simplified molecular detection of Leishmania parasites in various clinical samples from patients with leishmaniasis. Parasites and Vectors 3, 13.
  • Van Der Medier WF, Guerra J, Schoone GJ et al. (2008) Comparison between QT-NASBA, Real-time Reverse-Transcriptase PCR and real-time PCR for quantification of Leishmania parasites. Journal of Clinical Microbiology 46, 7378.
  • Van Der Meide WF, Schoone GJ, Faber WR et al. (2005) Quantitative Nucleic Acid Sequence-Based assay as a new molecular tool for detection and quantification of Leishmania parasites in skin biopsy samples. Journal of Clinical Microbiology 43, 55605566.
  • Van Der Voet H & Van Der Raamsdon LWD (2004) Estimation of accordance and concordance in inter-laboratory trials of analytical methods with qualitative results. International Journal of Food Microbiology 95, 231234.