Comparative and collaborative studies for the validation of a nested PCR for the detection of Xanthomonas axonopodis pv. dieffenbachiae from Anthurium samples

Authors


Abstract

Efficient control of Xanthomonas axonopodis pv. dieffenbachiae, the causal agent of anthurium bacterial blight, requires sensitive and reliable diagnostic tools. The European standard EN ISO 16140:2003 has been followed to compare a nested PCR assay (N-PCR) to a reference method (isolation and serological identification of bacterial colonies) and to other alternative serological detection methods. The evaluation was performed in two steps: a comparative study and a collaborative study involving 15 European laboratories. Although inclusivity was maximal (100%) for all methods, a maximal exclusivity was obtained only with N-PCR followed by an enzymatic restriction digestion of the amplicons. Exclusivity indices of 90·6, 88·7 and 47·2% were found for indirect ELISA, immunofluorescence and double antibody sandwich ELISA, respectively. An exclusivity of 92·5% was obtained with the reference method, further increased to 100% if pathogenicity tests were performed as a supplemental assay. The best level of sensitivity (relative detection level) was obtained with the reference method followed by the N-PCR assay. The N-PCR performance in terms of relative accuracy, accordance and concordance was very similar to that of the reference method. Moreover, N-PCR had undeniable advantages compared to the reference method (less labour-intensive and less time-consuming). In addition, post-test probabilities of infection were calculated to select the most appropriate detection scheme related to the prevalence of the pathogen. The N-PCR assay has since been included in a revised version of the EPPO detection protocol.

Introduction

Xanthomonas axonopodis pv. dieffenbachiae is the aetiological agent of bacterial blight of aroids (BBA), first described in the USA on Dieffenbachia maculata in 1939 (McCulloch & Pirone, 1939). Xanthomonas axonopodis pv. dieffenbachiae can infect a broad range of plant genera in the family Araceae (aroids), including species and cultivars of Aglaonema, Alocasia, Anthurium, Caladium, Syngonium and Xanthosoma (Norman & Alvarez, 1989; Chase et al., 1992). Anthurium was first reported to be the major host in Brazil in 1952 (Robbs, 1955). BBA is now widely distributed (Anonymous, 2009) and is a major concern in most of the Anthurium-producing regions (Robène-Soustrade et al., 2006). The phenotypic and genetic diversity of strains originating from several aroid species is related to geographic origin and host plant (Chase et al., 1992; Berthier et al., 1994; Khoodoo & Jaufeerally-Fakim, 2004). Two phylogenetically distinct groups of strains pathogenic to aroids have been described. One group of strains, which is highly pathogenic to Anthurium, in addition to a wide range of other plant species, was assigned to X. axonopodis cluster 9 (Rademaker et al., 2005). The second group, genetically related to X. axonopodis cluster 9.6 (synonym X. citri sensu Ah-You et al., 2009), includes strains isolated from various aroid species and primarily pathogenic to their host of origin but weakly or not pathogenic to Anthurium (Anonymous, 2009).

Xanthomonas axonopodis pv. dieffenbachiae is a regulated pest in several countries and is included in the A2 list of the European and Mediterranean Plant Protection Organization (EPPO). One of the EPPO members, the Netherlands, has become the largest producer and exporter of Anthurium worldwide. Until recently, the EPPO diagnostic protocol for the detection of X. axonopodis pv. dieffenbachiae was based on a reference method which consisted of isolation of the bacterium on semiselective media, followed by identification of putative Xanthomonas colonies by immunofluorescence (IF) and/or enzyme-linked immunosorbent assays (ELISAs) using polyclonal or monoclonal antibodies (Anonymous, 2004). Few molecular tools are available to detect this pathogen. Khoodoo et al. (2005) have developed a multiplex polymerase chain reaction (PCR) combined with a genus-specific monoclonal antibody for the sensitive detection of the pathogen directly from plants. This triplex PCR was developed to detect and differentiate all the different groups among the pathovar dieffenbachiae, resulting in very complex amplification profiles, even for non-target strains. Unfortunately, this sensitive detection tool was not well adapted to routine diagnosis with an all-or-nothing response. A nested PCR test (N-PCR) has recently been developed to specifically detect and identify X. axonopodis pv. dieffenbachiae strains pathogenic to Anthurium. This PCR-based method was described as sensitive (detection threshold c. 103 colony-forming units (CFU) mL−1). Moreover a restriction digestion step performed on the N-PCR amplicons made it possible to distinguish X. axonopodis pv. dieffenbachiae strains from the genetically closely related pv. syngonii strains that are pathogenic to Syngonium but not to Anthurium (Robène-Soustrade et al., 2006).

The N-PCR method, as well as other methods proposed in the PM7/23(1) EPPO protocol (Anonymous, 2004) have never been directly compared, particularly in terms of specificity and sensitivity. Comparison from the literature is difficult because of marked differences in the isolate collections used in the original papers.

The EN ISO 16140:2003 standard (Anonymous, 2003) aims to provide a protocol for the validation of alternative methods to ensure that results obtained with such methods are at least equivalent to those provided by the reference method. The standard includes both comparative and interlaboratory collaborative studies.

The purpose of this study was to compare the N-PCR method to other methods in the current EPPO protocol for X. axonopodis pv. dieffenbachiae following the recommendations of the EN ISO 16140:2003 standard, in order to validate the N-PCR-based method and to provide the scientific basis for a revised version of the PM7/23 EPPO protocol. The approach of the EN ISO 16140:2003 standard was complemented by another approach based on Bayes' theorem which, although new in plant pathology (Massart et al., 2008; Olmos et al., 2008; Vidal et al., 2012), is widely used in medical and veterinary sciences to evaluate diagnostic tests. An estimation of the likelihood ratios and the post-test probabilities of infection give interesting results on the performance and limits of the different methods, and is particularly useful for selecting the most appropriate detection scheme.

Materials and methods

Comparative study

The four methods compared in this study comprised the N-PCR (Robène-Soustrade et al., 2006) and those listed in the EPPO diagnostic protocol for X. axonopodis pv. dieffenbachiae (Anonymous, 2004). In the EPPO protocol, the reference method has been defined as the cultivation of the pathogen on agar media followed by a serological identification of bacterial colonies (Anonymous, 2004). Indirect ELISA (monoclonal antibodies, Agdia-Biofords), IF (PRI polyclonal antibodies, Wageningen, The Netherlands), and double antibody sandwich ELISA (DAS ELISA) (PRI polyclonal antibodies) were performed following the instructions of the supplier of antibodies. N-PCR was performed using boiled bacterial suspensions or, when assaying plant samples, purified DNA extracted with the DNeasy Plant Mini Kit (QIAGEN) as reported previously (Robène-Soustrade et al., 2006).

Evaluation criteria

Inclusivity, exclusivity, relative detection level (RDL) and detection level 100% (DL100%), relative accuracy (AC), relative sensitivity (SE) and relative specificity (SP) were evaluated. The inclusivity is defined as the ability of a method to detect a wide range of target strains. The exclusivity is defined as the lack of interference in a method from a relevant range of non-target strains. The RDL and the DL100% of a given method are defined as the smallest number of cells (as determined from agar plate counts of the corresponding suspensions) that can be detected 50 or 100% of the time, respectively. The other parameters are based on the comparison of any alternative method to the reference method (see above) and were calculated from data gathered from naturally infected plant material. The relative accuracy (AC) is defined as the degree of correspondence between the response obtained by the reference method and the response obtained by the alternative method on identical samples. The relative sensitivity (SE) is defined as the ability of the alternative method to detect the target when it is detected by the reference method. The relative specificity (SP) is defined as the ability of the alternative method to fail to detect the target when it is not detected by the reference method. Details on how each parameter was calculated are provided in Supporting Information S1 and are available in the EN ISO 16140:2003 standard. Differences in exclusivity, SE and SP among the tested methods were assessed with Fisher's exact test using the R statistical software (v. 2·14·1; R Development Core Team).

Strains and samples

Bacterial strains used for testing inclusivity and exclusivity are listed in Tables S1 and S2, respectively. For exclusivity, a broad set of non-target strains (= 53) were tested, including strains described as X. axonopodis pv. dieffenbachiae but not pathogenic to Anthurium (group A), strains belonging to different pathovars of X. axonopodis (group B), saprophytic bacteria isolated from Anthurium (group C; Robène-Soustrade et al., 2006), and strains pathogenic to Anthurium belonging to other genera (Ralstonia solanacearum phylotype II sequevar 4NPB, Acidovorax anthurii and Dickeya dieffenbachiae; group D). Group B strains were selected because they were previously reported as non-target strains that gave a positive result with the N-PCR assay (Robène-Soustrade et al., 2006). All target strains, as well as group A and B non-target strains, were checked for pathogenicity by artificial inoculation of Anthurium andreanum cv. Florida plants as described previously (Robène-Soustrade et al., 2006). Four weeks after inoculation, bacterial isolations from plant tissues with symptoms and bacterial population size determinations were performed on NCTM4 semiselective medium (Laurent et al., 2009). The identification of reisolated cultures was carried out using indirect ELISA and N-PCR.

The evaluation of inclusivity and exclusivity was performed on bacterial suspensions prepared from 24 h-old cultures on YPGA (7 g L−1 yeast extract, 7 g L−1 peptone, 7 g L−1 glucose, 18 g L−1 agar, 20 μg mL−1 propiconazole; pH 7·2) at 28°C. Following guidelines in the EN ISO 16140:2003 standard, suspensions were adjusted to approximately 1 × 105 and 1 × 107 CFU mL−1 for assessment of inclusivity and exclusivity, respectively.

The evaluation of RDL and DL100% were performed with four X. axonopodis pv. dieffenbachiae strains pathogenic to Anthurium (LMG695, NCPPB3573, LMG12720 and LB96; Table S1). Healthy Anthurium leaves were ground with a Homex 6 grinder (BIOREBA) consisting of 1 g in 20 mL of appropriate extraction buffer. Leaf homogenates were spiked with tenfold dilutions of bacterial suspensions to obtain target population sizes ranging from 5 × 101 to 5 × 105 CFU mL−1. Spiked samples were preferred to inoculated tissues because of a better control of the inoculum quantities and benefits in terms of sample homogeneity and stability over time (preliminary studies, data not shown). Negative controls were prepared replacing bacterial suspensions with sterile Tris buffer. Extraction of total DNA was performed using the DNeasy Plant Mini Kit (QIAGEN) following the manufacturer's instructions. This experiment was repeated five times to obtain six replicates per strain and per bacterial concentration. The evaluation of the same two criteria was also performed with the same bacterial suspensions free of plant extract (not required by the ISO standard).

Thirty-four Anthurium samples, originating from Réunion Island, Martinique, Mauritius, New Caledonia and Madagascar, were analysed in triplicate for estimating AC, SE and SP. As recommended in the standard, half of the samples were naturally infected with X. axonopodis pv. dieffenbachiae (and showed BBA disease symptoms) and half of them were healthy.

Collaborative study

A collaborative study involving 15 European laboratories was performed to determine the interlaboratory variability of the results obtained from identical samples and to compare these results to those obtained in the comparative study. Detailed information on the participating laboratories is available in Table S3. Four methods were evaluated: two methods (the EPPO reference method and N-PCR) were conducted by all participating laboratories; laboratories also had to perform IF and/or DAS ELISA as optional methods. Indirect ELISA was not proposed as the RDL values obtained from plant samples in the comparative study were not satisfactory.

Samples and controls

Healthy, surface-sterilized A. andreanum cv. Florida leaves were used to prepare an homogenate in the organizing laboratory as follows. Anthurium leaf tissue was homogenized with a Homex 6 grinder (BIOREBA) using 10 mL of DNA-free water per g leaf tissue. This homogenate was used to prepare four samples: L0, L1, L2 and L*. Samples L0, L1 and L2 consisted of leaf homogenate spiked with bacterial suspensions prepared from LMG695 (the pathotype strain) to a final concentration of 0, c. 2 × 104 and 2 × 105 CFU mL−1, respectively. Sample L* consisted of the leaf homogenate spiked with a bacterial suspension of a saprophytic strain isolated from Anthurium (JX59, Sphingomonas sanguinis) to a final concentration of c. 107 CFU mL−1. Each of the four samples was dispensed as eight subsamples that were shipped at a low temperature (2–8°C) as blind samples to the 15 laboratories. Preliminary tests aiming to verify the homogeneity and the stability (namely the bacterial concentration and viability) of the different samples at 5°C for 168 h were successfully performed (data not shown).

The shipment to the 15 laboratories included one negative control and four positive controls. All the controls were prepared in the organizing laboratory and were checked before shipment to ensure that they would give the expected result. The negative control (T1) was an aliquot of the leaf homogenate used to prepare the samples (see above). The positive control T2 consisted of the leaf homogenate spiked with a bacterial suspension of strain LMG695 to obtain a final concentration of c. 106 CFU mL−1. Positive control T3 consisted of naturally contaminated Anthurium leaves with symptoms collected in Réunion Island. Positive controls T4 and T5 were bacterial suspensions of strain LMG695 containing c. 106 and 104 CFU mL−1, respectively.

Data analysis

The total number of true positives (TP, a positive result is obtained when a positive result is expected), true negatives (TN, a negative result is obtained when a negative result is expected), false positives (FP, a positive result is obtained when a negative result is expected) and false negatives (FN, a negative result is obtained when a positive result is expected) were determined for each laboratory and each method. The EN ISO 16140:2003 standard stipulates that collaborative studies should be based on data from laboratories with high competence on the techniques that are being compared. Consequently, the results of a laboratory were excluded for a given method when: (i) the expected result for at least one control was not obtained, or (ii) when ≥50% of FP results were recorded in at least one sample. Some indeterminate results (i.e. the operator was unable to determine the status of the subsample) were reported by some laboratories for DAS ELISA (i.e. when OD405 nm values ranged between two and three times the average of negative controls) and IF methods. In order to complete the analysis (the procedure in the EN ISO 16140:2003 standard cannot handle missing data), two scenarios were tested: H1, the laboratory hypothetically made the right decision for the indeterminate results in relation to the samples' real status (i.e. the indeterminate results were counted as positive for L1 and L2 and negative for L0 and L*); and H2, the reverse. This made it possible to estimate an interval, which included the range of the parameters' real values.

Details on how each parameter was calculated are provided in Supporting Information S1 and are available in the EN ISO 16140:2003 standard. Tests on the equality of SE and SP between methods were performed using Fisher's exact test.

The concordance odds ratio (COR) was calculated as the following ratio:

display math

where CO is concordance and DA is accordance (see Supporting Information S1). The larger the odds ratio is, the more interlaboratory variation is predominant. For COR values above 1·00, the Fisher's exact test was used to evaluate the statistical significance of the variation between laboratories.

The likelihood ratio (LR) indicates how much a given diagnostic test result will raise or lower the pre-test probability of the disease in question and is a useful tool for assessing the effectiveness of a diagnostic test (Jaeschke et al., 2002). LRs were calculated as follows:

display math

i.e. the probability of a positive test result for an infected sample divided by the probability of a positive result for a healthy sample;

display math

i.e. the probability of a negative test result for an infected sample divided by the probability of a negative test result for a healthy sample.

The interpretation of the LRs of a diagnostic test was established as follows (Jaeschke et al., 2002): LRs of >10 or <0·1 generate large and often conclusive changes from pre- to post-test probability, LRs of 5–10 or 0·1–0·2 generate moderate shifts in pre- to post-test probability, LRs of 2–5 or 0·2–0·5 generate small (but sometimes important) changes in probability, LRs of 1–2 or 0·5–1 alter probability to a small (and rarely important) degree.

Bayes' theorem was further used to translate the information given by the LRs into a probability of disease. Post-test probabilities were simulated using a range (0·01–99%) of pre-test probabilities using Bayes' theorem as follows (Deeks & Altman, 2004; Neves et al., 2004):

display math
display math
display math
display math

where prevalence is defined as the proportion of Anthurium plants infected by X. axonopodis pv. dieffenbachiae in a particular population of plants at a specific time.

To combine the results of two methods, the post-test odds were calculated as follows (Neves et al., 2004):

display math

Results

Comparative study

Inclusivity and exclusivity

In contrast to non-target strains, all target strains were pathogenic to Anthurium and produced lesions measuring c. 2–3 cm in diameter, with population sizes per lesion ranging from 8·0 × 106 to 1·0 × 109 CFU mL−1. The 50 target strains tested positive (inclusivity of 100%) for all assayed methods (Table 1).

Table 1. Comparison of identification methods performed on bacterial suspensions
Method/criteriaInclusivity (%)aExclusivity (%)bRDLc (CFU mL−1)DL100%c (CFU mL−1)
  1. a

    Inclusivity (95% confidence interval): the ability of a method to detect all target strains (n = 50).

  2. b

    Exclusivity (95% confidence interval): the lack of interference from a relevant range of non-target strains (n = 53). Values followed by the same letter in a column are not significantly different (= 0·05) according to Fisher's exact test (two-tailed).

  3. c

    Relative detection level (RDL), detection level 100% (DL100%): the smallest number of cells (as determined from agar plate counts of the corresponding suspensions) that can be detected 50% or 100% of the time, respectively. CFU: colony-forming units.

Reference method

(with pathogenicity tests)

100 (92·8–100)100A (93·2–100)≤ 5 × 1015 × 102–5 × 103

Reference method

(without pathogenicity tests)

100 (92·8–100)92·5A (82·1–97·0)≤ 5 × 1015 × 102–5 × 103
N-PCR with restriction digestion100 (92·8–100)100A (93·2–100)5 × 101–5 × 1025 × 102–5 × 103
Indirect ELISA100 (92·8–100)90·6AB (79·7–95·9)5 × 102–5 × 1035 × 103–5 × 104
DAS ELISA100 (92·8–100)47·2C (34·4–60·3)5 × 103–5 × 1045 × 103–5 × 104
Immunofluorescence100 (92·8–100)88·7B (77·4–94·7)5 × 102–5 × 1035 × 102–5 × 103

The evaluation of exclusivity revealed marked differences between methods (Table 1). The reference method could exclude all non-target strains (exclusivity of 100%) only when pathogenicity tests were performed; when the pathogenicity tests were not included in the reference method, it was unable to exclude one group A strain (X. axonopodis pv. dieffenbachiae LD117) and the three group B strains, lowering the exclusivity to 92·5%. Maximal exclusivity was achieved with N-PCR followed by the restriction digestion step, without performing pathogenicity tests. An exclusivity of 90·6% was obtained with the indirect ELISA assay as it did not exclude one group A strain (X. axonopodis pv. dieffenbachiae LD117), the three group B strains and one group C strain (Rhodobacter sphaeroides JX39). DAS ELISA did not exclude the strains from either group A and group B or two group C strains (Sphingomonas sanguinis JX59 and Curtobacterium flaccumfaciens JX77), resulting in an exclusivity of only 47·2%. An exclusivity of 88·7% was obtained with the IF test because it could not exclude three group A strains (X. axonopodis pv. dieffenbachiae LD116, LD117 and LD119) and the three group B strains.

Relative detection level and DL100%

The best RDL was obtained with the reference method (≤5 × 101 CFU mL−1) both on bacterial suspensions and spiked plant extracts (Tables 1, 2). Among the other methods tested, N-PCR gave the best results with an RDL between 5 × 101 and 5 × 102 CFU mL−1 on bacterial suspensions and between 5 × 102 and 5 × 103 CFU mL−1 on spiked plant extracts. Interestingly, DL100% for N-PCR was identical to that of the reference method when estimated from bacterial suspensions and was 10 times higher when estimated from plant extracts.

Table 2. Comparison of detection methods performed on plant extracts
Method/criteriaSpiked plant extractsNaturally infected plant material
RDLa (CFU mL−1)DL100%a (CFU mL−1)AC (%)bSE (%)cSP (%)d
  1. a

    Relative detection level (RDL), detection level 100% (DL100%): the smallest number of cells (as determined from agar plate counts of the corresponding suspensions) that can be detected 50% or 100% of the time, respectively. CFU: colony-forming units.

  2. b

    Relative accuracy, AC (95% confidence interval): the degree of correspondence between the response obtained by the reference method and the response obtained by the alternative method on identical samples.

  3. c

    Relative sensitivity, SE (95% confidence interval): the ability of the alternative method to detect the target when it is detected by the reference method. Values followed by the same letter in a column are not significantly different (= 0·05) according to Fisher's exact test (two-tailed).

  4. d

    Relative specificity, SP (95% confidence interval): the ability of the alternative method to fail to detect the target when it is not detected by the reference method. Values followed by the same letter in a column are not significantly different (= 0·05) according to Fisher's exact test (two-tailed).

  5. e

    The same RDL and DL100% values were obtained whether the reference method was supplemented or not with the optional pathogenicity tests.

Reference methode≤ 5 × 1015 × 102–5 × 103
N-PCR5 × 102–5 × 1035 × 103–5 × 10498·0 (93·1–99·5)100A (93·0–100)96·1A (86·8–98·9)
Indirect ELISA> 5 × 105> 5 × 10592·2 (85·3–96·0)86·3B (74·3–93·2)98·0A (89·7–99·6)
DAS-ELISA5 × 103–5 × 1045 × 103–5 × 10497·1 (91·7–99·0)100A (93·0–100)94·1A (84·1–98·0)
Immunofluorescence5 × 103–5 × 1045 × 104–5 × 10595·1 (89·0–97·8)90·2AB (79·0–95·7)100A (93·0–100)

Relative accuracy, relative sensitivity and relative specificity

These parameters were estimated from naturally infected plant material. All methods showed a high degree of correspondence of data to that obtained by the reference method (relative accuracy ≥90% for all the methods), the best of them being N-PCR (AC = 98·0%; Table 2). On the basis of the SE and SP parameters, two categories could be distinguished in the methods evaluated: (i) methods that tended to generate false positive results, namely N-PCR (because of a risk of contamination by DNA aerosols when opening the tubes from the first PCR round containing the amplicons used as template for the second round) and DAS ELISA (because of cross reactions due to weakness in the specificity of antibodies) with SP indices of 96·1 and 94·1%, respectively; and (ii) methods that tended to generate false negative results, such as indirect ELISA and IF with SE values of 86·3 and 90·2%, respectively.

Collaborative study

The performance criteria of the different methods are summarized in Table 3 (and Fig. S1). Detailed results obtained for each laboratory and for each type of sample are available in Tables S4 and S5, respectively. The results for the reference method were validated because expected results were obtained for all positive and negative controls in each laboratory. Conversely, some data obtained by a number of laboratories from other assayed methods were not validated by the control responses and were excluded from the analysis (see detailed results in Supporting Information S1). The best overall performance was obtained with the reference method, followed by the N-PCR assay. Less satisfactory results were achieved with the IF assay and DAS ELISA.

Table 3. Comparison of the performance criteria obtained during the collaborative study for the different methods
Method/criteriaAC (%)aSE (%)bSP (%)cDA (%)dCO (%)eSignificant variation in performance between laboratoriesSignificant variation between results produced by the method and theoretically expected results
  1. a

    Relative accuracy, AC (95% confidence interval): the degree of correspondence between the response obtained by the reference method and the response obtained by the alternative method on identical samples.

  2. b

    Relative sensitivity, SE (95% confidence interval): the ability of the alternative method to detect the target when it is detected by the reference method. Values followed by the same letter in a column are not significantly different (= 0·05) according to Fisher's exact test (two-tailed).

  3. c

    Relative specificity, SP (95% confidence interval): the ability of the alternative method to fail to detect the target when it is not detected by the reference method. Values followed by the same letter in a column are not significantly different (= 0·05) according to Fisher's exact test (two-tailed).

  4. d

    Accordance, DA (95% confidence interval) is the equivalent of the repeatability for quantitative methods.

  5. e

    Concordance, CO (95% confidence interval) is the equivalent of the reproducibility for quantitative methods.

  6. f

    For these techniques, data is derived from the two scenarios, H1 and H2, described in the Materials & Methods section for the interpretation of indeterminate results.

Reference method

(without pathogenicity tests)

99·6 (98·5–99·9)99·6A (97·7–99·9)99·6A (97·7–99·9)99·3 (98·5–100)99·2 (98·8–99·8)No, for all samplesNo test available
N-PCR96·4 (94·1–97·8)97·6A (94·5–99·0)95·2B (91·4–97·4)93·9 (90·4–98·7)93·0 (91·2–95·5)No, for all samplesNo (= 0·302)
DAS ELISAf(H1) 81·7(H1) 70·5B(H1) 92·9B(H1) 91·0(H1) 78·5(H1) No, for samples L0, L2 and L*;Yes
(76·1–86·2)(61·5–78·2)(86·5–96·3)(84·3–98·9)(75·4–82·6) yes, for sample L1(H1= 9·47 × 10−5
(H2) 77·2(H2) 63·4B(H2) 91·1B(H2) 89·0(H2) 74·3(H2) No, for samples L0 and L*;(H2) = 1·42 × 10−5
(71·3–82·2)(54·2–71·7)(84·3–95·1)(81·4–97·7)(70·1–78·0) yes, for samples L1 and L2 
Immunofluorescencef(H1) 85·4(H1) 76·0B(H1) 94·8B(H1) 83·9(H1) 75·5(H1) No, for samples L0 and L*;Yes
(79·7–89·7)(66·6–83·5)(88·4–97·8)(72·7–95·6)(70·5–81·1) yes, for samples L1 and L2(H1= 6·70 × 10−4
(H2) 81·8(H2) 72·9B(H2) 90·6B(H2) 82·9(H2) 69·6(H2) No, for sample L*;(H2= 4·06 × 10−3
(75·7–86·6)(63·3–80·8)(83·1–95·0)(71·2–96·5)(62·8–79·6) yes, for samples L0, L1 and L2 

The relative accuracy was superior to 95% for the reference method and N-PCR. Under the two scenarios presented in the Material & Methods section for the interpretation of indeterminate results, the relative accuracy ranged from 77·2 (H2) to 81·7% (H1) for DAS ELISA and from 81·8 (H2) to 85·4% (H1) for IF (Table 3). Less satisfactory results obtained for the serological methods are particularly related to low SE scores: between 35·7 (H2) and 42·9% (H1) for the L1 sample assayed by DAS ELISA, and 75·0% (H1; 70·8 with H2) and 77·1% (H1; 75·0 with H2) for L1 and L2 samples assayed by IF, respectively (Table S5).

The overall accordance was above 90% for the reference method, the N-PCR assay and the DAS ELISA assay (H1), and ranged from 82·9 (H2) to 83·9% (H1) for the IF assay (Table 3). The overall concordance was above 92% for the reference method and the N-PCR assay, was from 74·3 (H2) to 78·5% (H1) for the DAS ELISA assay, and ranged from 69·6 (H2) to 75·5% (H1) for the IF assay. The poor results for the serological methods are a result of particularly low concordance for sample L1 assayed by DAS ELISA (from 44·8 to 47·6%) and for L1 and L2 samples assayed by IF (from 54·2 to 59·6% and from 59·8 to 62·5%, respectively; Table S5).

The concordance odds ratio, COR, was equal or close to 1·00 for the four samples assayed with the reference method and Fisher's exact test showed no significant difference between laboratories for this method whatever the sample (= 1·000; Table S5). COR ranged from 1·10 (sample L0) to 1·21 (sample L1) with the N-PCR assay and Fisher's exact test showed no significant difference between laboratories (0·363 ≤   1·000). COR ranged from 1·15 to 1·82 for the L0, L2 and L* samples with the DAS ELISA assay (H1 or H2). It was noticeably ≫1·00 for the L1 sample, revealing a highly significant difference between laboratories (Table S5). COR ranged from 1·12 to 1·49 for the two negative samples (L0 and L*) with the IF assay (H1 or H2). It was noticeably ≫1·00 for the L1 and L2 samples. Fisher's exact test showed a highly significant variation between laboratories for the L1 and L2 samples (both H1 and H2; Table S5).

The total number of discordant results obtained with the reference method (Y = 2) was too low to be statistically compared to the expected theoretical results. The binomial distribution, used for the N-PCR, for which 15 discordant results were obtained from the 13 laboratories, highlighted no significant variation from the expected theoretical results (= 0·302; Table S5). The McNemar's test was used for the DAS ELISA and IF assays because the total number of discordant results was above 22 (41 and 28, respectively with H1). This test showed that there was a significant variation from the expected theoretical results for the DAS ELISA and IF assays (Table 3).

The likelihood ratios are shown in Table 4. The positive and negative LRs of both reference method and N-PCR (>10) correspond to a large change from pre- to post-test probability, indicating that these two methods are useful in the detection of X. axonopodis pv. dieffenbachiae. The LR+ values from ELISA and IF are much lower, indicating that these methods generate a moderate (to large for IF with H1) change from pre- to post-test probability. The reliability of a positive test result is, therefore, lower for these two methods than for the reference and N-PCR methods. The LR– of ELISA and IF indicates that these methods generate small changes from pre- to post-test probability. The reliability of a negative test result is, therefore, much lower for these two methods than for the reference and N-PCR methods.

Table 4. Comparison of likelihood ratios obtained during the collaborative study for the different methods
Method/criteriaLR+aLR–b
ValuecChange from pre- to post-probabilityValuecChange from pre- to post-probability
  1. a

    The positive likelihood ratio LR+ was defined as the ratio SE/(1 – SP), where SE refers to relative sensitivity and SP refers to relative specificity.

  2. b

    The negative likelihood ratio LR– was defined as the ratio (1 – SE)/SP where SE refers to relative sensitivity and SP refers to relative specificity.

  3. c

    Value of likelihood ratio (95% confidence interval).

  4. d

    For these techniques, data is derived from the two scenarios, H1 and H2, described in the Materials & Methods section for the interpretation of indeterminate results.

Reference method39·00 (33·80–1689·85)Large

4·18 × 10−3

(5·91 × 10−4–2·96 × 10−2)

Large
N-PCR20·30 (11·08–37·18)Large

2·52 × 10−2

(6·01 × 10−2–1·06 × 10−2)

Large
DAS ELISAd(H1) 9·87 (5·01–19·46)(H1) Moderate(H1) 0·317 (0·237–0·425)(H1) Small
(H2) 7·10 (3·87–13·04)(H2) Moderate(H2) 0·402 (0·313–0·516)(H2) Small
Immunofluorescenced(H1) 14·60 (6·17–34·53)(H1) Large(H1) 0·253 (0·176–0·362)(H1) Small
(H2) 7·77 (4·13–14·66)(H2) Moderate(H2) 0·299 (0·214–0·418)(H2) Small

The LR can be combined with the prevalence of the infection to determine the post-test probability of the infection. Figure 1 and Table S6 illustrate the post-test probabilities of X. axonopodis pv. dieffenbachiae infection (i.e. after a test result) as a function of the pre-test probabilities for each evaluated method and also for the combination of the two most reliable methods (reference method and N-PCR). As an example, in a population with a prevalence of 50%, the probability of a tested individual being really infected after a positive result is high for all methods (99·6% for the reference method, 95·3% for the N-PCR, between 93·6 (H1) and 88·6% (H2) for IF and between 90·8 (H1) and 87·6% (H2) for ELISA). After a negative result, there is only 0·4% and 2·5% probability that the Anthurium plant is infected by X. axonopodis pv. dieffenbachiae when tested with the reference method and the N-PCR assay, respectively. Conversely, relatively high probabilities of infection are found for samples tested negative with ELISA (24·1 to 28·7% for H1 or H2) or IF (20·2 to 23% for H1 or H2).

Figure 1.

Relation between pre- and post-test probabilities of Xanthomonas axonopodis pv. dieffenbachiae infection, according to the results obtained during the collaborative study for each evaluated method and for the combination of both reference method and nested PCR (N-PCR) method. aPre-test probability (prevalence) was defined as the proportion of Anthurium plants infected by X. axonopodis pv. dieffenbachiae in a particular population at a specific time. bThe post-test probability was calculated as follows: post-test odds/(1 +  post-test odds) where post-test odds = pre-test probability/(1 – pre-test probability) × likelihood ratio. cFor each method, the solid line represents the post-test probabilities of X. axonopodis pv. dieffenbachiae infection after a positive test result for different prevalences. The broken line represents the post-test probabilities of X. axonopodis pv. dieffenbachiae infection after a negative test result for different prevalences. dFor techniques labelled H1 and H2, data was derived from the two scenarios described in the Materials & Methods section for the interpretation of indeterminate results.

Discussion

Comparative and/or collaborative trials of diagnostic methods based on reference protocols are commonly performed to ensure a reliable validation of diagnostic methods for human or animal pathogens (Noordhoek et al., 1996; Vinjé et al., 2003; Krause et al., 2006; Wang et al., 2008). For plant pathogens, including xanthomonads, several comparative studies are available, although they are not always based on standards (Wang et al., 1999; Songwattana & Ketudat-Cairns, 2011). Collaborative studies are also rare (Ioos & Iancu, 2008), although some EPPO diagnostic protocols have been ring-tested in different European laboratories. However, details of the results and analysis have not been provided (e.g. for the diagnostic protocol of Erwinia amylovora or Xanthomonas fragariae).

The performance characteristics of the PCR-based assay (N-PCR) were evaluated through both collaborative and comparative trials, which covered complementary criteria. In the comparative study, the analytical specificity of the tests are of particular interest because they were performed on the same large panel of target and unrelated strains chosen in the light of available diversity data (Rademaker et al., 2005; E. Jouen, unpublished data). The inclusivity was maximal for both the reference method and N-PCR assay, whereas only the N-PCR assay followed by a restriction digestion step allowed for the exclusion of all non-target strains, consistent with previous data (Robène-Soustrade et al., 2006). The reference method was able to produce the same maximal exclusivity only if the final step of the EPPO scheme, consisting of pathogenicity tests, was included. Nevertheless, pathogenicity tests are not systematically performed during routine diagnosis because they require quarantine facilities and are time-consuming (at least 10–15 days before symptom development).

In contrast, the best level of sensitivity was obtained with the reference method, regardless of the type of tested sample (RDL of the reference method: <5 × 101 CFU mL−1 for both bacterial suspensions and spiked plant extracts, RDL of the N-PCR assay = RDL of the reference method × 10 or  × 100 for bacterial suspensions and spiked plant extracts, respectively). The differences in sensitivity are less marked when comparing the DL100% parameter (DL100% of the N-PCR assay = DL100% of the reference method × 1 or × 10 for bacterial suspensions and spiked plant extracts, respectively). These results are consistent with the data found in the literature (detection threshold of 101–102 CFU mL−1 and 103 CFU mL−1 for culture- and PCR-based methods, respectively (López et al., 2003)).

There was a very high correspondence between the results obtained by the reference method and the N-PCR assay on plant extracts, with a relative accuracy of 98·0% and a relative sensitivity of 100%. Unsurprisingly, the relative specificity was not maximal (96·1%), indicating that the N-PCR had generated a few false positive responses. This is a typical problem with N-PCR tests, due to contamination by DNA aerosols when opening the tubes from the first PCR round containing the amplicons used as template for the second round. Results could be improved by reinforcing the application of different strategies to prevent and/or eliminate contaminations by target templates (Champlot et al., 2010).

The reference method showed the best performance in the collaborative trial, with all measured criteria being ≥99%. The results of the N-PCR assay were close to those of the reference method, with all criteria being ≥93%. Both methods displayed results that were not statistically different from the theoretically expected results. The heterogeneity of some of the reagents and equipment used for both methods, an inherent feature of large-scale collaborative studies, may have had a greater effect on the quality of the PCR-based method. For example, differences in PCR efficiency have been reported when different cycler types or Taq DNA polymerases are used (Abu Al-Soud & Rådström, 1998; Saunders et al., 2001). Moreover, recent experiments in the organizing laboratory suggest that the N-PCR technique is more sensitive and repeatable when Eurogentec RedGoldStar Taq DNA polymerase is replaced by Promega Go Taq DNA polymerase, for example (data not shown).

When other criteria besides technical performance were considered, the N-PCR assay had undeniable advantages over the reference method. The target bacterium was identified directly using the N-PCR assay, whereas the reference method required several steps to confirm its identity. The reference method is very time-consuming: it requires 3–4 days for the bacterial isolation step, 2 more days for the serological identification and an extra 15–30 days if optional pathogenicity tests are carried out. In contrast, the N-PCR test can be completed within 2 days.

The serological methods included in the EPPO diagnostic protocol, i.e. indirect ELISA, DAS ELISA and IF, were evaluated in the comparative study, and also in the collaborative study for the two last methods. All three serological methods showed a full inclusivity in the collaborative study but displayed an exclusivity lower than that for the reference and the N-PCR methods, most critically for the DAS ELISA assay that could only exclude about half of the non-target strains tested. Not surprisingly, this ELISA assay, based on polyclonal antibodies, showed cross-reactivity against unrelated strains (López et al., 2003; Alvarez, 2004) and is therefore not highly suitable for the purpose of diagnosis. The same polyclonal antiserum was used in the IF assay but the consideration of morphological criteria helped to eliminate some false positives (88·7% of exclusivity). The best exclusivity (90·6%) was found for indirect ELISA. This is probably because this assay is based on a monoclonal antibody (MAb Xcd108), consistent with previous data (Robène-Soustrade et al., 2006). When compared to N-PCR with the restriction enzyme digestion step, the main and most significant difference in performance for indirect ELISA is largely its inability to distinguish between Anthurium (X. axonopodis pv. dieffenbachiae) and Syngonium (X. axonopodis pv. syngonii) strains, as both pathovars can be pathogenic to the latter plant species.

Generally, when considering both RDL and DL100% criteria, the serological methods showed lower sensitivity than that of the reference method or N-PCR, whatever the type of sample. The sensitivity levels of IF and ELISA, for detection of bacteria in plant material, are of about 103 CFU mL−1 and 105 to 106 CFU mL−1, respectively (Norman & Alvarez, 1994; López et al., 2003). In this respect, the DAS ELISA assay showed quite a high performance (RDL = 5 × 103 to 5 × 10CFU mL−1 for both RDL and DL100%). The difference in the RDL observed for IF on bacterial suspensions and plant extracts could probably be explained by the dilution needed to read the slides clearly, which lowers the RDL by the same factor. Moreover, the relative sensitivity scores obtained with IF and indirect ELISA showed that they tended to generate false negative responses (relative sensitivity of 90·2 and 86·3%, respectively).

The collaborative study showed that, unlike the reference method and the N-PCR, the results obtained with DAS ELISA and IF were mediocre overall and statistically different from the theoretically expected results. For DAS ELISA, the problem was largely linked to a lower relative sensitivity than the reference method or N-PCR. Furthermore, the concordance value for DAS ELISA was quite low (78·5%), highlighting that there was a significant difference in performance between laboratories (particularly for the L1 sample). The major limitation of the IF test was its very low concordance on positive samples (59·6% for the L1 and 62·5% for the L2 samples). The laboratories that were familiar with using IF tests for detecting bacteria produced excellent results, which were comparable to the results obtained with the reference method or N-PCR. However, the other laboratories either failed to produce data or obtained results with high rates of false negatives and also some false positives. In contrast with the reference method and N-PCR, both DAS ELISA and IF produced many indeterminate results.

Discrepancies between N-PCR and the serological methods results are therefore confirmed both by comparative and collaborative studies, with the N-PCR being superior to the serological tests in terms of sensitivity (enhanced by two rounds of PCR) and specificity (supported by two successive specific primer pairs).

In conclusion, the EN ISO 16140:2003 standard for evaluating methods provides very useful information on the methods' performance. The interlaboratory collaborative study is particularly important to ensure the reliability of the results obtained during the comparative study in the organizing laboratory. It enables the evaluation of accordance (i.e. repeatability) and concordance (i.e. reproducibility) of the results of a method, criteria that are rarely taken into account in the normal process used to evaluate a method despite their being essential to evaluate how transferable a method is among laboratories performing routine analyses. This study confirmed that N-PCR is an alternative method that is worth considering for the detection of X. axonopodis pv. dieffenbachiae, thanks to the high specificity of its results and a high sensitivity level. Additionally, it is much less labour-intensive and time-consuming than the reference method. The study also demonstrated the high accordance and concordance of the N-PCR results obtained within a laboratory and in different laboratories, respectively. The Bayesian approach commonly used in medical and veterinary sciences provides supplementary information and it can be applied to help choose the most appropriate detection scheme depending on the context. This approach is particularly useful for achieving better risk management and should be more widely used in plant pathology. Low prevalence populations (<5%, e.g. control imports) constitute the most common epidemiological situation. In this case, a negative result obtained with the reference method or with the N-PCR greatly minimizes the risk of releasing infected material, in comparison to a negative result obtained with the serological methods. For critical material (propagating plant material, plant material for exportation), this risk can be reduced further using both the reference method and the N-PCR. Nevertheless, for the same prevalence, the maximum probability that a sample testing positive is really infected is only 51·6% for the N-PCR and only from 27·2 to 43·4 for ELISA and IF. This result underlines the necessity for further confirmation of a positive result using an independent diagnostic technique based on different biological principles. As a result of this work, a revision of the X. axonopodis pv. dieffenbachiae diagnostic protocol has been proposed to the EPPO panel on diagnostics in bacteriology, where the N-PCR assay was included in the decision scheme as an alternative method to the biochemical and serological tests (PM 7/23(2)). This validation work could be transposed to the detection of many other plant pathogens. The use of a validated method is essential for guaranteeing the import/export of pathogen-free plant material. It is a prerequisite for the accreditation process of laboratories in plant pathology (Anonymous, 2005, 2010).

Acknowledgements

This work was supported by the Anses Plant Health Laboratory, the CIRAD and OSEO Innovation Réunion. The authors acknowledge the LMG international collection (Gent, Belgium) for allowing the distribution of the LMG695 strain to the participating laboratories. The authors acknowledge A. Saison, C. Boyer, L. Gagnevin, N. Cassam, A. Laurent, D. Linderme and F. Perefarres for their technical assistance in the organizing laboratory and the staff of the participating laboratories involved in this collaborative study, in particular C. D. Karafla and P. E. Glynos (Benaki Phytopathological Institute), S. Crump (Fera), B. De Paepe (ILVO), V. Spina and G. Serratore (ENSE), M. Odasso and E. Derin (IRF), C. Audusseau, S. Paillard and C. Rivoal (Anses Plant Health Laboratory, Angers). The authors acknowledge the SRAL Provence Alpes Côte d'Azur (France), SIVAP/DAVAR New Caledonia (France), AREU (Mauritius) and CTHA (Madagascar) for providing samples for the comparative study. Finally, the authors acknowledge the SRAL at Roissy, France (P. Perret and M. C. Legal) for its help in sending the samples for the interlaboratory test from Paris.

Ancillary