Laboratory studies on surface sampling of Bacillus anthracis contamination: summary, gaps and recommendations
Greg F. Piepel, Pacific Northwest National Laboratory, PO Box 999, Richland, WA 99352, USA. E-mail: email@example.com.
This article summarizes previous laboratory studies to characterize the performance of methods for collecting, storing/transporting, processing and analysing samples from surfaces contaminated by Bacillus anthracis or related surrogates. The focus is on plate culture and count estimates of surface contamination for swab, wipe and vacuum samples of porous and nonporous surfaces. Summaries of the previous studies and their results were assessed to identify gaps in information needed as inputs to calculate key parameters critical to risk management in biothreat incidents. One key parameter is the number of samples needed to make characterization or clearance decisions with specified statistical confidence. Other key parameters include the ability to calculate, following contamination incidents, the (i) estimates of B. anthracis contamination, as well as the bias and uncertainties in the estimates and (ii) confidence in characterization and clearance decisions for contaminated or decontaminated buildings. Gaps in knowledge and understanding identified during the summary of the studies are discussed. Additional work is needed to quantify (i) the false-negative rates of surface-sampling methods with lower concentrations on various surfaces and (ii) the effects on performance characteristics of: aerosol vs liquid deposition of spores, using surrogates instead of B. anthracis, real-world vs laboratory conditions and storage and transportation conditions. Recommendations are given for future evaluations of data from existing studies and possible new studies.
Provide feedback or get help You are viewing our new enhanced HTML article.
If you can't find a tool you're looking for, please click the link at the top of the page to "Go to old article view". Alternatively, view our Knowledge Base articles for additional help. Your feedback is important to us, so please let us know if you have comments or ideas for improvement.
In 2001, letters containing Bacillus anthracis (BA) contaminated areas of government office buildings in Washington, DC and postal facilities that processed the letters. A congressional inquiry and Government Accountability Office (GAO) investigation (Government Accountability Office 2005a,b) identified two main concerns regarding the methods used to characterize and clear buildings. One main concern was the reliance on sampling specific locations where it was thought BA would be found. This type of sampling approach is referred to as judgmental (or targeted) sampling. The GAO reports identified the need to use statistically based sampling so that when all results are negative, a building (or area within a building) can be cleared with a specified level of statistical confidence. The second main concern was that methods used in the steps of the sampling and analysis process (SAP) (i.e. sample collection, storage/transportation, processing/extraction and analytical) were not validated. The lack of validated methods raised questions about the reliability of the negative sampling results obtained.
Several organizations are working to address the GAO concerns, including the U.S. Department of Homeland Security (DHS), U.S. Environmental Protection Agency (EPA), Centers for Disease Control and Prevention (CDC), National Institute of Standards and Technology (NIST), U.S. Department of Defense (DoD), Federal Bureau of Investigation (FBI), Pacific Northwest National Laboratory (PNNL) and Sandia National Laboratories (SNL). Some key activities to address the GAO's main concerns are the following:
- Developing a sampling guidance document that describes appropriate uses of judgmental and statistical sampling approaches
- Validating methods associated with the SAP
- Developing formulas to estimate the BA contamination, quantify the bias and uncertainty of the estimate, calculate the confidence in characterization and clearance decisions, and calculate the number of samples necessary to achieve the desired uncertainty and confidence in such decisions.
For conciseness in subsequent discussions, these are referred to as Activities A, B and C.
Activities B and C require quantitative information on the performance of sampling and analysis methods for indoor surface samples. Such information can be obtained from laboratory and field studies, although laboratory studies are best suited for certain performance measures because the level of contamination can be better controlled. Hence, we identified 20 previous laboratory studies that were conducted to assess the performance of swab, wipe, and vacuum sample collection methods and subsequent sample storage/transportation, processing/extraction and analysis methods for surfaces contaminated by BA or surrogates. We summarized the test conditions and results of the studies and assessed whether there were any substantive gaps. The summary information, gaps and needs that were found, and recommendations made to address the gaps and needs, are presented and discussed in this article. Edmonds (2009) also reviewed the literature and discusses difficulties, needs and perspectives for future work. Later, in this article, we compare our results with those of Edmonds (2009).
The next section discusses how Activities B and C motivated the work discussed in this article. The following section identifies the previous laboratory studies that were considered in this effort and summarizes their characteristics and results. The final section discusses the gaps and needs that were identified and makes several recommendations for future studies and evaluations.
Motivation for summarizing and assessing the gaps of previous laboratory studies with Bacillus spores
The motivation for summarizing the performance characteristics of swab, wipe and vacuum surface-sampling methods was to use the results as inputs to Activities A, B and C. Activities A and C involve developing sampling strategies, approaches and formulas to provide high confidence in the results of characterizing building contamination and clearing an uncontaminated or decontaminated building for reoccupation. Statistical and combined judgmental and statistical (Sego et al. 2010) sampling approaches provide for calculating the number of samples necessary to achieve the desired confidence for detecting contamination and clearing uncontaminated or decontaminated buildings. These approaches also provide for calculating the confidence in characterization or clearance decisions based on sampling results following contamination incidents. Well-quantified performance characteristics are required to select sampling approaches and methods, as well as perform the calculations mentioned.
Activity B involves validating methods in the SAP for BA contamination in buildings. Validating a method consists of several actions (ISO/IEC 2005), two of which include (i) quantifying the uncertainty (repeatability and reproducibility) and accuracy of results obtained by the method and (ii) quantifying relevant performance metrics, such as recovery efficiency (RE), false-negative rate (FNR), false-positive rate (FPR) and limit of detection (LOD). Hence, it is important to understand how much of this type of information exists for the previous laboratory studies that investigated sampling of surfaces contaminated with BA or surrogates. This article summarizes estimates of the above performance metrics and corresponding uncertainties reported in the references that document the studies.
The FNR plays an important role in Activity C, because a higher FNR will reduce the confidence in contamination detection and clearance decisions, which can be offset by taking more samples. Further, the overall RE of a sampling process contributes to determining the overall FNR. Here, ‘overall’ means over all steps in the sampling process: sample collection, storage/transportation, processing/extraction and analysis (e.g. culturing and counting). Having sufficient experimental data to determine the overall RE and FNR (and their uncertainties) for swab, wipe and vacuum samples on various surfaces at various levels of contamination (and any other influencing factors) is important in assessing risks (of failing to detect spores or erroneously clearing an area) and in developing sampling strategies and approaches.
Standard statistical formulas assume that the overall FNR = 0 when calculating (i) the number of samples required to achieve the desired confidence for a characterization or clearance sampling goal and (ii) the uncertainty and confidence associated with a characterization or clearance decision using a specific sampling approach implemented following a contamination incident. When FNR = 0, the formulas account only for the uncertainty in results associated with the specific type of statistical or hybrid sampling approach being used. However, the overall FNR is affected by the concentration of the contaminant and anything in the sampling process that might yield a false negative, including (i) the RE of a sampling method (e.g. swab, wipe or vacuum), (ii) the RE of storage or transportation steps, if applicable, (iii) the RE of the processing/extraction step (i.e. extracting the contaminant from the sample medium) and (iv) the uncertainty of the analytical/detection method and equipment. Hence, it is important in laboratory studies to quantify the REs and uncertainties affecting the results of a method at each step in the process, so that the overall RE and overall FNR can be determined.
Standard statistical formulas for calculating (i) and (ii) in the previous paragraph can be extended to address situations in which the overall FNR > 0. The Visual Sample Plan (VSP) software (VSP Development Team 2010) implements many statistical- and nonstatistical-sampling approaches and methods for environmental sampling. VSP generally implements standard formulas to calculate the numbers of samples to address situations in which the overall FNR = 0. Extended formulas (applicable when the overall FNR > 0) have only been developed for a statistical, grid-sampling approach for detecting contaminated areas of specified size (so-called hotspot sampling). PNNL plans to develop extended formulas for calculations (i) and (ii) when FNR > 0 with other statistical-sampling approaches recommended in a multiagency sampling strategy document (not yet completed). Adequate information on the FNR performance of sampling and analysis methods is needed as input to this work.
Previous study data indicate the overall RE and FNR may depend on the surface concentration of BA contamination, how the contamination is deposited on a surface, the surface material, the specifics of a sampling method and possibly other factors (references cited in the first paragraph of the following section). Hence, laboratory studies should investigate the performance of the SAP for the most commonly used sampling methods over a range of BA (or surrogate) surface concentrations, different deposition methods, different surface materials, etc. In fact, several such studies have been performed, but important questions include whether the studies adequately (i) cover the combinations of factors that may affect results and (ii) quantify performance measures of the methods and conditions studied.
Summary of anthrax-related laboratory studies
Previous laboratory studies were identified that used (i) swab, wipe and vacuum-sampling methods to collect BA (or surrogate) contamination from various porous and nonporous surfaces and (ii) culture and count methods to quantify the contamination. Only BA/surrogate contamination and count methods were considered to keep the scope of the effort manageable. We focused on laboratory studies (as opposed to field studies or data from BA contamination incidents) because they provide a basis for quantifying actual contamination levels, and using those values to estimate REs and other method performance metrics such as FNR. The 20 studies included in our summary and gap assessment are: Buttner et al. (2001, 2004a,b), Rose et al. (2004), Hodges et al. (2006), Nellen et al. (2006), Brown et al. (2007a,b,c), Quizon et al. (2007), Almeida et al. (2008), Frawley et al. (2008), Montgomery and Camp (2008), Valentine et al. (2008), Edmonds et al. (2009), Estill et al. (2009), Hodges et al. (2010), Einfeld et al. (2011), Krauter et al. (2012), and Rose et al. (2011).
The characteristics and results of relevant tests in each study are summarized in four tables, each with ‘a’ and ‘b’ parts. The ‘a’ part of each table summarizes the characteristics of the studies, while the ‘b’ part summarizes the results of the studies. The ‘b’ parts of tables are further divided into three categories of information: (i) Recovery concentration results – Mean and %RSDs (ii) Recovery efficiency (RE) – Mean and %RSDs and (iii) LOD, FNR and FPR. All per cent relative standard deviation (%RSD) values are rounded to zero decimal places. Table 1 describes the study characteristics and results in the columns of the summary tables. In each table, different studies are represented by groups of rows, with rows corresponding to the tests performed within a study. Because the four tables that summarize groups of tests and results are long, they are not included in this article. They are included in the Appendix of Piepel et al. (2011) as Tables A.1 (swab sampling), A.2 (wipe sampling), A.3 (vacuum sampling) and A.4 (one storage/transportation study that did not involve surface sampling). Condensed summaries of Tables A.1 to A.4 are presented in this article as Tables 2–5, with ‘a’ and ‘b’ parts as described previously. Acronyms and abbreviations used in Tables 2–5 are defined in Table 6.
Table 1. Descriptions of columns in tables summarizing the characteristics and results of laboratory studies on surface sampling of Bacillus anthracis and surrogates
|Reference||Lead author (Year) citation of publication that documents the study|
|Test no.||A number of the form x.y where x is 1, 2, 3, … for each study, and y = 1, 2, 3, … for the tests within a study|
|Agent||B. anthracis or related surrogate|
|Agent deposition||Method used to deposit agent on test material|
|Agent concentration||Concentration or amount of agent on surface|
|Swab/Wipe/Vacuum type||Type or material of sample collection medium|
|Wetting agent||Swab or Wipe: Liquid, if any, used to wet or premoisten the sampling material|
|Vacuum technique||Vacuum: Technique in vacuuming (e.g. pattern and pressure applied)|
|Relative humidity||Per cent relative humidity in chamber or laboratory during testing|
|Surface type||Material type of surface sampled|
|Surface area sampled||Area of surface sampled|
|Extraction liquid||Liquid used to extract spores from the sample|
|Extraction method||Method used to prepare a sample and extract the contaminant|
|Culture method/medium||Method/medium used to culture samples|
|No. labs||The number of laboratories (labs) that participated in a study|
|No. test runs||The number of test runs (set up and performed separately at different times)|
|Total no. test samples||Total number of samples tested (i.e. over labs, runs and samples within runs)|
|Study results (in Tables 2b, 3b, 4b and 5b)|
|Recovery Concentration Results – Mean and %RSDs|
|Reference||Same as for ‘Study characteristics’ tables|
|Test no.||Same as for ‘Study characteristics’ tables|
|Mean (CFU cm−2)||Mean surface concentration recovered|
|Lab %RSD||Lab-to-lab per cent relative standard deviation, which includes the variation from preparing the samples, extraction and analysis.|
|Run %RSD||Per cent relative standard deviation from replicate runs of a test performed at different times.|
|Sample-within-run %RSD||Per cent relative standard deviation from replicate tests performed at the same time (in one run)|
|Total %RSD||Total per cent relative standard deviation (including Lab, Run and Sample-within-run)|
|Recovery efficiency (RE) – Mean and %RSDs|
|RE mean (%)||Mean recovery efficiency|
|RE lab %RSD||Lab-to-lab per cent relative standard deviation of RE|
|RE run %RSD||Run per cent relative standard deviation of RE|
|RE sample-within-run %RSD||Sample-within-run per cent relative standard deviation of RE|
|RE total %RSD||Total per cent relative standard deviation of RE (including Lab, Run, and Sample-within-run)|
|LOD, FNR, and FPR|
|Positive result||How a positive result (detection) was defined (e.g. CFU ≥ 1)|
|LOD definition||How the limit of detection (LOD) is defined|
|LOD||Value of the limit of detection|
|LOD SD or 95% CI (CFU cm−2)||Standard deviation or 95% confidence interval for the LOD|
|FNR||False-negative rate (FNR) based on controlled tests where the sampled surface was known to be contaminated but yielded a negative result|
|FPR||False-positive rate (FPR) based on controlled tests where the sampled surface was known to be uncontaminated but yielded a positive result|
In reading the publications that document the studies summarized in this article, we noticed many differences in the way tests and calculations were performed and reported. The main differences are documented in Tables 2–5 (and Tables A.1 to A.4 of Piepel et al. 2011), but some differences were too detailed to document. A common difference was in the units used to report results. Whenever possible, we performed calculations using information in the publications to convert results reported in a given set of units to the common set of units used in the ‘b’ parts of Tables 2–5 (and Tables A.1 to A.4). There were also experimental and calculational differences in the ways studies used positive controls to obtain the ‘actual’ contamination values used as the denominator in calculating RE. For example, some studies had positive controls colocated with every sample, which allows RE values to be calculated using the colocated sample and positive control results. This way of calculating RE values can correct for inadvertent differences in contaminant deposition or other aspects of testing. In some studies without colocated positive controls, the average result of positive controls was used as the denominator to calculate RE values. This can inflate the uncertainty of the RE values because of any inadvertent differences.
Table 7 documents key study characteristics and results from Tables 2–5. Specifically, it summarizes the numbers of the 20 laboratory studies that varied several study factors and reported various performance metrics. Table 7 shows that 13, 12 and 5 studies investigated swab, wipe and vacuum sampling, respectively. Some studies investigated more than one type of sampling. Only Almeida et al. (2008) investigated storage/transportation effects on sampling results. An additional storage/transportation study has been completed, but the results have only been partially released in two poster presentations (O'Connell et al. 2010; Perry et al. 2010). The information in those references was not complete enough to include the study in this article.
The ‘zero’ entries in Table 7 denote combinations of study characteristics/results that were not addressed by the 20 studies. Table 7 shows that few of the 20 studies had results available from varying the factors: (i) agent (contaminant), (ii) agent deposition method, (iii) sample collection medium, (iv) wetting agent for the sampling medium (swab and wipe), (v) storage/transport conditions and (vi) processing/extraction method. Laboratory-to-laboratory uncertainties of REs were reported only by Estill et al. (2009) for swabs, wipes and vacuum samples; Hodges et al. (2010) for swab samples; and Rose et al. (2011) for wipe samples. Run-to-run uncertainties were quantified by Estill et al. (2009) for swab, wipe and vacuum samples; Edmonds et al. (2009) and Hodges et al. (2006) for swab samples; and Montgomery and Camp (2008) for vacuum samples. Even when uncertainties reported in some studies include laboratory-to-laboratory and/or run-to-run uncertainties, the numbers of laboratories and runs were typically small, indicating these sources of uncertainty are poorly estimated. Laboratory-to-laboratory and run-to-run uncertainties are expected to be substantial contributors to uncertainty and were not quantified in most of the studies. Hence, the uncertainties reported by many studies can be expected to underestimate the total uncertainties in surface-sampling results. Also, the uncertainties may be underestimated because of smaller uncertainties when applying sampling and analysis methods in controlled laboratory environments than may occur in actual contamination incidents.
LOD and FNR/FPR results are rarely reported in the laboratory study results, as shown in Table 7. As discussed previously, the FNR and RE (each of which may be a function of contaminant concentration and other factors) play a key role in the framework for determining the number of samples necessary to achieve a characterization or clearance goal with a specified confidence. Hence, it is important to experimentally quantify how RE and FNR depend on contaminant surface concentration and other sampling and environmental factors. Table 7 shows that only a few of the 20 studies varied the contaminant at different surface concentrations. Most of these studies investigated three or fewer concentrations, although Hodges et al. (2006) investigated six and Krauter et al. (2012) investigated eight. Montgomery and Camp (2008) [vacuum results], Estill et al. (2009), Hodges et al. (2010), Rose et al. (2011) and Krauter et al. (2012) found that RE did not depend on surface concentration for the range of concentrations tested. However, Hodges et al. (2006) and Edmonds et al. (2009) reported that RE increased as concentration increased. Of the studies that investigated different surface concentrations, only Krauter et al. (2012) presented equations for FNR as a function of surface concentration (for each of six surface materials). Estill et al. (2009) mentioned using probit regression to develop a probability of detection (equivalent to FNR) curve as a function of contaminant concentration, but did not present the results. The lack of FNR data (and of FNR and RE curves as functions of surface concentration, surface material and other influencing factors) is a significant gap in all studies except Krauter et al. (2012).
The studies summarized in Tables 2–5 have large ranges of mean RE values and sample-within-run uncertainty (%RSDRE) values. The following ranges of RE and %RSDRE exclude data from direct-inoculation tests in which surfaces were not sampled with a swab, wipe or vacuum. For swabs, RE ranged from 0 to 92·7% and %RSDRE ranged from 6 to 550. For wipes, RE ranged from 1 to 97% (excluding a questionable 120% value in the Estill et al. 2009 study) and %RSDRE ranged from 0 to 316. For vacuums, RE ranged from 0·02 to 36% and %RSDRE ranged from 10 to 130. These ranges of %RSDRE values represent the uncertainty in RE results from replicate tests performed at the same time, generally by the same samplers and the same laboratory personnel (because that is what most studies reported). Hence, these %RSDRE ranges do not include all relevant sources of variation and can be expected to underestimate the total uncertainty.
The wide ranges of RE values summarized earlier are a result of the effects of several factors varied within and across the studies (e.g. contaminant surface concentration; contaminant deposition method; surface material being sampled; materials and specifics of swabs, wipes, and vacuum socks; wetting agent for swabs and wipes; specifics of sample preparation and extraction methods and counting method) as shown in Tables 2 (a)–5 (a). Krauter et al. (2012) also indicate that FNR values can be affected by such factors. Hence, the dependence of RE, FNR, LOD, and their total uncertainties (including all sources of uncertainty in the SAP) on such factors needs to be quantified. Ideally, all data from each of the swab, wipe, and vacuum studies would be combined and a statistical analysis performed to estimate the effects of the quantitative and qualitative test factors on RE, FNR and LOD. However, FNR and LOD were not reported in enough studies to do this. Even for the widely reported RE, the number of test factors is large with test factors sometimes having many possible values/options. Also, there may have been interactive effects on performance metrics between some test factors in the studies. Although it was beyond the scope of this work to attempt such a statistical analysis of the combined data, our initial assessment is that there are not enough data for this exercise to be successful given the large number of factor combinations and possible interactions in the swab, wipe and vacuum studies.
Table 2 (a). Summary of test conditions for swab-sampling studies. Acronyms and abbreviations are defined in Table 6 and the footnotes
|Estill 2009||6||BA-S||DA||3||0·03–2||MF||BBT||SS, Cpt||103||BBT||V+S||FP, TSAB||3||2–4||24–36|
|Hodges 2006||6||BA-S||LD||6||0·4–6000||MF||PBST||SS||10||PBST||V||Plate/FP, TSAB||2a||NR||15–45|
|Rose 2004||24||BA-S||LD||1||1937·5||Ct, P, R, MF||Dry, PBST||SS||25·8||PBST||U, V, S||Plate, TSAB||1||NR||10|
|4||BA-S||LI||1||1E+4 CFU/swab||DI: Ct, PE, R, MF||NA||None||NA||PBST||V||Plate, TSAB||1||NR||10|
|Frawley 2008||6||BA-S||LD||1||1E+2–1E+5 CFU/sample||PE||Dry, PBST/Tr||P, W, Cl||1, NR||PBST||V||Plate, SBA||1||NR||12–24|
|7||BA (4)||LD||1||50 CFU/sample||Ct||PBST/Tr||P, G, F, M, Cpt, B, Ct Cloth||NR||NR||NR||Plate, NR||1||NR||11|
|Brown 2007b||4||BAtr||DA||2||102–103, 105–106||R||DW||SS||25||BBT||S, H, V||Plate, PF||1||NR||20|
|1||BAtr||LI||1||1E+6 CFU/swab||DI: R||NA||None||NA||BBT||S, H, V||Plate, PF||1||NR||20|
|1||BAtr||LD||1||2·0E+5||NA||NA||DI: SS||6·25||BBT||S, H||Plate, BHIA||1||NR||24|
|Edmonds 2009||16||BAtr||LD||1||1·0E+5||Ct, DP, R, MF||DW||G||10||PBSTr||V, S||SprP, NR||1||3–4||28–40|
|16||BAtr||DA||1||1·0E+9||Ct, DP, R, MF||DW||G, SS, PC, VL||10||PBSTr||V, S||SprP, NR||1||3||24–30|
|4||BAtr||LD||4||4·77E+3–2·52E+6||DP||DW||G||10||PBSTr||V, S||SprP, NR||1||3||30|
|Valentine 2008||16||BS||LD||2||9·03E+4, 2·82E+5||Ct, PF, PE, DP||PBST||P, O, PEUF, Cpt||104·04||PBST||V||Plate, TSA||1||10||10|
|4||BS||LD||2||9·03E+4, 2·82E+5||Ct, PF, PE, DP||PBST||Monitor||25||PBST||V||Plate, TSA||1||10||10|
|Buttner 2001||2||BS||LD||1||1·48E+6||SK||PBST||G||5||PBT||V||Plate, TSAC||1||3||3|
|8||BS||DA||1||100–1000||SK, Ct||PBST||VL, Cpt(3)||32·49||PBT||V||Plate, TSAC||1||3||3|
|8||BS+PC2||DA||1||100–1000||SK||PBST||VL, Cpt(3)||32·49||PBT||V||Plate, TSAC||1||2||2|
|Buttner 2004a||6||BAtr||DA||1||107·6–1076·4||SSPK||NR||VL, W, M||929||NR||HM||Plate, TSA||1||3||3|
|3||BAtr||DA||1||107·6–1076·4||SSPK||NR||VL, W, M||929||NR||HM||Plate, TSA||1||1||2|
|Buttner 2004b||4||BAtr||DA||1||NR||SSPK, Ct||NR||M, W||100, 317||NR||HM, Shake||Plate, TSA||1||NR||4, 8|
|Quizon 2007||4||BAtr||WA||1||NR||PEUF||PBST||PWB, SS, VL, W||100||PBST||V, S||SP/FP, TSA||1||4||4, 10|
|Nellen 2006||2||BS-168||LD||1||1000 CFU/swab||DI: Ct, R||NA||NA||NA||PBS||Untreated||Plate, TSA||1||3||15|
|6||BS-168||LD||1||1000 CFU/swab||DI: Ct, R||NA||NA||NA||PBS||V, S, V+S||Plate, TSA||1||3||15|
|4||BS-168||LD||1||4||Ct, R||DW||PD||25||PBS||NR, H+V+S||Plate, TSA||1||3||15|
|10||BS-168||LD||1||16||R||DW||b||25||PBS||NR, H+V+S||Plate, TSA||1||3||3|
|Hodges 2010||6||BA-S||LD||3||1·4, 15·3, 1607·2||MF||PBST, PBST+||SS||26||PBST||V||FP, TSAB||12||1||118–120|
|6||BA-S||LI||3||36–33300 CFU/swab||DI: MF||NA, PBST+||NA||NA||PBST||V||FP, TSAB||12||1||24–48|
Table 2 (b). Summary of test results for swab sampling studies. Acronyms and abbreviations are defined in Table 6
|Estill 2009||6||0·0012–0·18||0–110||0–98||57–460||73–473||3·4–14·0||0–110||0–110||76–550||81–560||≥1||LOD95||0·4 (SS) 1·9 (Cpt)||NR||0·037|
|Hodges 2006||6||0·1–2900||NR||NR||27–100||NR||31·7–49·1||NA||NA, 28–33||21–91||NA||≥1||LOD90||1·2||0–0·27||NR|
Table 3 (a). Summary of test conditions for wipe sampling studies. Acronyms and abbreviations are defined in Table 6 and the footnotes
|Estill 2009||3||BA-S||DA||3||0·03–2||Sponge||BBT||SS||929||BBT||A+C+V+S||FP, TSAB||3||3||27|
|Brown 2007a||4||BA||DA||2||102–103, 104–105||PR gauze||DW||SS, PWB||25||BBT||S+H+V||Plate, PF||1||NR||20|
|1||BA||LI||1||NR||DI: PR gauze||DW||None||NA||BBT||S+H+V||Plate, PF||1||NR||40|
|1||BA||LD||1||2·0E+5||NA||NA||DI: SS||25||BBT||S+H||Plate, BHIA||1||NR||24|
|Buttner 2001||1||BS||LD||1||1·48E+7||Sponge||PBST||Glass||32·49||PBST||HS||Plate, TSAC||1||3||3|
|12||BS,BS+PC2, BS+PC4a||DA||1||1E+2–1E+3||Sponge||PBST||VL, Cpt-R, Cpt-S, Cpt-C||32·49||PBST||HS||Plate, TSAC||1||2, 3||2, 3|
|Valentine 2008||12||BS||LD||1||90349·9||Ct, HCW,PR||PBST||P, O, PEUF, Cpt||104·04||PBST||V+C||Plate, TSAC||1||10||10|
|3||BS||LD||1||282,000||Ct, HCW,PR||PBST||Monitor||25||PBST||V+C||Plate, TSAC||1||10||10|
|Buttner 2004a||12||BAtr||DA||1||107·6–1076·4||Swipe, HW||PBST||VL, W, M||929||NR||HM||Plate, TSA||1||3||3|
|6||BAtr||DA||1||107·6–1076·4||Swipe, HW||PBST||VL W, M||929||NR||HM||Plate, TSA||1||1||1, 2|
|Buttner 2004b||4||BAtr||LI||2||10, 18·2||BiSKit||Dry, PBST||M||10 000||PBST||BiSKit||Plate, TSA||1||NR||4, 8|
|6||BAtr||DA||1||NR||BiSKit||Dry, PBST, NR||M, W||10 000||PBST||BiSKit||Plate, TSA||1||NR||4, 8|
|Frawley 2008||6||BA-S||LD||1||1E+2–1E+5/sample||Gauze||Dry, PBST/Tr||P, W, Ct, Cloth||1, NR||PBST||V||Plate, SBA||1||NR||12, 24|
|Einfeld 2011||10||BAtr, BAtr+Grime||DA||1||9,55E+0–6·81E+4||PR||DW, RH = 10-15, 82-90||SS, G, Marble||100||BBT||S+H+V||Plate, PF||1||2, 3, 5||24–60|
|Quizon 2007||4||BAtr||WA||1||NR||PR||PBST||PWB, SS, VL, W||900||PBST||C+V+S||SP/FP, TSA||1||4||4, 10|
|Krauter 2012||27||BAtr||LD||8||0·0025–1·8538||Sponge stick||NB||SS, CerT, VL||645·16||PBST||St+C+V+S||b||1||1||9, 10|
|27||BAtr||LD||8||0·0078–0·1537||Sponge stick||NB||FL, PW, PLCP||645·16||PBST||St+C+V+S||b||1||1||9, 10|
|Rose 2011||3||BA-S||LS||3||0·0135–17·123||DI: Sp. stick||NB+ATD||SS||645·16||PBST||St+C||SP/FP, TSAB||9||1||17 –18|
|3||BA-S||LS||3||0·0405–51·37||Sponge stick||NB+ATD||SS||645·16||PBST||St+C||SP/FP, TSAB||9||1||54–63|
|1||BA-S||LS||1||15·5||Rayon gauze||PBST||SS||645·16||PBST||St+C||SP/FP, TSAB||1||1||10|
|1||BA-S||LS||1||15·5||Sponge wipe||DE broth||SS||645·16||PBST||St+C||SP/FP, TSAB||1||1||10|
|1||BA-S||LS||1||15·5||Sponge wipe||BB||SS||645·16||PBST||St+C||SP/FP, TSAB||1||1||10|
|1||BA-S||LS||1||15·5||Sponge stick||PBST||SS||645·16||PBST||St+C||SP/FP, TSAB||1||1||10|
|3||BA-S||LS||3||0·0155, 0·155, 15·5||PEF sponge||PBST||SS||645·16||PBST||St+C||SP/FP, TSAB||1||3||15|
|Montgomery 2008||4||BA-S||EWD||1||NR||PR gauze, pad 2 wipe tech's||PBSTr||HVAC filter||100||PBSTr||Shake+C||Plate, NR||1||2, 3||2–5|
Table 3 (b). Summary of test conditions for wipe sampling studies. Acronyms and abbreviations are defined in Table 6 and the footnotes
|Brown 2007a||4||NR||NA||NR||NR||NA||25·2–39·2||NA||NR||32–59||NA||≥1CFU ml−1||NR||3·6–4·2||NR||NR|
|Buttner 2004b||4||2·05–3·35||NA||NR||NR||NA||11·3–18·4||NA||NR||NR||NA||≥1 CFU ml−1||NR||0·0042–0·01||NR||NR|
|Quizon 2007||4||6670–14615||NA||8–20a||a||NA||39·5, NR||NA||NR||NR||NA||NR||NR||NR||NR||NR|
|Krauter 2012||27||0·0003–0·994||NA||NR||6·6–316||NA||12·5–75·5||NA||NR||6·6–316||NA||≥1||LOD95, LOD90||0·013–0·038||0–0·933||0·0|
Table 4 (a). Summary of test conditions for vacuum-sampling studies. Acronyms and abbreviations are defined in Table 6 and the footnote
|Estill 2009||3||BA-S||DA||3||0·03–2||HEPA sock||P2D||NR||SS||929||BBT||A+C+V+S||FP, TSAB||1||3||27|
|3||BA-S||DA||3||0·03–2||HEPA sock||P2D||NR||Cpt||929||BBT||A+C+V+S||FP, TSAB||1||2–4||18–36|
|Brown 2007c||8||BA||DA||2||1E+2–1E+3 1E+4–1E+5||HEPA filter||PP2D||NR||SS, PWB, Cpt, Concrete||100||BBT||S+H+V||Plate, PF||1||NR||13–24|
|Einfeld 2011||6||BAtr, BAtr +grime||DA||1||1·31E+3–8·78E+4||PE filter||NR||36–48, 77–90||Marble, Concrete||100||BBT||S+H+V||Plate, PF||1||3–5||33–59|
|Quizon 2007||6||BAtr||WA||1||NR||HEPA sock||SVH||NR||CelT, PWB, SS, VL, W||900||PBST||C+V+S||SP/FP, TSA||1||4||4–10|
|Montgomery 2008||1||BA-S||EWD||1||7·98E+2||HEPA sock||R||NR||HVAC filter||309·68||PBSTr||Shake+C||NR, NR||1||3||NR|
|4||BA-S||EWD||3||7·98E+2, 9·0E+4, 9·1E+6||TECF||G, R||NR||HVAC filter||400||PBSTr||Shake+C||NR, NR||1||3||9, NR|
|3||BA-S||EWD||2||167, 201||TECF||VS, TD||NR||Cpt||400||PBSTr||Shake+C||NR, NR||1||1||3, 6|
|2||BA-S||WD||1||111||TECF||VS, TP||NR||Cpt||400||PBSTr||Shake+C||NR, NR||1||1||3|
Table 4 (b). Summary of test results for vacuum sampling studies. Acronyms and abbreviations are defined in Table 6 and the footnote
|Brown 2007c||8||NR||NA||NR||NR||NA||16·4–36·1||NA||NR||28–90||NA||≥1 CFU ml−1||NR||NR||NR||NR|
|Quizon 2007||6||890–5200||NA||9–20a||a||NA||4·4, NR||NA||NR||NR||NA||NR||NR||NR||NR||NR|
|2||NR||NA||NR||NR||NA||2·26, 1·69||NA||NA||18, 32||NA||NR||NR||NR||NR||NR|
Table 5 (a). Summary of test conditions for storage and stability tests. Acronyms and abbreviations are defined in Table 6
|Almeida 2008||1||BA Sterne||Liquid||NR||NA||NA||NR||NA||None||4||0||NA||NA||Plate, LBA||NR||3 lots||19|
|15||BA Sterne||Liquid||NR||NA||NA||NR||NA||None, Phenol, EDTA. Ethanol, PBSTr||4||0, 182, 279||NA||NA||Plate, LBA||NR||1||3|
|4||BA Sterne||Liquid||NR||NA||NA||NR||NA||None||-20, -80||182, 279||NA||NA||Plate, LBA||NR||1||3|
Gaps, needs and recommendations
This section discusses several gaps and needs identified based on the summaries of laboratory studies of swab, wipe and vacuum-sampling and analysis methods in Tables 2–5. Recommendations are made for additional laboratory studies and other evaluations to address these gaps and needs. In what follows, ‘performance results’ refers to RE, FNR, LOD and their uncertainties, considering uncertainty contributions from all steps of the SAP.
Realistic vs laboratory conditions
Price et al. (2009) emphasize the importance of quantifying the performance of methods (sample collection, storage/transportation, processing/extraction and analysis) under realistic conditions and not just highly sanitized laboratory conditions. Some studies investigated other material added with the BA or surrogate spores: Penicillium chrysogenum organism (Buttner et al. 2001), grime (Einfeld et al. 2011) and silicon dioxide (Brown et al. 2007a,b,c; Estill et al. 2009). The latter studies did not perform tests without silicon dioxide added, so no conclusion can be made about its effect. Buttner et al. (2001) and Einfeld et al. (2011) found that REs were the same or better with the other materials added to the Bacillus contaminants they investigated. Data from field studies (e.g. Amidan et al. 2007; Piepel et al. 2009) and the 2001 BA letter incident that involve realistic conditions are available, but do not provide for accurately assessing RE and FNR in the way that laboratory studies do. We recommend that more work be performed to supplement the studies of Buttner et al. (2001) and Einfeld et al. (2011) and quantify the effects on method performance results of surfaces contaminated with other material and/or biological organisms in addition to BA. Ideally, such studies should be performed with three levels of each other organism/material (none, low and high) to better assess the effects of such factors on sampling performance results.
Bacillus anthracis vs surrogates
Previous laboratory studies have used BA or one of several surrogates, but never investigated BA and one or more surrogates in the same study (see Tables 2 (a)–5 (a)). The number of test variables whose values differ among studies is large enough that the existing studies do not provide for quantifying the relationships between performance results of swab, wipe and vacuum methods with BA and with its surrogates. Hence, we recommend that more work be performed to establish the relationships between sampling performance results for surrogates and BA. One way to do that would be to use BA and key surrogates in a new laboratory study to establish these relationships for various combinations of sampling conditions (i.e. various concentrations and surface materials) for a each set of sampling methods (sample collection device, storage and transportation, extraction and analysis). Another option would be to measure, for BA and its surrogates, organism properties relevant to sampling and resuspension. The data could then be used to establish the relationships for sampling and analysis performance results between BA and its surrogates.
Previous laboratory studies generally used one of two contaminant deposition methods: (i) contaminant in liquid (water, ethanol or a mixture of the two) that was allowed to dry after deposition by wet aerosol or liquid drops and (ii) dry aerosol. Only one swab study (Edmonds et al. 2009) compared REs for different deposition methods. No wipe or vacuum studies varied the deposition method. Edmonds et al. (2009) found that for high contaminant concentrations, REs differed significantly depending on liquid vs dry deposition, with the difference depending on the swab-sampling method. We recommend that more work be performed to quantify the difference in performance results of swab, wipe and vacuum-sampling and analysis methods as a function of the contaminant deposition method, as well as functions of other influencing factors (e.g. contaminant concentration).
A wide variety of surface materials were investigated in the swab, wipe and vacuum-sampling studies summarized in Tables 2–4. However, some studies investigated only stainless steel or a limited number of surface materials. Method performance results were generally reported separately for each surface material, without any attempt to develop relationships between performance results and characteristics of the surface materials. The exception was Krauter et al. (2012), who compared mean REs, FNR and LOD to the roughness index of the surfaces investigated. We recommend using existing data where possible, or performing new studies to develop equations that relate method performance to one or more surface-characteristic variables (as well as any other influencing factors). Such equations can then be used to estimate method performance for various surface materials without additional experimental work.
Table 5 (b). Summary of results for storage and stability tests. Acronyms and abbreviations are defined in Table 6
Table 6. List of acronyms and abbreviations
|A||Agitation (during processing/extraction)|
|ATD||Arizona test dust|
|BA (4)||Four strains of Bacillus anthracis|
|BA-S||Bacillus anthracis Sterne|
|BAtr + grime||Bacillus atrophaeus + standard grime (Arizona test dust + diesel carbon + oil + biologicals)|
|BBT||Butterfield buffer with Tween|
|BHIA||Brain heart infusion agar|
|BiSKit™||Biological Sampling Kit|
|BS + PC2, BS + PC4||Bacillus subtilis with 102 or 104 CFU cm−2 P. chrysogenum background contamination|
|C||Centrifuging (during processing/extraction)|
|CFU||Colony forming unit|
|Cpt-C||Carpet, commercial loop|
|Cpt-R||Carpet, residential cut-pile|
|Cpt-S||Carpet, soiled residential cut-pile|
|Cpt(3)||Cpt-C, Cpt-R, Cpt-S|
|Ct cloth||Cotton cloth|
|DE broth||Dey Engley broth|
|DW||De-ionized, distilled, or sterile water|
|EWD||Drops of agent in 50% ethanol, 50% water|
|EtOH||Ethyl alcohol, enthanol|
|H||Heat treatment (extraction/processing)|
|HCW||HS II cleanroom wiper|
|HEPA||High-efficiency particulate air|
|HVAC||Heating, ventilation and air conditioning|
|LBA||Luria broth agar|
|LOD||Limit of detection|
|LOD90, LOD95||Lowest concentration with 90 or 95% probability of detection|
|MLI foil||Multilayer insulation foil (aluminized Kapton)|
|NB+ATD||Sponge wipe premoistened with neutralizing buffer, wipe samples collected, then inoculated with Arizona Test Dust|
|P2D||Pulled one direction and perpendicular direction|
|PBS||Phosphate buffered saline|
|PBST||PBS with Tween|
|PBST+||PBST + Arizona Test Dust + Bacillus atrophaeus + Staphylococcus epidermidis|
|PBST/Tr||PBS with 0·1% Tween (PBST) or 0·1% Triton-X (PBSTr)|
|PBT||Potassium phosphate buffer w/Tween|
|PE filter||Polyethylene filter|
|PEUF||Polyester upholstery fabric|
|PLCP||Plastic light cover panel|
|PP2D||Push-pull one direction, then other|
|PR gauze||Polyester-rayon blend gauze|
|PU foam||Polyurethane foam|
|%RSD||Per cent relative standard deviation|
|SBA||Sheep blood agar|
|Sp. stick||Sponge stick|
|Sp. wipe||Sponge wipe|
|SP/FP||Spread/filter plate for high/low concentrations|
|SSPK||Swab sample processing kit|
|SVH||S strokes, vertical then horizontal|
|TECF||3M Trace Evidence Collection Filter|
|TSA||Trypticase soy agar|
|TSAB||Trypticase soy agar with 5% sheep blood (also referred to as TSAII)|
|TSAC||Trypticase soy agar + 100μg cycloheximide per ml|
|V||Vortexing (during processing/extraction)|
|VL||Vinyl or vinyl tile|
|WD||Drops of agent dispersed in water|
Table 7. Numbers of the 20 surface-sampling studies that investigated various factors and reported specific results
|Sample collection medium type||3||0||5||0||0||0||1||0||3||0||0||0||0||0||0||0||0||0|
|Surface area sampled||1||0||1||0||0||0||0||0||1||0||0||0||0||0||1||0||0||0|
Storage and transportation of samples
Only two previous studies have investigated the effects of storage and/or transportation conditions on method performance results. Almeida et al. (2008) investigated the effects of different additives, storage times and storage temperatures on the stability of BA in water. However, that work is not directly relevant to surface sampling, which would involve storing and/or transporting swab, wipe and vacuum samples to laboratories (which would then perform the processing/extraction and analysis steps). A second study investigated the effects of storage and transportation factors on swab samples, but only partial results have been released in two poster presentations (O'Connell et al. 2010; Perry et al. 2010). Because that study only addressed swab samples, we recommend that additional studies or evaluations be performed to quantify how sampling and analysis methods for wipe and vacuum samples are affected by storage and transportation methods.
Recovery efficiency of sample collection vs processing
Several studies investigated the contributions of sample collection and sample processing/extraction to the overall RE of a method. For directly inoculated Petri dishes, Buttner et al. (2001) reported sample collection efficiency, processing efficiency and overall RE for two swab and one wipe sampling methods. For all three, the majority of the inefficiency came in the processing step, not the sample collection step. Results from other studies that investigated and reported REs for sample collection and processing steps were similar in some cases (Rose et al. 2004), while in other cases, the sample collection inefficiency was similar to or larger than processing inefficiency (Buttner et al. 2001; and Nellen et al. 2006; Brown et al. 2007a,b). In all cases, processing/extraction inefficiency is large enough that optimizing the specifics of the processing/extraction step would be very important in maximizing RE. The optimal processing conditions may depend on the sample collection material, the extraction solution and the methods of dissociation (e.g. vortexing and sonication). While it is desirable to understand how steps of the SAP affect the overall RE and improve methods when possible, this is not a ‘gap’ from the standpoint of adequately quantifying the overall performace of methods in a given SAP. Hence, we recommend additional work of this type only when it is desired to improve the performance of a particular type of surface sampling (e.g. vacuum sampling).
RE and FNR as functions of concentration and other influencing factors
Few of the 20 studies reported FNR data and only Krauter et al. (2012) developed relationships between low contaminant concentrations and the FNR performance (for the sponge-stick wipe method). Similar work is needed for other surface sampling and analysis methods that may be used when BA surface concentrations are low. Quantifying the FNR-concentration relationships is critical because FNR > 0 significantly affects (i) the number of samples required to achieve desired confidences in characterization and clearance decisions and (ii) the confidence in characterization and clearance decisions made using sample and analysis data following a contamination incident. On the other hand, if surface concentrations are at levels much higher than when false negatives begin to appear, then the magnitude of RE and its dependence on surface concentration are not very important if the goal is to merely detect contamination (vs to accurately quantify the amount/concentration of contaminant). We recommend that additional work be performed to quantify the FNR performance (as a function of lower contaminant concentrations, surface material characteristics and other relevant factors) of one or two primary sets of SAP methods, for each of swab and vacuum sampling.
Estimates of main sources of uncertainty
Previous laboratory studies did not capture the main sources of uncertainty affecting performance results. Many of the studies investigated only short-term, within-run uncertainties (repeatability) and did not investigate run-to-run or laboratory-to-laboratory uncertainties (reproducibility). Hence, the estimates of uncertainty in performance measures for most of the studies in Tables 2–5 can be expected to underestimate the total uncertainty. Further, in some studies that captured more than one source of uncertainty, the statistical measures of uncertainty reported may have been improperly calculated. A common error is to calculate the standard deviation (or %RSD) from a set of data subject to more than one source of uncertainty when the data provides for separately estimating the uncertainties. The correct approach is to use statistical variance-component estimation to separately estimate the standard deviation (or %RSD) for each source of uncertainty and then properly combine the separate estimates into an estimate of total uncertainty. The studies that best met the goals of quantifying repeatability and reproducibility uncertainties are the swab and wipe validation studies (Hodges et al. 2010 and Rose et al. 2011, respectively). We recommend that additional studies of this type be designed and performed to capture repeatability and reproducibility uncertainties in the steps of the SAP.
Control samples (e.g. positive, negative, sample medium blanks and laboratory blanks) are necessary to quantify performance measures such as RE, FNR and FPR. Although the studies summarized in this article included control samples, FNR and FPR were seldom reported. Positive controls for deposition load used to calculate RE should be colocated with each surface sample, although this was not carried out for several of the studies. Colocating a positive control with each test sample enables calculating the RE for each surface sample, if necessary. This approach corrects for unavoidable systematic variation in contaminant deposition in a chamber or controlled test area, so that such differences do not inflate the uncertainties of RE. Hence, we recommend colocating a positive control with each test sample in designing laboratory studies for surface sampling.
To close this section, we briefly compare the results in this article to those in the mini-review article by Edmonds (2009). He summarized REs for swab, wipe and vacuum methods from selected studies, whereas this article summarizes in tabular form the test conditions and several performance measures (RE, FNR, LOD, FPR and uncertainties) from 20 laboratory studies. The tabular summaries provided for indentifying several gaps and needs not discussed by Edmonds (2009). There is little overlap of this article with Edmonds (2009), because even for the few ‘needs’ addressed in both articles, the discussions are quite different. On the other hand, Edmonds (2009) discusses some difficulties, needs and perspectives that are different than discussed in this article. Hence, this article and the Edmonds (2009) article provide a comprehensive review of the performance of swab, wipe and vacuum surface-sampling methods and gaps/needs and recommendations for additional work in the future.
In each of the topic areas discussed in the preceding section, there have been useful results from the 20 studies summarized in this article. The studies of Hodges et al. (2010) and Rose et al. (2011) are noteworthy because they substantially validated the swab and wipe methods recommended by CDC (at the time of this article). The study of Krauter et al. (2012) provided FNR performance results for the CDC-recommended wipe method that had not been addressed by Rose et al. (2011). The study discussed by O'Connell et al. (2010) and Perry et al. (2010), when published, will address a gap related to the effects of storage and transportion on performance of the CDC-recommended swab method. The studies of Buttner et al. (2001) and Einfeld et al. (2011) concluded that grime and another organism, respectively, did not affect the performance of the swab and wipe, respectively, sampling methods they investigated. All of the studies have contributed to the body of knowledge on performance of swab, wipe and vacuum surface-sampling methods for BA contamination.
Although 20 laboratory studies were reviewed and summarized in this article, some of the results are for methods that are out-performed by other methods and hence would not be used in the present time. In other cases, the studies that have been performed are limited and only partially address a topic. Hence, we identified in the previous section several remaining needs and made recommendations for future work or studies. Several gaps/needs/recommendations involve quantifying the difference in surface-sampling performance results for
- surrogates vs BA (no studies to date)
- liquid vs aerosol deposition (only Edmonds et al. 2009 with varying results)
- other material (e.g. dust and grime) or organisms in addition to BA (only Buttner et al. 2001 and Einfeld et al. 2011)
- different surface materials as a function of material characteristics (only Krauter et al. 2012).
Only Krauter et al. (2012) have quantified the FNR performance of a method (sponge-stick wipe) as a function of concentration for various surface materials. So, additional work of that type for other methods is needed to be able to assess the confidence that there is no contamination given a set of negative sample results. Finally, any future studies should consider colocating positive control samples with each test sample and be designed to provide for quantifying reproducibility uncertainty and not just repeatability uncertainty.
While additional work is recommended to fill gaps/needs in several topic areas, such work could involve new evaluations of existing data (some of which may not have been published), as well as new experimental studies. It is hoped that this review article will spur work of both kinds, so that the combination of previous and new results provide for adequately quantifying the performance characteristics (e.g. RE, FNR, LOD and uncertainties) of swab, wipe and vacuum-sampling methods, accounting for all the steps of the SAP.
The Pacific Northwest National Laboratory (PNNL) work summarized in this article was funded by the Biological Research and Development Branch of the Chemical and Biological Division in the Science and Technology Directorate of the US Department of Homeland Security (DHS). We also acknowledge the US Department of Energy's Young Women in Science program, which funded the work of student intern Rebecca Hu. PNNL is a multiprogram national laboratory operated for the US Department of Energy by Battelle under Contract DE-AC05-76RL01830.
The authors acknowledge Jayne Morrow (NIST), Paula Krauter (SNL), and Brent Pulsipher (PNNL) for reviewing the technical report from which this article was adapted, as well as Landon Sego (PNNL) for reviewing the report and the article. Finally, we acknowledge Maura Zimmerscheid for excellent copy editing. We also thank the authors of several publications we cited who provided (i) copies of presentations or prepublication copies of articles, (ii) additional data or calculations, (iii) additional information not included in their publications and (iv) feedback on initial tabular summaries of their work. These include Jamie Almeida (NIST), Mark Buttner (University of Nevada), Kenneth Cole (US Army Dugway Proving Grounds), Wayne Einfeld (SNL), Cheryl Estill (CDC), Misty Hein (CDC), Robert Knowlton (SNL), Paula Krauter (SNL), Laura Rose (CDC), and Nancy Valentine (PNNL).
Versions of Tables 2 (a)–5 (a) with larger font size and less white space are available in the downloadable Piepel et al. (2011).