SEARCH

SEARCH BY CITATION

Keywords:

  • high-throughput analysis;
  • software;
  • statistics;
  • microarray

Dear Sir,

Properly dealing with DNA microarray data involves employing statistical methods beyond that which we apply to regular experimental findings. For example, inferring statistically significant changes in gene expression requires taking into account the multiple testing problem inherent to microarray-based experiments (Allison et al., 2006). Ignoring this aspect alone has led to significant problems in reproducing results from several microarray-based melanoma research programs (Hoek, 2007). Therefore, it is no surprise that user-friendly softwares promising to deal appropriately with microarray data are gaining popularity. Such tools aim to eliminate the need for laboratories to contract specialist staff and, if properly executed, have the potential to rapidly expedite the translation of high-throughput data into useful hypotheses.

One on-line data analysis tool is Genesifter (http://www.genesifter.net) which is produced by VizX laboratories and marketed towards biologists who are having difficulty coming to grips with very large DNA microarray data sets. The Genesifter layout is relatively straightforward and there is a section of the website where one can try out the software (following helpful tips in the margins) free of charge using publicly accessible data sets. One simply uploads data and (prompted appropriately) applies the desired algorithms. The results are shown in ranked tabular form and may be readily downloaded. Currently, the going subscription rate is $4000 a year which is substantially cheaper than hiring a dedicated biostatistician. With this model it would be no surprise that Genesifter is, as it claims, ‘the world’s fastest growing gene expression analysis system.’ However, there are critical problems with Genesifter’s pairwise analysis algorithm as it commits at least two errors in data handling: the insufficient normalization of data and the introduction of selection bias while calculating significance.

During the course of microarray experimentation, errors in measurement can arise from (just to name a few) between-sample differences in the quantity of RNA, labelling efficiency, errors in platform manufacture and systematic measurement bias. To control for these errors requires normalization of the raw data so that chip experiments may be comparable (Quackenbush, 2002). There are several ways of going about this but Genesifter’s method is to conduct normalization so that differences between chips are not accounted for. Instead, normalization is conducted only within each chip such that every raw data point is divided by the median (or mean) of all data points on the chip. This half-way approach, which omits an additional division of each data point by the median (or mean) of its performance across different chips to account for between-sample error, makes the inadvisable assumption that there is no significant between-chip variation to account for.

Inferring significance for a gene’s differential expression between sample classes involves performing a statistical test. For DNA microarray analyses, a critical aspect requires taking into account the multiple testing problem. Each probe on a microarray platform represents an individual experiment, and due to biological and systematic variation the performance of thousands of such experiments ensures that many apparently significant results will be the result of chance events (false positives). To control this various multiple testing correction measures can be applied. Genesifter allows users to choose a statistical test and apply a suitable multiple testing correction. Additionally, users may apply a fold-change filter to further narrow down the list of genes with significantly changed expression. However, including the fold-change filter with statistical testing is problematic, because Genesifter applies the fold-change filter to averaged data before performing statistical tests. This practise is a form of selection bias and serves to increase false positives in the data when statistical testing is complete. When seeking to identify which of a population of genes undergo significant changes between sample classes, it is important to remember that the entire available population of genes must be subject to statistical analysis. By first cherry-picking for genes which (on average) show a fold-change between sample classes, subsequent testing is biased towards incorrectly identifying outcomes as having rejected the null hypothesis. The reason for this is because multiple testing correction is sensitive to the number of tests conducted, with fewer tests the correction becomes less stringent. Essentially, the Genesifter protocol determines that genes failing the fold-change test are outliers not to be considered in statistical testing. When the greater proportion of your data set becomes statistical ‘outliers’ this makes no sense.

The impact of these errors can be seen when comparing the performance of Genesifter against that of Genespring, in which users have control over the application of normalization procedures, data filters, and statistical tests. Genesifter provides a number of data sets which can be assessed by anyone wishing to try out the program. One of these is a collection of colon cancer samples with both primary and metastasis data (which is also accessible through NCBI’s Gene Expression Omnibus GEO website, http://www.ncbi.nlm.nih.gov/geo/, accession no. GDS756). Using the Genesifter program, including fold-change and multiple testing controls, yields 1556 genes which show a significant differential in expression of at least 1.5-fold between sample classes. An analysis of the same data performed in Genespring (Agilent Technologies, Santa Clara, CA, USA), including cross-chip normalization and t-testing prior to filtering, yields a substantially reduced list of 449 genes. This Genespring analysis estimates that more than two-thirds of the results determined by Genesifter are false positives.

It is one thing for a software to allow users to make mistakes on their own, but it is entirely another when users do not have the option to perform the necessary steps in the appropriate manner. Indeed, Genesifter customers are kept wholly blind to its deficiencies. Genesifter is touted as ‘designed and built by biologists who understand how biology research is done,’ the makers might have done better if they had employed statisticians who understand how high-throughput research is done.

References

  1. Top of page
  2. References