SEARCH

SEARCH BY CITATION

Keywords:

  • Goodness of fit test;
  • Higher criticism;
  • Local levels;
  • Normal and Poisson approximation;
  • Order statistics

Abstract

  1. Top of page
  2. Abstract
  3. 1 Introduction and summary
  4. 2 Background and notation
  5. 3 Why do intermediates take it all?
  6. 4 Why the asymptotics of the HC statistic is so poor
  7. 5 New HC tests with improved finite properties
  8. 6 Concluding remarks
  9. Acknowledgment
  10. Conflict of interest
  11. References

The higher criticism (HC) statistic, which can be seen as a normalized version of the famous Kolmogorov–Smirnov statistic, has a long history, dating back to the mid seventies. Originally, HC statistics were used in connection with goodness of fit (GOF) tests but they recently gained some attention in the context of testing the global null hypothesis in high dimensional data. The continuing interest for HC seems to be inspired by a series of nice asymptotic properties related to this statistic. For example, unlike Kolmogorov–Smirnov tests, GOF tests based on the HC statistic are known to be asymptotically sensitive in the moderate tails, hence it is favorably applied for detecting the presence of signals in sparse mixture models. However, some questions around the asymptotic behavior of the HC statistic are still open. We focus on two of them, namely, why a specific intermediate range is crucial for GOF tests based on the HC statistic and why the convergence of the HC distribution to the limiting one is extremely slow. Moreover, the inconsistency in the asymptotic and finite behavior of the HC statistic prompts us to provide a new HC test that has better finite properties than the original HC test while showing the same asymptotics. This test is motivated by the asymptotic behavior of the so-called local levels related to the original HC test. By means of numerical calculations and simulations we show that the new HC test is typically more powerful than the original HC test in normal mixture models.

1 Introduction and summary

  1. Top of page
  2. Abstract
  3. 1 Introduction and summary
  4. 2 Background and notation
  5. 3 Why do intermediates take it all?
  6. 4 Why the asymptotics of the HC statistic is so poor
  7. 5 New HC tests with improved finite properties
  8. 6 Concluding remarks
  9. Acknowledgment
  10. Conflict of interest
  11. References

Many important theoretical results related to the so-called higher criticism (HC) test statistic have been obtained during the past three decades, where it has typically been applied in the context of goodness of fit (GOF) and detection problems. The HC statistic can be seen as a normalized or standardized version of the well-known Kolmogorov–Smirnov test statistic. The asymptotics of the HC statistic was extensively investigated by Jaeschke (1979) and Eicker (1979) in the late 1970s. For earlier references see Anderson and Darling (1952). However, neither Jaeschke nor Eicker made use of the term higher criticism statistic. The notion of the higher criticism was first introduced by J. W. Tukey in the mid 1970s. Later, Tukey (1989) wrote:

If we look at many comparisons, say n, and assess the significance of each at 5% individually [...]. We know that, even if the underlying value of each comparison is blah [...] we will get an average of inline image (i.e. 5% of n) apparent significance.

Various ways of relating the observed number, k, of individual-5% significances to inline image are mnemonically referred to as “the higher criticism” [...].

It was not until a decade ago that Donoho and Jin (2004) termed the normalized Kolmogorov–Smirnov statistic as Tukey's higher criticism and rediscovered the HC in the context of detecting signals that are both sparse and weak.

Among others, Donoho and Jin (2004) showed the optimality of the HC statistic in the sense that a test based on the HC statistic asymptotically mimics the performance of an oracle likelihood ratio test under several conditions. A substantial contribution to results in Donoho and Jin (2004) was made by Ingster (1997, 1999). Jager and Wellner (2007) provided a family of GOF test statistics based on ϕ-divergences that have the same optimal detection boundary in sparse normal mixtures as the HC statistic. Hall and Jin (2008) focused on HC in the case of dependent data. Later, Hall and Jin (2010) modified the standard HC statistic used by Donoho and Jin (2004) to account for correlated noise.

Instead of approaching HC tests in terms of their test statistics a viewpoint from so-called local levels was introduced in Gontscharuk et al. (2013). There has been a lot of further interest and developments in connection with this statistic, e.g. cf. Cai et al. (2007), Donoho and Jin (2008), Hall et al. (2008), Donoho and Jin (2009), and Cai et al. (2011).

Practical applications of the HC statistic can be found in several areas. Often, these applications result from questions arising in the context of large-scale multiple testing. In particular, scientific areas such as genomics, astronomy, or image processing, have seen a growing need for statistical tools to analyze high-dimensional data. In these areas the aim is often to identify whether there are signals present in the data. For example, Parkhomenko et al. (2009) employed the HC test to detect the presence of small effects in a genome-wide association study (GWAS) on rheumatoid arthritis. In Sabatti et al. (2009), the authors analyzed GWAS data to determine genetic influences on certain metabolic traits and tested the global null hypothesis of no genetic effect using the HC statistic. Besides these applications the HC is generally applicable in any areas in which there emerges an interest in GOF testing. Here, a number of questions and issues in the context of HC goodness of fit tests remain, three of which will be addressed in this paper.

  1. It is well known that a GOF test based on the HC statistic is asymptotically sensitive for some special kinds of alternatives that differ from the null distribution in the moderate tails. However, it is not clear how to explain this behavior, since the proofs of related results are typically of pure technical nature. Based on the theory of stochastic processes we will show why a specific intermediate range plays a crucial role.
  2. It is known that the convergence of the distribution of the HC statistic to the limiting distribution is extremely slow so that the application of asymptotic results may lead to doubtful outcomes for a finite sample size. Results based on simulations, cf. Donoho and Jin (2004) or Hall and Jin (2010), show that this irregular behavior is frequently caused by the smallest order statistics of the underlying sample. However, there seems no theoretical result justifying this observation. In this paper, we will provide a simple condition how to check whether the asymptotic HC distribution approximates the finite one quite well for a given sample size.
  3. Due to a huge discrepancy between the HC's finite and asymptotic behavior, it is desirable to construct a new level α test that has the same asymptotic properties as the original HC test but shows an improved finite sample behavior. It was at the MCP2011 Conference where we introduced such a test for the first time. Eventually, at the MCP2013, we presented asymptotic as well as finite properties and power considerations of this new HC test, which will be addressed as a final topic of this paper.

This paper is organized as follows. In Section 'Background and notation', we provide the basic notation and necessary background and illustrate the key issues in detail. Section 'Why do intermediates take it all?' is devoted to the first problem. We introduce continuous stochastic processes that are related to the HC statistic. More precisely, we consider the normalized Brownian bridge and its approximation property in the region where the HC statistic is particularly sensitive. Further, we introduce the Ornstein–Uhlenbeck process that results from a suitable time transformation of the normalized Brownian bridge and helps us in the investigation of the phenomena that appear in the asymptotics of HC statistics. Section 'Why the asymptotics of the HC statistic is so poor' addresses the second key issue of this paper, that is a discussion about the quality of the HC asymptotics applied in the finite sample size case. Thereby, we study the performance of various truncated versions of the HC statistic. An essential observation is that the left tail in form of the smallest order statistics involved is key in contributing to the distribution of the HC statistic. In Section 'New HC tests with improved finite properties', we refer to the third key aspect. We construct a new HC test by considering so-called local levels and by setting these quantities to be all equal. We compare the new and original HC tests under the null hypothesis as well as under alternatives. Concluding remarks are given in Section 'Concluding remarks'.

2 Background and notation

  1. Top of page
  2. Abstract
  3. 1 Introduction and summary
  4. 2 Background and notation
  5. 3 Why do intermediates take it all?
  6. 4 Why the asymptotics of the HC statistic is so poor
  7. 5 New HC tests with improved finite properties
  8. 6 Concluding remarks
  9. Acknowledgment
  10. Conflict of interest
  11. References

Let inline image be the order statistics of n independent and identically distributed (iid) random variables. Typically, HC statistics can be expressed in terms of the order statistics inline image, inline image, from the viewpoint of the abscissa or in terms of the corresponding empirical cumulative distribution function (ecdf) inline image from the ordinate viewpoint. Below we restrict our attention to the standardized versions of the HC test statistic, that is,

  • display math(1)
  • display math(2)

Depending on the research question posed, either definition will be considered. Under the assumption that the given order statistics inline image, inline image, stem from the uniform distribution on the interval [0, 1], the limit distribution of HC0, 1 is given by the Gumbel distribution in the sense that

  • display math(3)

where inline image,

  • display math(4)

inline image and inline image, cf. Jaeschke (1979) and Eicker (1979). Then we can define a GOF test based on the HC test statistic, which we will call the HC test, for testing the null hypothesis H0 that the underlying sample is, in fact, a realization of iid standard uniform distributed random variables. We say that the HC test rejects H0 if the test statistic HC0, 1 is larger than the (asymptotic) critical value inline image. Setting inline image, the corresponding HC test is an asymptotic level α test, that is

  • display math

It is known that such HC tests are typically more powerful than the classical (asymptotic level α) Kolmogorov–Smirnov tests if an alternative distribution deviates from the null distribution in moderate tails. This is due to the fact that under H0 the supremum and/or maximum in (1) and/or (2), respectively, is asymptotically taken only over a specific intermediate range. More precisely, let us consider a truncated version of the HC statistic defined by

  • display math(5)

Thereby, we say inline image lies in the central range of (0, 1) if there exist inline image and inline image such that inline image, inline image. If inline image or inline image as inline image, we say inline image lies in the left or right tail, respectively. Moreover, inline image belongs to the intermediate range if either inline image and inline image or inline image and inline image as inline image. Note that the intermediate range is a part of the respective tail.

For what follows, we assume that H0 is true, that is, inline image are iid uniformly distributed on [0, 1], if nothing else is mentioned. Jaeschke (1979) and Eicker (1979) showed that the distribution of inline image is asymptotically degenerated in the sense that

  • display math

whenever inline image and inline image meet one of the following conditions:

  1. inline image and inline image as inline image,
  2. inline image and inline image too slowly as inline image,
  3. inline image and inline image for a inline image,
  4. inline image for a inline image and inline image.

For example, it follows that the maximum taken over the central range or the maximum over extreme tails is asymptotically stochastically smaller than the asymptotic critical value inline image for any inline image. Furthermore, applying results in Jaeschke (1979), we even get that the maximum taken over a specific intermediate range is asymptotically (stochastically) equal to the maximum taken over the entire interval (0, 1), that is

  • display math(6)

for inline image and inline image with inline image and inline image. This is why we say that intermediates take it all. In view of (6) we denote

  • display math(7)

with inline image and inline image as a sensitivity range related to the HC statistic. Roughly speaking, the sensitivity range is given by

  • display math

Surprisingly, the sensitivity range inline image is very small compared to the entire interval (0, 1) but crucial for the HC asymptotics. Unfortunately, proofs related to HC results are typically of technical nature so that it is not clear why the supremum taken over inline image is asymptotically stochastically larger than the supremum taken over the remaining area. In Section 'Why do intermediates take it all?', we will give an explanation for this phenomenon by considering the HC statistic as the maximum of a stochastic process.

Another interesting issue that is addressed in this paper is the behavior of the HC statistic in the finite case. First, we note that available formulas for the cumulative distribution function (cdf) of the HC statistic inline image (say) for a given inline image are not easy to handle analytically. One way out would be the numeric calculation or simulation of inline image for a given n. However, such computations take much time even for inline image. On the other hand, due to (3) we can approximate inline image by the (asymptotic) cdf

  • display math

where inline image is the inverse function of inline image, that is

  • display math

Unfortunately, it seems that the asymptotic distribution does not yield a good approximation even for larger n-values. For example, Jaeschke (1979) already noted that he would not recommend to use the confidence intervals that are computed on the basis of the asymptotic theory. Also Khmaladze and Shinjikashvili (2001) showed that the exact (finite) distributions of the HC statistic are quite far from the limiting distribution. Even worse, the authors note that for sample sizes up to inline image the exact distributions even diverge from the limiting one. The discrepancy between the finite and asymptotic behavior becomes clear in Fig. 1, where the (simulated) cdf of the HC statistic inline image and its asymptotic law inline image are shown for various n-values.

image

Figure 1. The cdf of the HC statistic HC0, 1 (dotted curves) simulated by 105 repetitions, that is inline image, and its respective asymptotic version inline image (solid curves) for inline image (left curves) and inline image (right curves). The vertical dashed lines locate the inline image-quantiles of inline image and inline image for inline image and inline image.

Download figure to PowerPoint

As compared to the critical values inline image of the limiting distribution, it is apparent that larger quantiles of the finite HC0, 1 statistic, which are typically relevant for testing purposes, are too large even for large sample sizes. As an example, we consider the 0.95-quantile of the limiting distribution, that is inline image with inline image and inline image. For inline image the asymptotic critical value inline image should approximate the 0.95-quantile of the exact HC0, 1-distribution; however, this quantity is equal to the 0.876-quantile. The 0.95-quantile of the exact distribution turns out to be approximately 4.74, showing the discrepancy to the asymptotic value, which is also visible in Fig. 1. Therefore, using the critical values of the asymptotic distribution in the context of testing may lead to a considerable exceedance of the prechosen level α.

Moreover, it is known that for a finite sample size unusual large values of HC0, 1 are most frequently caused by the smallest order statistic inline image. This is why truncated HC versions were considered in the literature. For example, Donoho and Jin (2004) proposed to restrict the range of the maximum by applying

  • display math

and Hall and Jin (2010) considered

  • display math

Of course, inline image and inline image are not larger than the original HC statistic so that the distributions of these truncated versions are closer to the limiting distribution. Consequently, the asymptotic critical value inline image approximates the corresponding quantiles of inline image and inline image better than the same quantile related to the HC0, 1. On the other hand, to apply a truncated HC statistic in the context of GOF tests, that is, to exclude the smallest values of the underlying sample, which typically can be seen as an indicator that the null hypothesis is false, seems to be too wasteful from a statistical point of view. This is why, in the finite case, we restrict our attention to the original HC statistic HC0, 1. Thereby, the second focus of this paper lies on addressing the question why we observe a rather poor agreement of the asymptotic and finite distributions of the HC0, 1 statistic even in a large sample size case.

The final aspect we focus on is a modification of the HC test that seems to be essential in view of the previous topic of this paper. In Section 'New HC tests with improved finite properties', we derive the new (better) HC test motivated by results in Gontscharuk et al. (2013) and show that this test is typically more powerful than the original HC test in a normal mixture model with rather sparse signals as studied by, for example, Donoho and Jin (2004).

3 Why do intermediates take it all?

  1. Top of page
  2. Abstract
  3. 1 Introduction and summary
  4. 2 Background and notation
  5. 3 Why do intermediates take it all?
  6. 4 Why the asymptotics of the HC statistic is so poor
  7. 5 New HC tests with improved finite properties
  8. 6 Concluding remarks
  9. Acknowledgment
  10. Conflict of interest
  11. References

This section is intended to provide an answer to the first issue raised in Section 'Introduction and summary' by studying the behavior of the supremum of truncated versions of the HC statistic and related approximations in various ranges.

3.1 Empirical HC process and its approximations

HC statistics defined in (5) can be seen as a supremum of the normalized (uniform) empirical process inline image over the interval inline image, where inline image is a natural normalization of the empirical cdf inline image, and I is the identity function on [0, 1]. Below, we denote inline image as the empirical HC process. By definition,

  • display math

It is well known that the empirical process inline image can be approximated by a Brownian bridge as well as by a Poisson process. The Brownian bridge is a suitable approximation in a central range, while the Poisson approximation can be more appropriate in the tails. Both approximations lead to interesting results concerning the asymptotics of the HC statistic.

In order to separate regions with different approximations, for a suitable inline image we chose the area inline image for the Poisson approximation and the interval inline image for the normal approximation. Note that there are infinitely many sequences inline image, inline image, that lead to the same results concerning the asymptotics of the HC statistic. For technical reasons, we choose

  • display math

Thereby, inline image is fulfilled for inline image.

First, we consider the HC process evaluated in the range inline image. Let inline image, inline image, denote a Poisson process with parameter inline image and paths constant except for upward jumps of height one. Lemma 3 in Jaeschke (1979) provides that for any inline image and inline image the corresponding normalized Poisson process inline image fulfills

  • display math

Moreover, following the proof of Lemma 4 in Jaeschke (1979), we get

  • display math(8)

so that values of the HC process taken in inline image are asymptotically negligible with respect to their contribution to the distribution of the original HC statistic. Hence, it suffices to study the HC process and its suprema taken over subsets of the interval inline image.

Denoting a Brownian bridge on [0, 1] by inline image, a continuous process related to the HC statistic is given by

  • display math

Clearly, inline image converges in distribution to inline image for any inline image. Moreover, it holds

  • display math(9)

cf. Shorack and Wellner (2009), p. 601. This allows us to work with the continuous process inline image instead of the discrete analog inline image on any subinterval of inline image. Therefore, we denote the normalized Brownian bridge inline image as the continuous HC process. In conclusion, we summarize the observations to get the following result.

Result 1. The cdf of the original HC statistic HC0, 1 can be approximated with any accuracy (for n large enough) by the cdf of the supremum of the continuous HC process inline image, that is by the cdf of the supremum of the normalized Brownian bridge, taken over inline image with inline image.

3.2 Relation to Gaussian stationary processes

First, we summarize results from Shorack and Wellner (2009) related to the continuous HC process inline image that show that inline image can be seen as a transformation of a Gaussian stationary process. Let inline image denote a Brownian motion on [0, ∞) and

  • display math

The process inline image is known as the Ornstein–Uhlenbeck (OU) process. The OU process inline image is a stationary zero-mean Gaussian process with unit variance and

  • display math(10)

Hence, the dependence between inline image and inline image for inline image is determined by their distance inline image only. Then Doob's transformation leads to

  • display math(11)

where inline image denotes equality in distribution, cf. Shorack and Wellner (2009, p. 20 and p. 598). Moreover, in view of the stationarity of inline image, we obtain

  • display math(12)

with

  • display math(13)

Thereby, according to (11) an interval inline image related to the continuous HC process inline image leads to the interval inline image related to the OU process inline image. Noting that inline image is stationary, the supremum of inline image over the latter interval is equal in distribution to the supremum over any interval of the same length, which is inline image. Hence, we get inline image. Moreover, due to (11) an interval (0, γ) related to inline image immediately leads to the interval inline image related to the Brownian motion inline image. Finally, we set inline image.

In view of (12) and (13) we obtain the following result.

Result 2. The supremum of the continuous HC process inline image over an interval inline image is equal in distribution to the supremum of the OU process inline image over (0, γ) with inline image defined in (13).

This observation implies, for example, that suprema of inline image over intervals, which are symmetric about inline image, are equal in distribution, that is, for inline image we get

  • display math

Furthermore, for any intervals inline image and inline image satisfying inline image and inline image, we obtain

  • display math

with inline image defined in (13). For example, Fig. 2 shows values of inline image (left picture) and the length of intervals inline image, that is inline image, (right picture) fulfilling inline image for all inline image and inline image. Note that for a fixed γ the distribution of the supremum of inline image over intervals inline image is the same for all values of d.

image

Figure 2. Values of the upper boundary e (left graph) and values of the interval's length inline image as a function of d such that the distribution of inline image is the same for all intervals inline image fulfilling inline image (right graph). Here, inline image (from the bottom to the top).

Download figure to PowerPoint

Surprisingly, Fig. 2 illustrates that the supremum of inline image over small intervals that are close to zero, can be stochastically equal to and even larger than the supremum over much larger intervals, which are further away from zero. In order to reach a deeper understanding of the nature of this phenomenon we will study the OU process in more detail.

3.3 Distribution of the supremum of the OU process

First, we turn to the OU process on a fixed interval. An exact formula for the cdf of the supremum of the OU process inline image on (0, γ) with inline image is given by

  • display math(14)

where inline image is a parabolic cylinder function, inline image is the ith root of the equation inline image as a function of ν, inline image is the derivative with respect to ν evaluated at inline image and inline image is the standard normal density function, cf. Estrella and Rodrigues (2005) and De Long (1981). To the best of our knowledge the cdf of the supremum of the OU process over a fixed interval and, particularly, the formula (14) have not been considered in connection with the HC statistic until now. At first glance, Results 1 and 2 offer a ray of hope that for a suitable inline image the expression in (14) leads to a better approximation of the cdf of HC0, 1 in the finite case than the well known Gumbel-related cdf inline image. Although the expression given in (14) seems to be hard to handle analytically, the cdf inline image can be calculated numerically as precisely as one may wish. All in all, inline image offers an alternative approximation for the finite (exact) distribution of the original HC statistic.

The limiting behavior of the supremum of inline image over an increasing interval is given by the Gumbel distribution in the sense that

  • display math(15)

where

  • display math(16)

cf. Theorem 12.3.5 in Leadbetter et al. (1989, p. 237). Setting inline image, inline image, we obtain

  • display math

where inline image is an asymptotic critical value of the HC test, cf. (4). Then the asymptotic distribution of the supremum of the OU process on the interval inline image is the same as the asymptotic distribution of HC0, 1, hence we have

  • display math(17)

As a consequence, inline image is a candidate for a better approximation of the cdf inline image in a finite case. For example, Fig. 3 shows the asymptotic and finite cdf's of the supremum of inline image over inline image for inline image. Unfortunately, the cdf inline image is an even worse approximation of the cdf of HC0, 1 than the approximation by the limit cdf inline image in the sense that critical values, which can be calculated via inline image, are typically smaller than the corresponding critical values related to inline image, that is inline image, and hence much smaller than the test statistic HC0, 1. Consequently, the search for a suitable approximation of the finite cdf of HC0, 1 has to be continued.

image

Figure 3. Asymptotic cdf inline image (solid curves) and finite cdf inline image (dash-dotted curves) of the supremum of the OU process on the interval inline image for inline image (from left to right).

Download figure to PowerPoint

Note that due to Results 1 and 2 the original HC statistic is also asymptotically equal in distribution to the supremum of the OU process over the interval inline image with inline image and inline image. That is, for such inline image, which particularly fulfills inline image, we get

  • display math(18)

Expressions (17) and (18) indicate that values of the OU process inline image over a smaller interval may not contribute much to the distribution of the supremum of inline image over distinctly larger intervals. Indeed, the next lemma provides the corresponding result.

Lemma 3.1. Let inline image be such that inline image as inline image and inline image fulfill inline image. Then for any inline image we get

  • display math(19)

Proof. Note that (19) can be written as

  • display math

In view of (15)(16), it suffices to show that inline image, which implies inline image. Applying Taylor series expansions, we get

  • display math

which completes the proof. inline image

Hence, we get the following result.

Result 3. Distributions of the supremum of the OU process over increasing intervals with lengths inline image and inline image, respectively, are asymptotically equal if inline image as inline image.

3.4 The supremum of the HC process over the intermediate range

Thanks to Results 13 we are now able to give an explanation why the intermediate range inline image plays a key role for the original HC statistic HC0, 1. Results 13 show that HC0, 1 can be approximated as well as one may wish (for n large enough) by the supremum of the maximum of the OU process over any interval of the length inline image. That is, any interval inline image would be crucial for the statistic HC0, 1 in the sense that Eq. (6) is fulfilled for this interval inline image, if the length of the corresponding OU-interval inline image is approximately inline image, that is inline image. Since the sensitivity range inline image defined in (7) corresponds to the OU-interval inline image with the length

  • display math

it becomes clear why the intermediates asymptotically take it all. It remains to take a look at the region between the two parts of inline image, that is, inline image with a suitable inline image. Although the length of this interval tends to one, the length of the corresponding OU-interval is

  • display math

For this, Lemma 3.1 yields that values of the continuous (and hence empirical) HC process on inline image asymptotically have no influence on the supremum on the entire interval (0, 1).

3.5 Asymptotic dependence structure of the HC process

In order to round out our study of the HC sensitivity range, we investigate the asymptotic dependence structure of the HC process. We show that the HC process in the central region of the interval (0, 1) is more dependent than in the sensitivity range inline image.

Let inline image be a function related to Doob's transformation, that is,

  • display math

Applying (11), we arrive at

  • display math

Then the dependence structure of the continuous HC process inline image can be described by the covariance between two points of this process as follows:

  • display math(20)

Remember that the empirical and continuous HC processes asymptotically coincide on the interval inline image for at least inline image so that in this interval also the dependence structure of the continuous process asymptotically coincides with the dependence structure of the corresponding empirical process.

To illustrate the dependence visually, we consider the covariance structure of inline image evaluated in equidistant points inline image, inline image. This choice can be motivated by noting that under the null hypothesis H0 the expectation of the ith smallest order statistic is inline image.

Figure 4 shows the transformation function inline image together with points inline image, inline image, inline image, on this curve, which are equidistant on the horizontal axis. Note that inline image maps intervals lying in any inline image with inline image (i.e. intervals lying in the central range) to intervals with bounded lengths, while intervals lying in the tails are mapped to increasing ones. For example, (0.1, 0.9) is mapped to ( − 1.0986, 1.0986), that is, any subinterval in (0.1, 0.9) is mapped to an interval with length not larger than 2.197. On the other hand, inline image with the length smaller than 0.1 is mapped to inline image, which has a length larger than inline image.

image

Figure 4. Left graph: The function inline image (solid curve), for which the processes inline image and inline image are equal in distribution for all inline image, together with points inline image, inline image, inline image for inline image (asterisks). Right graph: Points inline image, inline image, inline image for inline image.

Download figure to PowerPoint

What is more, Fig. 4 shows that the distance on the vertical axis (i.e. the distance on the OU-scale) between two adjacent points in the central range is considerably smaller than the corresponding distance in the tails. Therefore, (20) implies that for two points t and inline image, which lie in the central range of the interval (0, 1), the corresponding random variables inline image and inline image are more dependent than for t and inline image lying in the tails.

More precisely,

  • display math

For example, it follows for inline image that

  • display math

and hence for such a choice of inline image and inline image the random quantities inline image and inline image are asymptotically independent. Note that asymptotic uncorrelation and hence asymptotic independence of inline image and inline image is possible only if either inline image or inline image, that is, either inline image, inline image, lie in the left tail or inline image, inline image, lie in the right tail. Therefore, there exists a considerable amount of points t in the sensitivity range inline image such that inline image evaluated in these points are asymptotically independent. On the other hand, we get for any central interval inline image, that is for inline image, that

  • display math

and consequently there are no asymptotically independent points related to inline image in the central range.

Altogether, we obtain that the continuous as well as empirical HC process taken in the central range is asymptotically more (positively) dependent than in the sensitivity range inline image. In general, the supremum of any positively correlated normal variables is stochastically smaller than the supremum of the corresponding uncorrelated variables. This supports the fact that intermediates play the key role for the asymptotics of the HC statistic.

4 Why the asymptotics of the HC statistic is so poor

  1. Top of page
  2. Abstract
  3. 1 Introduction and summary
  4. 2 Background and notation
  5. 3 Why do intermediates take it all?
  6. 4 Why the asymptotics of the HC statistic is so poor
  7. 5 New HC tests with improved finite properties
  8. 6 Concluding remarks
  9. Acknowledgment
  10. Conflict of interest
  11. References

In this section, we approach the question why the asymptotics of the HC statistic is extremely slow, or, in other words, why HC0, 1 is too large compared to the asymptotic critical value inline image given in (4). Below, we restrict attention to the HC statistic represented in the form (2) and study the quality of various asymptotic results related to truncated versions of the HC statistic when applied in the finite sample size case.

4.1 Left-truncated HC statistics and the role of the smallest order statistics

Now we show that the finite distribution of a left-truncated HC statistic is typically dominated by the asymptotic HC distribution inline image so that, in contrast to the original HC statistic, the asymptotic critical value inline image is not too small for such a left-truncated statistic.

First, we provide a simple numerical example. For inline image and inline image we get that the asymptotic critical value inline image is too small for the HC statistic, that is inline image. By means of numerical simulations we obtain that the critical value inline image leads to an (approximately) level α HC test for inline image and inline image, that is inline image. Thereby, an interesting observation is that the first (three) smallest order statistics take most of the level α in the sense that inline image, inline image and inline image. Note that a similar phenomenon can be observed for various other α- and n-values. All this indicates that the first (e.g. three) order statistics are crucial for the HC statistic in the finite case that leads to the question whether left-truncated versions of the HC statistics are (stochastically) considerably smaller than HC0, 1 so that the limiting cdf inline image approximates (or even dominates) the distribution of these left-truncated statistics much better than the HC0, 1-distribution.

A left-truncated HC statistic we are dealing with is defined as the maximum of the empirical HC process inline image evaluated at the order statistics inline image except the first k ones, that is

  • display math

Since inline image can be seen as the maximum of just inline image points of the empirical process inline image, one can expect that for k large enough such a maximum is not larger than the supremum of the continuous process inline image evaluated in a suitable interval inline image with inline image. Clearly, the larger k, the smaller should be the corresponding interval inline image. Remembering that the cdf of the supremum of inline image over inline image is given by inline image for inline image defined in (13) and that for γ small enough inline image leads to smaller critical values than the limiting cdf inline image, cf. Section 'Why do intermediates take it all?', one may expect that the asymptotic critical value inline image is not too small for inline image if k is large enough. Moreover, it seems that even for smaller k-values we are “on the safe side” due to the discreteness of the HC statistic.

For illustration we choose inline image. Figure 5 shows the cdf of HC4, n (inline image) for inline image together with the corresponding asymptotic cdf's inline image. Apparently, almost all quantiles related to the truncated HC versions are smaller than the corresponding quantiles of the corresponding cdf inline image and hence the exceedance probability inline image is smaller than α for a lot of α-values. For example, for inline image and inline image we get inline image and inline image, respectively. For other choices of n the distribution of left-truncated HC statistics seems to behave similarly. Our simulations showed that for inline image and inline image the exceedance probability inline image is larger than α for inline image, whereas inline image for inline image.

image

Figure 5. The cdf of inline image for inline image (dash-dotted curves) simulated by 105 repetitions together with the asymptotic cdf inline image (solid curves) for inline image (left curves) and inline image (right curves).

Download figure to PowerPoint

Altogether, we have seen that for a finite (appropriate) inline image, the fact that the HC0, 1 statistic is too large compared to the asymptotic critical value inline image results from the contribution of the smallest order statistics. Thus, in order to get some insight into the reasons for the poor HC asymptotics, we now focus on the behavior of the empirical HC process inline image evaluated at the first few order statistics.

4.2 Condition for the quality of the asymptotic approximation of the HC statistic

In this section, we provide a simple condition in terms of the first order statistic how to check for given inline image and inline image whether the asymptotic critical value inline image is too small for the test statistic HC0, 1. For example, it turns out that for inline image and inline image the exceedance probability inline image is strictly larger than α and hence inline image is too small compared to HC0, 1 for such n-values.

Firstly, we note that the finite cdf inline image of the statistic HC0, 1 is not larger than the cdf of the HC process evaluated in any order statistic inline image, that is,

  • display math

An easy calculation yields

  • display math

where

  • display math(21)

Setting inline image, we get

  • display math

An upper bound inline image can be represented as the cdf of a binomially distributed random variable with parameters n and inline image so that inline image, inline image, can be calculated (at least numerically for a given inline image) by

  • display math

For example, Fig. 6 shows the first three upper bounds inline image, inline image, together with the cdf inline image (simulated by 105 repetitions) and the asymptotic cdf inline image for inline image. It looks like inline image is the smallest of inline image, inline image, and hence the best (tighter) upper bound for inline image. The picture on the right-hand side of Fig. 6 even reveals that inline image and inline image nearly coincide in the right tail. What is more, based on results related to the theory of local levels (see Gontscharuk et al., 2013) we obtain that, in fact, the minimum of inline image, inline image, is asymptotically attained for inline image. All in all, it seems that inline image is a good choice of an upper bound, which is pretty close to inline image in larger quantiles for appropriate values of n.

image

Figure 6. The cdf of HC0, 1, that is inline image, simulated by 105 repetitions (dotted curve), the cdf's of the HC process in the point inline image , that is inline image, for inline image (dashed curves from the bottom to the top in inline image) and the corresponding asymptotic cdf inline image (solid curve) for inline image. The right graph is zoomed.

Download figure to PowerPoint

For a given inline image we show how to determine the sample size inline image (say) such that the asymptotic critical value inline image is too small for the HC statistic HC0, 1 for all inline image. First, it is clear that inline image is a good approximation for the exact critical value if

  • display math

On the other hand, the following inequality is always fulfilled:

  • display math

Therefore,

  • display math(22)

would be a necessary requirement that the asymptotic critical value inline image is not too small compared to HC0, 1, so that inline image gives a good approximation to the inline image-quantile of HC0, 1. This condition can be verified for fixed values of n and α at least by means of numerical calculations. As expected, it turns out that a good approximation of the critical value by the asymptotic one can only be achieved by working with a very large sample size n. For example, for inline image the inequality (22) is not fulfilled for inline image, while it is for inline image. Hence, for practically relevant sample sizes, the asymptotic critical value inline image with inline image will always be too small compared to that of HC0, 1.

In general, even for larger n the higher quantiles of the original HC statistic can be approximated much better by the corresponding quantiles of the HC process inline image evaluated only in the first order statistic inline image than by those related to the asymptotic distribution. Hence, the inline image-quantile related to inline image is a better critical value for a test based on HC0, 1. Clearly, it makes little sense to consider such a test, while instead of HC0, 1 one may immediately choose inline image as a test statistic.

5 New HC tests with improved finite properties

  1. Top of page
  2. Abstract
  3. 1 Introduction and summary
  4. 2 Background and notation
  5. 3 Why do intermediates take it all?
  6. 4 Why the asymptotics of the HC statistic is so poor
  7. 5 New HC tests with improved finite properties
  8. 6 Concluding remarks
  9. Acknowledgment
  10. Conflict of interest
  11. References

In view of the slow convergence of the HC statistic to the asymptotic distribution as discussed in the previous section, it is desirable to modify the HC test in order to improve its applicability in finite sample size settings. To this end, Gontscharuk et al. (2013) introduced the concept of so-called local levels.

For the HC test with critical value y a local level inline image is defined as the probability that the ith order statistic inline image falls below its respective critical value inline image defined in (21). Formally, local levels can be calculated via

  • display math

These quantities can be seen as an indicator as to where one would expect high/low local sensitivity of the test.

Theorem 5.1 in Gontscharuk et al. (2013) provides the result that the first local level inline image is asymptotically the largest. Even more, simulations in that paper show that inline image is much larger than the remaining local levels and takes up almost the entire α-level; for example, for the level α HC test in Subsection 4.1 we get inline image for inline image and inline image. From a practical point of view, this finite behavior is in contradiction with the asymptotic results related to the sensitivity range. In Theorem 5.1 in Gontscharuk et al. (2013) it is shown that HC local levels related to the sensitivity range are asymptotically equal in the sense that a fraction of two local levels tends to one as n increases. Motivated by this result, a new GOF test with equal local levels in the finite sample size case seems to be a good candidate for a better HC test. This test can be defined as follows. For a prechosen inline image (say) we define a set of critical values inline image such that

  • display math(23)

Note that inline image, inline image, are determined in a unique way and can be calculated at least numerically. Then the new HC test rejects the null hypothesis H0 that the underlying sample comes from the standard uniform distribution if inline image for at least one inline image. Thereby, in order to construct a level α test, inline image has to be such that

  • display math(24)

Simple considerations imply inline image for inline image. By means of Lemma 4.3 in Gontscharuk et al. (2013) it can be seen that GOF tests with all local levels equal to

  • display math(25)

yield an asymptotic level α test. Moreover, for level α GOF tests with local levels equal to inline image we get

  • display math

The new HC level α test can alternatively be defined in terms of a test statistic. Let inline image be the cdf of inline image, that is, inline image is the cdf of the Beta distribution with parameters i and inline image. Noting that inline image via (23), condition (24) leads to

  • display math(26)

where inline image is the inverse function of the standard normal cdf and inline image. Once inline image is determined, the new HC tests rejects if inline image for some i. Note that inline image, inline image, are uniformly distributed on [0, 1]. Alternatively, setting inline image, inline image, the new HC statistic can be represented as the maximum of standard normally distributed random variables, that is

  • display math

Consequently, the new HC test with equal local levels rejects H0 if inline image. The p-value of this test can be calculated by

  • display math

where inline image is a realization of inline image. Thereby, the probability in the aforementioned expression can be calculated via one of the recursions provided in Shorack and Wellner (2009), p. 362–370.

Note that the transformation to the normal distribution in (26) may allow or facilitate the comparison of the new and original HC statistics due to the fact that HC0, 1 can be approximated by the maximum of asymptotically standard normally distributed variables. Figure 7 shows the (simulated) cdfs of HC0, 1 and inline image together with the asymptotic cdf inline image and the cdf inline image of the maximum of the OU process. Here the cdf of the new HC statistic lies to the left of the asymptotic cdf inline image, which leads to assume that the new HC test with the critical value inline image instead of inline image is a (conservative) level α test for a finite sample size. Moreover, it seems that the cdf of inline image can be approximated by inline image much better than by inline image. The behavior of the new and original HC tests under the null hypothesis as well as the concept of local levels are discussed in more detail in Gontscharuk et al (2013).

image

Figure 7. The cdf of inline image (dashed curves) and the cdf of HC0, 1 (dotted curves) simulated by 105 repetitions, asymptotic cdf inline image (solid curves) and finite OU cdf inline image (dash-dotted curves) for inline image (left picture) and inline image (right picture).

Download figure to PowerPoint

Finally, we briefly study and compare the power of the original and new HC test. We restrict attention to the following normal mixture model with sparse signals, a prominent example in the HC literature. Let inline image, inline image, inline image, inline image and let inline image be iid random variables with the cdf

  • display math

We test whether any signals are present, that is, we test H0 that inline image against the alternatives inline image that inline image for inline image. This can be reworded as GOF testing for uniformity, where inline image, inline image, are iid uniformly distributed on [0, 1] under H0. Donoho and Jin (2004) provided a detection boundary that separates the parameter plane inline image into two regions, where it is possible to reliably detect signals and where it is impossible to do so. They showed that depending on the considered parameters the power of the asymptotic level α HC test, which is defined as inline image under inline image, tends to one or α. More precisely, for the function

  • display math

the power of the asymptotic level α HC test converges to one if inline image (detectable), and tends to α if inline image (undetectable), see also Ingster (1997, 1999). Figure 8 illustrates these two regions separated by the boundary function inline image.

image

Figure 8. The detectable and undetectable regions of the (inline image) parameter plane of the HC test separated by the boundary function inline image.

Download figure to PowerPoint

Here, we take a look at the power of the HC and new HC tests in the finite case. Let inline image and inline image. Numerical calculations yield that the HC test with critical value inline image and the new HC test based on inline image are level α tests and hence can be compared in a fair way. In contrast to the asymptotic case with parameter plane inline image we consider the power in the inline image-plane, where inline image is the number of signals. Figure 9 shows the power difference

  • display math

for various values of n1 and inline image. It seems that the new HC test is more powerful in the largest part of the inline image-plane with the power difference being considerable in a large part of the asymptotically detectable region, which lies above the dash-dotted line in Figure 9. On the contrary, the original HC test has larger power only if very few signals are present. For example, for inline image and inline image the power of the HC test is ≈0.628 and the power of the new HC test is ≈0.855.

image

Figure 9. Difference in the power between the level α HC and new HC tests as a function of inline image and inline image for inline image and inline image simulated by 4 × 104 repetitions. The area with a positive (negative) power difference, where the new HC test is more (less) powerful, is marked by solid (dashed) contours. The asymptotically detectable (undetectable) region of the HC test lies above (below) the dash-dotted curve.

Download figure to PowerPoint

Finally, we consider “local powers” of the level α HC and new HC tests, which are defined as local levels under alternatives, that is

  • display math

respectively. Hence, the ith “local power” is the probability that inline image leads to the rejection of H0 under inline image. Figure 10 shows these “local powers” of the level α HC tests for inline image, inline image, inline image (i.e. inline image) and inline image. It is only the first 7 order statistics for which the “local powers” are larger for the original HC test than that for the new one while the “local power” for the new HC is much better than that for the original one for a large (especially the intermediate) range of i. It seems very likely that in the finite case the power of the original HC test is concentrated on the first few order statistics (extreme values), while the power of the new HC test is concentrated on the intermediates (moderate tail).

image

Figure 10. “Local powers” of the original level α HC test (dashed lines in the left graph, circles in the right graph) and the new level α HC test (solid lines in the left graph, solid diamonds in the right graph) for inline image and inline image locally at each inline image (left graph) and inline image (right graph).

Download figure to PowerPoint

Altogether, it can be supposed that the new HC test finitely offers what the HC test asymptotically promises. Even more, beyond the context of testing the global null hypothesis in high dimensional data, the proposed new test with equal local levels is an attractive, easy to compute competitor to classical GOF tests and also allows for easy to compute simultaneous confidence bands for the related test statistic.

6 Concluding remarks

  1. Top of page
  2. Abstract
  3. 1 Introduction and summary
  4. 2 Background and notation
  5. 3 Why do intermediates take it all?
  6. 4 Why the asymptotics of the HC statistic is so poor
  7. 5 New HC tests with improved finite properties
  8. 6 Concluding remarks
  9. Acknowledgment
  10. Conflict of interest
  11. References

In this paper, we first focused on two topics that explain some important properties of the HC test statistic. The first one concerns the range of sensitivity of GOF tests based on the HC statistic, the second addresses the convergence of the finite HC distribution to its limiting counterpart. Eventually, we proposed a modification of the HC test showing better finite properties.

The fact that tests based on the HC statistic effectively detect signals which are very weak and very sparse, aroused a lot of interest among statisticians. Such tests are even asymptotically successful throughout the same region in the amplitude/sparsity plane where the oracle likelihood ratio test would succeed. The nature of this phenomenon becomes clearer by looking at the sensitivity region of the HC statistic. The explanation given in Section 'Why do intermediates take it all?', why a certain intermediate range of order statistics plays a crucial role for the HC asymptotics, contributes to the knowledge about the behavior of the HC statistic. Unfortunately, due to the fact that the distribution of the HC statistic converges to the limiting one very slowly, the type I error by the corresponding HC tests is not controlled even for a very large sample size, cf. Section 'Why the asymptotics of the HC statistic is so poor'.

In view of this point, it would be favorable to have a “better” HC statistic at hand. It is worthy to note that the application of truncated HC versions leads to an exclusion of several order statistics of the underlying random sample and hence to a loss of information. This is why, in our opinion, truncated HC statistics are not suited to be “better” HC statistics. New approaches seem to be necessary in order to construct more favorable HC tests, that is a test with the same asymptotic properties as the original HC test, but with a more appropriate finite sample size behavior. The concept of the so-called local levels introduced in Section 'New HC tests with improved finite properties' seems to be such a promising approach. Local levels can be seen as an indicator as to where one would expect high/low local sensitivity, for more details see Gontscharuk et al. (2013). Motivated by a result in this work that almost all HC local levels are asymptotically equal, a new GOF test with equal local levels in the finite sample size case seems a good candidate that might show the properties mentioned. In Section 'New HC tests with improved finite properties', we provide new results concerning the asymptotics of the new HC test and show by means of simulations that the new test is typically more powerful than the original procedure in a normal mixture model. A more detailed study of the new HC test with equal local levels will be reported in our forthcoming work.

References

  1. Top of page
  2. Abstract
  3. 1 Introduction and summary
  4. 2 Background and notation
  5. 3 Why do intermediates take it all?
  6. 4 Why the asymptotics of the HC statistic is so poor
  7. 5 New HC tests with improved finite properties
  8. 6 Concluding remarks
  9. Acknowledgment
  10. Conflict of interest
  11. References
  • Anderson, T. W. and Darling, D. A. (1952). Asymptotic theory of certain “Goodness of fit” criteria based on stochastic processes. Annals of Mathematical Statistics 23, 193212.
  • Cai, T., Jin, J. and Low, M. (2007). Estimation and confidence sets for sparse normal mixtures. Annals of Statistics 35, 24212449.
  • Cai, T., Jeng, J. and Jin, J. (2011). Optimal detection of heterogeneous and heteroscedastic mixtures. Journal of the Royal Statistical Society B 73, 629662.
  • De Long, D. M. (1981). Crossing probabilities for a square root boundary by a bessel process. Communications in Statistics – Theory and Methods 10, 21972213.
  • Donoho, D. and Jin, J. (2004). Higher criticism for detecting sparse heterogeneous mixtures. Annals of Statistics 32, 962994.
  • Donoho, D. and Jin, J. (2008). Higher criticism thresholding: optimal feature selection when useful features are rare and weak. Proceedings of the National Academy of Sciences USA 105, 1479014795.
  • Donoho, D. and Jin, J. (2009). Feature selection by higher criticism thresholding achieves the optimal phase diagram. Philosophical Transactions of the Royal Society A 367, 44494470.
  • Eicker, F. (1979). The asymptotic distribution of the suprema of the standardized empirical processes. Annals of Statistics 7, 116138.
  • Estrella, A. and Rodrigues, A. P. (2005). One-sided test for an unknown breakpoint: theory, computation and application to monetary theory. Staff Report, no. 232.
  • Gontscharuk, V., Landwehr, S. and Finner, H. (2013). Goodness of fit tests in terms of local levels. Submitted for publication.
  • Hall, P. and Jin, J. (2008). Properties of higher criticism under strong dependence. Annals of Statistics 36, 381402.
  • Hall, P. and Jin, J. (2010). Innovated higher criticism for detecting sparse signals in correlated noise. Annals of Statistics 38, 16861732.
  • Hall, P., Pittelkow, Y. and Ghosh, M. (2008). Theoretical measures of relative performance of classifiers for high dimensional data with small sample sizes. Journal of the Royal Statistical Society B 70, 159173.
  • Ingster, Y. I. (1997). Some problems of hypothesis testing leading to infinitely divisible distribution. Mathematical Methods of Statistics 6, 4769.
  • Ingster, Y. I. (1999). Minimax detection of a signal for inline image-balls. Mathematical Methods of Statistics 7, 401428.
  • Jaeschke, D. (1979). The asymptotic distribution of the supremum of the standardized empirical distribution function on subintervals. Annals of Statistics 7, 108115.
  • Jager, L. and Wellner, J. (2007). Goodness-of-fit tests via phi-divergences. Annals of Statistics 35, 20182053.
  • Khmaladze, E. and Shinjikashvili, E. (2001). Calculation of noncrossing probabilities for Poisson processes and its corollaries. Advances in Applied Probability 33, 702716.
  • Leadbetter, M. R., Lindgren, G. and Rootzen, H. (1989). Extremes and Related Properties of Random Sequences and Processes. Springer, New York, NY.
  • Parkhomenko, E., Tritchler, D., Lemire, M., Hu, P. and Beyene, J. (2009). Using a higher criticism statistic to detect modest effects in a genome-wide study of rheumatoid arthritis. BMC Proceedings 3 Suppl 7, S40.
  • Sabatti, C., Service, S. K., Hartikainen, A.l., Pouta, A., Ripatti, S., Brodsky, J., Jones, C. G., Zaitlen, N. A., Varilo, T., Kaakinen, M., Sovio, U., Ruokonen, A., Laitinen, J., Jakkula, E., Coin, L., Hoggart, C., Collins, A., Turunen, H., Gabriel, S., Elliot, P., McCarthy, M. I., Daly, M. J., Järvelin, M. R., Freimer, N. B. and Peltonen, L. (2009). Genome-wide association analysis of metabolic traits in a birth cohort from a founder population. Nature Genetics 41, 3546.
  • Shorack, G. and Wellner, J. (2009). Empirical Processes with Applications to Statistics. Society for Industrial and Applied Mathematics, Philadelphia, PA.
  • Tukey, J. (1989). Higher criticism for individual significances in several tables or parts of tables. Internal Working Paper 89-9, Princeton University, Princeton, NJ.