SEARCH

SEARCH BY CITATION

Abstract

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE GARCH UNIVERSE
  5. 3. FORECAST EVALUATION
  6. 4. REALIZED VARIANCE
  7. 5. TEST FOR SUPERIOR PREDICTIVE ABILITY
  8. 6. DATA AND EMPIRICAL RESULTS
  9. 7. CONCLUSIONS
  10. Acknowledgements
  11. REFERENCES
  12. Supporting Information

We compare 330 ARCH-type models in terms of their ability to describe the conditional variance. The models are compared out-of-sample using DM–$ exchange rate data and IBM return data, where the latter is based on a new data set of realized variance. We find no evidence that a GARCH(1,1) is outperformed by more sophisticated models in our analysis of exchange rates, whereas the GARCH(1,1) is clearly inferior to models that can accommodate a leverage effect in our analysis of IBM returns. The models are compared with the test for superior predictive ability (SPA) and the reality check for data snooping (RC). Our empirical results show that the RC lacks power to an extent that makes it unable to distinguish ‘good’ and ‘bad’ models in our analysis. Copyright © 2005 John Wiley & Sons, Ltd.


1. INTRODUCTION

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE GARCH UNIVERSE
  5. 3. FORECAST EVALUATION
  6. 4. REALIZED VARIANCE
  7. 5. TEST FOR SUPERIOR PREDICTIVE ABILITY
  8. 6. DATA AND EMPIRICAL RESULTS
  9. 7. CONCLUSIONS
  10. Acknowledgements
  11. REFERENCES
  12. Supporting Information

The conditional variance of financial time series is important for pricing derivatives, calculating measures of risk, and hedging. This has sparked an enormous interest in modelling the conditional variance and a large number of volatility models have been developed since the seminal paper of Engle (1982); see Poon and Granger (2003) for an extensive review and references.

The aim of this paper is to examine whether sophisticated volatility models provide a better description of financial time series than more parsimonious models. We address this question by comparing 330 GARCH-type models in terms of their ability to forecast the one-day-ahead conditional variance. The models are evaluated out-of-sample using six different loss functions, where the realized variance is substituted for the latent conditional variance. We use the test for superior predictive ability (SPA) of Hansen (2001) and the reality check for data snooping (RC) by White (2000) to benchmark the 330 volatility models to the GARCH(1,1) of Bollerslev (1986). These tests have the advantage that they properly account for the full set of models, without the use of probability inequalities, such as the Bonferroni bound, that typically lead to conservative tests.

We compare the models using daily DM–$ exchange rate data and daily IBM returns. There are three main findings of our empirical analysis. First, in the analysis of the exchange rate data we find no evidence that the GARCH(1,1) is inferior to other models, whereas the GARCH(1,1) is clearly outperformed in the analysis of IBM returns. Second, our model space includes models with many distinct characteristics that are interesting to compare,1 and some interesting details emerge from the out-of-sample analysis. The models that perform well in the IBM return data are primarily those that can accommodate a leverage effect, and the best overall performance is achieved by the A-PARCH(2,2) model of Ding et al. (1993). Other aspects of the volatility models are more ambiguous. While the t-distributed specification of standardized returns generally leads to a better average performance than the Gaussian in the analysis of exchange rates, the opposite is the case in our analysis of IBM returns. The different mean specifications, zero-mean, constant mean and GARCH-in-mean, result in almost identical performances. Third, our empirical analysis shows that the RC has less power than the SPA test. This makes an important difference in our application, because the RC cannot detect that the GARCH(1,1) is significantly outperformed by other models in the analysis of IBM returns. In fact, the RC even suggests that an ARCH(1) may be the best model in many cases, which does not conform with the existing empirical evidence. The SPA test always finds the ARCH(1) model to be inferior, which shows that the SPA test has power in these applications and is therefore more likely to detect superior models when such exist.

Ideally, we would evaluate the models' ability to forecast all aspects of the conditional distribution. However, it is not possible to extract precise information about the conditional distribution without making restrictive assumptions. Instead we focus on the central component of the models—the conditional variance—that can be estimated by the realized variance. Initially, it was common to substitute the squared return for the unobserved conditional variance in out-of-sample evaluations of volatility models. This typically resulted in a poor performance, which instigated a discussion of the practical relevance of volatility models. However, Andersen and Bollerslev (1998) showed that the ‘poor’ performance could be explained by the fact that the squared return is a noisy proxy for the conditional variance. By substituting the realized variance (instead of the squared return), Andersen and Bollerslev (1998) showed that volatility models perform quite well. Hansen and Lunde (2003) provide another important argument for using the realized variance rather than the squared return. They show that substituting the squared returns for the conditional variance can severely distort the comparison, in the sense that the empirical ranking of models may be inconsistent for the true (population) ranking. So an evaluation that is based on squared returns may select an inferior model as the ‘best’ with a probability that converges to one as the sample size increases. For this reason, our evaluation is based on the realized variance.

Comparing multiple models is a non-standard inference problem, and spurious results are likely to appear unless inference controls for the multiple comparisons. An inferior model can be ‘lucky’ and perform better than all other models, and the more models that are being compared the higher is the probability that the best model (in population) has a much smaller sample performance than some inferior model. It is therefore important to control for the full set of models and their interdependence when evaluating the significance of an excess performance. In our analysis we employ the SPA test and the RC, which are based on the work of Diebold and Mariano (1995) and West (1996). These tests can evaluate whether a particular model (benchmark) is significantly outperformed by other models, while taking into account the large number of models that are being compared. In other words, these tests are designed to evaluate whether an observed excess performance is significant or could have occurred by chance.

This paper is organized as follows. Section 2 describes the 330 volatility models under consideration and the loss functions are defined in Section 3. In Section 4, we describe our measures of realized variance and Section 5 contains some details of the SPA test and its bootstrap implementation. We present our empirical results in Section 6 and Section 7 contains some concluding remarks.

2. THE GARCH UNIVERSE

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE GARCH UNIVERSE
  5. 3. FORECAST EVALUATION
  6. 4. REALIZED VARIANCE
  7. 5. TEST FOR SUPERIOR PREDICTIVE ABILITY
  8. 6. DATA AND EMPIRICAL RESULTS
  9. 7. CONCLUSIONS
  10. Acknowledgements
  11. REFERENCES
  12. Supporting Information

Given a price process, pt, we define the compounded daily return by rt = log(pt)− log(pt−1), t = −R + 1, …, n. Later we split the sample into an estimation period (the first R observations) and an evaluation period (the last n observations).

The conditional density of rt is denoted by f(r|t−1), where t−1 is the σ-algebra induced by variables that are observed at time t − 1. We define the conditional mean by µtE(rt|t−1) (the location parameter) and the conditional variance by equation image (the scale parameter), assuming that both are well defined. Subsequently we can define the standardized return, et ≡ (rt − µt)/σt, and denote its conditional density by g(e|t−1). Following Hansen (1994) we consider a parametric specification, f(r|ψ(t−1;θ)), where θ ∈ Θ ⊂ q is a vector of parameters. It now follows that the time-varying vector of parameters, ψt ≡ ψ(t−1;θ), can be divided into equation image, where ηt is a vector of shape parameters for the conditional density of et. Thus, we have a family of density functions for rt, which is a location-scale family with (possibly time-varying) shape parameters, and we shall model equation image and ηt individually. Most GARCH-type models can be formulated in this framework and ηt typically does not depend on t.

The notation for our modelling of the conditional mean and variance is mt = µ(t−1;θ) and equation image, respectively, and we employ two specifications for g(et) in our empirical analysis. One is a Gaussian specification that is free of parameters g(et) = g(e), and the other is a t-specification that has degrees of freedom, υ, as the only parameter, g(et) = g(e|υ).2 Our specifications for the conditional mean are: equation image (GARCH-in-mean), mt = µ0 and mt = 0.

The conditional variance is the main object of interest and our analysis includes a large number of parametric specifications for σt that are listed in Table I. The use of acronyms has not been fully consistent in the existing literature, for example, A-GARCH has been used to represent four different specifications. So to avoid any confusion we use ‘A-GARCH’ to refer to a model by Engle and Ng (1993) and use different acronyms for all other models, e.g., we use H-GARCH to refer to the model by Hentshel (1995). Several specifications nest other specifications, as is evident from Table I. In particular, the flexible specifications of the H-GARCH and the Aug-GARCH, see Duan (1997), nest many of the simpler specifications. An empirical comparison of several of the models that are nested in the Aug-GARCH model can be found in Loudon et al. (2000).

Table I. Specifications for the conditional variance
  • a

    This is the A-PARCH model without the leverage effect.

  • b

    Here f(x, ν) = (xν − 1)/ν.

ARCH:equation image
GARCH:equation image
IGARCHequation image
Taylor/Schwert:equation image
A-GARCH:equation image
NA-GARCH:equation image
V-GARCH:equation image
Thr.-GARCH:equation image
GJR-GARCH:equation image
log-GARCH:equation image
EGARCH:equation image
NGARCH:aequation image
A-PARCH:equation image
GQ-ARCH:equation image
H-GARCH:equation image
Aug-GARCH:bequation image
equation image
equation image
equation image

The evolution of volatility models has been motivated by empirical findings and economic interpretations. Ding et al. (1993) used Monte Carlo simulations to demonstrate that both the GARCH specification (model for equation image) and the TS-GARCH specification3 (model for σt) are capable of producing the autocorrelation pattern that is seen in financial data. So in this respect there is no argument for modelling σt rather than equation image or vice versa. More generally, we can consider a modelling of equation image where δ is a parameter to be estimated, and this motivated the Box–Cox transformations that involve σt and εt. The empirically observed leverage effect motivated the development of models with an asymmetric response in volatility to positive and negative shocks. The leverage effect was first noted by Black (1976) and is best illustrated by the news impact curve, which was introduced by Pagan and Schwert (1990) and named by Engle and Ng (1993). This curve is a plot of equation image against εt−1 that illustrates how the volatility reacts to good and bad news.

In our analysis, we have included the four combinations of p, q = 1, 2 for the lag length parameters, with the following exceptions: the ARCH is only estimated for q = 1; H-GARCH and Aug-GARCH are only estimated for (p, q) = (1, 1), because these are quite burdensome to estimate. It is well known that an ARCH(1) model is unable to fully capture the persistence in volatility, and this model is only included as a point of reference, and to verify that the tests, SPA and RC, have power. This is an important aspect of the analysis, because a test that is unable to reject that the ARCH(1) is the best model cannot be very informative about which is a better model. Restricting the models to have two lags (or less) should not affect the main conclusions of our empirical analysis, because it is unlikely that a model with more lags would outperform a simple benchmark in the out-of-sample comparison, unless the same model with two lags can outperform the benchmark. This aspect is also evident from our analysis, where a model with p = q = 2 rarely performs better (out-of-sample) than the same model with fewer lags, even though most parameters are found to be significant (in-sample).

3. FORECAST EVALUATION

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE GARCH UNIVERSE
  5. 3. FORECAST EVALUATION
  6. 4. REALIZED VARIANCE
  7. 5. TEST FOR SUPERIOR PREDICTIVE ABILITY
  8. 6. DATA AND EMPIRICAL RESULTS
  9. 7. CONCLUSIONS
  10. Acknowledgements
  11. REFERENCES
  12. Supporting Information

A popular way to evaluate volatility models out-of-sample is in terms of the R2 from a Mincer–Zarnowitz (MZ) regression, equation image, where squared returns are regressed on the model forecast of equation image and a constant. Or the logarithmic version, equation image, that is less sensitive to outliers, as was noted by Pagan and Schwert (1990) and Engle and Patton (2001).4 However, the R2 of a MZ regression is not an ideal criterion for comparing volatility models, because it does not penalize a biased forecast. For example, a poor biased forecast may achieve a higher R2 than a good unbiased forecast, because the bias can be eliminated artificially through estimates of (a, b) that differ from (0, 1).

It is not obvious which loss function is more appropriate for the evaluation of volatility models, as discussed by Bollerslev et al. (1994), Diebold and Lopez (1996) and Lopez (2001). So rather than making a single choice we use the following six loss functions in our empirical analysis:

  • equation image

The criteria MSE2 and R2LOG are similar to the R2 of the MZ regressions,5 and QLIKE corresponds to the loss implied by a Gaussian likelihood. The mean absolute error criteria, MAE2 and MAE1, are interesting because they are more robust to outliers than, say, MSE2. Additional discussions of the MSE2, QLIKE and R2LOG criteria can be found in Bollerslev et al. (1994).

4. REALIZED VARIANCE

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE GARCH UNIVERSE
  5. 3. FORECAST EVALUATION
  6. 4. REALIZED VARIANCE
  7. 5. TEST FOR SUPERIOR PREDICTIVE ABILITY
  8. 6. DATA AND EMPIRICAL RESULTS
  9. 7. CONCLUSIONS
  10. Acknowledgements
  11. REFERENCES
  12. Supporting Information

In our empirical analysis we substitute the realized variance for the latent equation image. The realized variance for a particular day is calculated from intraday returns, ri, m, where rt, i, mpt−(i−1)/mpti/m for i = 1, …, m. Thus rt, i, m is the return over a time interval with length 1/m on day t, and we note that equation image. It will often be reasonable to assume that E(rt, i, m|t−1)≃0 and that intraday returns are conditionally uncorrelated, cov(rt, i, m, rt, j, m|t−1) = 0 for ij, such that equation image, where we have defined the realized variance (at frequency m) equation image. Thus equation image is approximately unbiased for equation image (given our assumptions above), and it can often be shown that equation image is decreasing in m, such that equation image is an increasingly more precise estimator of equation image as m increases.6 Further, the equation image is (by definition) consistent for the quadratic variation of pt, which is identical to the conditional variance, equation image, for certain data generating processes (DGPs) such as the ARCH-type models considered in this paper.7

Several assets are not traded 24 hours a day, because the market is closed overnight and over weekends. In these situations we only observe fm (of the m possible) intraday returns. Assume for simplicity that we observe, rt, 1, m, …, rt, f, m and define equation image. Since equation image only captures the volatility during the part of the day that the market is open, we need to extend equation image to a measure of volatility for the full day. One resolution is to add the squared close-to-open return to equation image, but this leads to a noisy measure because ‘overnight’ returns are relatively noisy. A better solution is to scale equation image, and use the estimator

  • equation image(1)

This yields an estimator that is approximately unbiased for equation image under fairly reasonable assumptions. See Martens (2002), Hol and Koopman (2002) and Fleming et al. (2003), who applied similar scaling estimators to obtain a measure of volatility for the whole day.

5. TEST FOR SUPERIOR PREDICTIVE ABILITY

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE GARCH UNIVERSE
  5. 3. FORECAST EVALUATION
  6. 4. REALIZED VARIANCE
  7. 5. TEST FOR SUPERIOR PREDICTIVE ABILITY
  8. 6. DATA AND EMPIRICAL RESULTS
  9. 7. CONCLUSIONS
  10. Acknowledgements
  11. REFERENCES
  12. Supporting Information

We divide the observations into an estimation period and an evaluation period:

  • equation image

The parameters of the volatility models are estimated using the first R interday observations, and these estimates are used to make one-step-ahead forecasts for the remaining n periods. During the evaluation period we calculate the realized variance from intraday returns and obtain equation image using (1). Thus model k yields a sequence of forecasts, equation image, that are compared to equation image, using a loss function L. Let the first model, k = 0, be the benchmark model that is compared to models k = 1, …, l. Each model leads to a sequence of losses, equation image, and we define the relative performance variables

  • equation image

Our null hypothesis is that the benchmark model is as good as any other model in terms of expected loss. This can be formulated as the hypothesis H0kE(Xk, t)≤ 0 for all k = 1, …, l, because λk > 0 corresponds to the case where model k is better than the benchmark. In order to apply the stationary bootstrap of Politis and Romano (1994) in our empirical analysis, we assume that Xt = (X1, t, …, Xl, t)′ is strictly stationary, E|Xt|r < ∞ for some r > 2 and some δ > 0, and that Xt is α-mixing of order −r/(r − 2). These assumptions are due to Goncalves and de Jong (2003) and are weaker than those formulated in Politis and Romano (1994). The stationarity of {Xt} would be satisfied if {rt} is strictly stationary, because {Xt} is a function of {rt}. Next, the moment condition is not alarming, because {Xt} measures the difference in performance of pairs of models, and it is unlikely that the predictions would be so different that the relative loss would violate the moment condition, since the models are quite similar and have the same information. Finally, the mixing condition for {Xt} is satisfied if it holds for rt. It is important to note that we have not assumed that any of the volatility models are correctly specified. Nor is such an assumption needed, since our ranking of volatility models is entirely measured in terms of expected loss. The assumptions about {rt} will suffice for the comparison and inference, and it is not necessary to make a reference to the true specification of the conditional variance. On the other hand, there is nothing preventing one of the volatility models being correctly specified.8

The bootstrap implementation can be justified under weaker assumptions than those above. For example, the stationarity assumption about {rt} can be relaxed and replaced by a near-epoch condition for Xt, see Goncalves and de Jong (2003). This is valuable to have in mind in the present context, since the returns may not satisfy the strict stationarity requirement. A structural change in the DGP would be more critical for our analysis. While a structural change need not invalidate the bootstrap inference (if the break occurs prior to the evaluation period), it would make it very difficult to interpret the results, because the models are estimated using data that have different stochastic properties.

As stated above, the null hypothesis is given by H0:λ0, where λ = (λ1, …, λl)′. The SPA test is based on the test statistic, equation image, where k is the kth element of equation image and equation image is a consistent estimator of equation image. Thus, equation image represents the largest t-statistic (of relative performance) and the relevant question is whether equation image is too large for it to be plausible that λ0. This is precisely the question that the test for SPA is designed to answer, as it estimates the distribution of equation image under the null hypothesis and obtains the critical value for equation image.

A closely related test is the RC of White (2000) that employs the non-standardized test statistic equation image. The critical values of the SPA test and the RC are derived in different ways, and this causes the latter to be sensitive to the inclusion of poor and irrelevant models, and to be less powerful, see Hansen (2003) for details. Power is important for our application, because a more powerful test is more likely to detect superior volatility models, if such exist.

Given the assumptions stated earlier in this section, it holds that equation image, where ‘equation image’ denotes convergence in distribution, where λ = (λ1, …, λl)′ and Ω≡limn[RIGHTWARDS ARROW]E[n(Xλ)(Xλ)′]. This result makes it possible to test the hypothesis, H0:λ0.

5.1. Bootstrap Implementation

Unless n is large relative to l it is not possible to obtain a precise estimate of the l × l covariance matrix, Ω. It is therefore convenient to use a bootstrap implementation, which does not require an explicit estimate of Ω, and the tests of White (2000) and Hansen (2001) can both be implemented with the stationary bootstrap of Politis and Romano (1994).9 From the bootstrap resamples, equation image, we can construct random draws of quantities of interest, which can be used to estimate the distributions of these quantities. In our setting we seek an estimate of equation image and estimates of the distributions of equation image and equation image. First we calculate the sample averages, equation image, and it follows from Goncalves and de Jong (2003) that the empirical distribution of equation image converges to the true asymptotic distribution of n1/2X. The resamples also allow us to calculate equation image, which is consistent for equation image. We seek the distribution of the test statistics, equation image and equation image, under the null hypothesis, λ0, so we must re-centre the bootstrap variables, such that they satisfy the null hypothesis.10 Ideally, the variables should be re-centred about the true value of λ, but since λ is unknown we must use an estimate and Hansen (2001) proposed the estimates:

  • equation image

where equation image. Thus we define equation image, for i = l, c, u, where gl(·)≡max(x, 0), gc(x)≡x·1math image and gu(x)≡x, and it follows that equation image for i = l, c, u. This enables us to approximate the distribution of equation image by the empirical distribution of

  • equation image(2)

and we calculate the p-value: equation image, for i = l, c, u. The null hypothesis is rejected for small p-values. In the event that equation image, there is no evidence against the null hypothesis, and in this case we use the convention: SPA≡1.

The three choices for equation image will typically yield three different p-values, and Hansen (2001) has shown that the p-value based on equation image is consistent for the true p-value, whereas equation image and equation image provide an upper and lower bound for the true p-value, respectively.11 We denote the three resulting tests by SPAl, SPAc and SPAu, where the subscripts refer to lower, consistent and upper. The purpose of the correction factor, Ak, n, that defines equation image, is to ensure that equation image and equation image. This is important for the consistency, because the models with λk < 0 do not influence the asymptotic distribution of equation image, see Hansen (2001). However, the choice of Ak, n is not unique, and it is therefore useful to include the p-values of the two other tests, SPAl and SPAu, because they define the range of p-values that can be obtained by varying the choice for Ak, n. The p-values based on the tests statistic, equation image, are obtained similarly. These are denoted by RCl, RCc and RCu, where RCu corresponds to the original RC of White (2000).

6. DATA AND EMPIRICAL RESULTS

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE GARCH UNIVERSE
  5. 3. FORECAST EVALUATION
  6. 4. REALIZED VARIANCE
  7. 5. TEST FOR SUPERIOR PREDICTIVE ABILITY
  8. 6. DATA AND EMPIRICAL RESULTS
  9. 7. CONCLUSIONS
  10. Acknowledgements
  11. REFERENCES
  12. Supporting Information

The models are estimated by maximum likelihood using the estimation sample, and the model's forecasts are compared to the realized variance in the evaluation sample.

The first data set consists of DM–$ spot exchange rate data, where the estimation sample spans the period from October 1, 1987 through September 30, 1992 (1254 observations) and the out-of-sample evaluation sample spans the period from October 1, 1992 through September 30, 1993(n = 260). The realized variance data for the exchange rate have previously been analysed in Andersen and Bollerslev (1998) and are based on m = 288 intraday returns per day. See Andersen and Bollerslev (1997) for additional details. We adjust their measure of realized variance and use equation image, where ĉ = 0.8418 is defined in (1).

The second data set consists of IBM stock returns, where the estimation period spans the period from January 2, 1990 through May 28, 1999 (2378 days) and the evaluation period spans the period from June 1, 1999 through May 31, 2000 (n = 254). The realized variances were constructed from high-frequency data that were extracted from the Trade and Quote (TAQ) database. The intraday returns, rt, i, m, were constructed artificially by fitting a cubic spline to all mid-quotes of a given trading day, using the time interval 9 : 30 EST–16 : 00 EST.12 From the splines we extract f = 130 artificial three-minute returns per day (out of the hypothetical m = 480 three-minute returns) and calculate equation image. There are several other methods for constructing the realized variance and several of these are discussed in Andersen et al. (2003). Later we verify that our empirical results are not influenced by our choice of estimator, as we reach the same conclusions by using six other measures of the realized variance.

The estimate of the adjustment coefficient, (1), is ĉ = 4.4938, which exceeds 480/130 ≃ 3.7. This indicates that equation image underestimates the daily variance by more than would be expected if the daily volatility was evenly spread over the 24 hours of the day. There are several possible explanations to the fact that we need to adjust the volatilities by a number different than 3.7. First of all, it could be the result of sample variation, but this seems unlikely as n is too large for sampling error to explain this large a difference. A second explanation is that our intraday returns are positively autocorrelated. The autocorrelation can arise from the market microstructure effects or can be an artifact of the way intraday returns are constructed. A third explanation is that returns are relatively more volatile between close and open, than between open and close, measured per unit of time. This requires that more information arrives to the market while it is closed than while it is open. This contradicts the findings of French and Roll (1986) and Baillie and Bollerslev (1989), so we find this explanation to be unrealistic. Finally, a fourth factor that can create a difference between squared interday returns and the sum of squared intraday returns is the omission of the conditional expected value E(rt, i, m|t−1), i = 1, …, m in the calculations. Suppose that E(rt, i, m|t−1) = 0 for i = 1, …, f, but is positive during the time the market is closed. Then equation image would, on average, be larger than equation image, even if intraday returns were independent and homoskedastic. Such a difference between expected returns during the time the market is open and closed could be explained as a compensation for the lack of opportunities to hedge against risk overnight. It is not important which of the four explanations cause the difference, as long as our adjustment does not favour some models over others. Because the adjustment is made ex post and does not depend on the model forecasts, it is unlikely that a particular model would benefit more than other models.

6.1. Results from the Model Comparison

Table II contains the results from the model comparisons in the form of p-values.13 The p-values correspond to the hypothesis that the benchmark model, ARCH(1) or GARCH(1,1), is the best model. The naive p-value is the p-value that one would obtain by comparing the best performing model to the benchmark without controlling for the full set of models. So the naive p-value is not a valid p-value and it will often be too small, and therefore more likely to indicate an unjustified ‘significance’. The p-values of the SPA test and the RC control for the full set of models. Those of SPAc and RCc are asymptotically valid p-values, whereas those with subscript l and u provide lower and upper bounds for the p-values. Although the naive p-value is not valid, it can exceed that of the SPAc, because the best performing model need not be the model that results in the largest t-statistic.

Table II. Exchange rate data (DM/USD)
Panel A: Exchange rate data (DM/USD), SPA p-values
MetricBenchmark: ARCH(1)Benchmark: GARCH(1,1)
NaiveSPAlSPAcSPAuNaiveSPAlSPAcSPAu
  1. Notes: The table presents p-values of the SPA test and the RC for two null hypotheses: that the benchmark model, ARCH(1) or GARCH(1,1), is the best model. Conclusions should be based on the SPAc test (boldface) in Panels A and B. The naive ‘p-value’ compares the best performing model to the benchmark, but ignores the full set of models. So the naive ‘p-value’ is not a valid p-value and the difference between it and that of SPAc(RCc) shows the effects of data mining. Panel C contains the p-values that are based on the RC non-standardized test statistic. The p-values of the original RC are in boldface. A comparison of the results of Panels B and C shows that the SPA test is more powerful than the RC, and the latter is unable to detect the inferiority of the GARCH(1,1), and the ARCH(1) in some cases.

MSE10.00770.01790.01790.02090.29110.31640.45890.7887
MSE20.03920.06950.07480.07970.20250.60060.76520.9279
QLIKE0.00670.01690.01840.01940.25280.58310.77070.9639
R2LOG<0.00010.00020.00020.00020.07080.21440.32690.6627
MAE1<0.00010.00020.00020.00020.06360.22740.32960.6309
MAE20.00020.00110.00110.00120.18320.21770.29200.5663
Panel B: IBM Data, SPA p-values
MetricBenchmark: ARCH(1)Benchmark: GARCH(1,1)
NaiveSPAlSPAcSPAuNaiveSPAlSPAcSPAu
MSE10.00520.00020.00020.00020.03550.02450.03000.0358
MSE20.00610.00010.00010.00010.04090.02600.02880.0316
QLIKE0.0003<0.0001<0.0001<0.00010.02130.03790.04630.0528
R2LOG0.01080.00110.00110.00140.01660.05260.06300.0741
MAE10.00120.00800.00860.01040.00260.00400.00510.0058
MAE20.00140.00970.01000.01150.00260.00540.00650.0078
Panel C: IBM Data, RC p-values
MetricBenchmark: ARCH(1)Benchmark: GARCH(1,1)
NaiveRClRCcRCuNaiveRClRCcRCu
MSE10.00520.01640.01640.01640.03550.10000.14990.2811
MSE20.00610.02050.02050.02050.04090.10530.10560.1472
QLIKE0.00030.00170.00170.00170.02130.09430.11530.3750
R2LOG0.01080.06010.07130.07130.01660.29080.35350.6039
MAE10.00120.09720.12270.13990.00260.05050.11440.1522
MAE20.00140.12190.16490.19410.00260.06440.11350.1734

Panel A contains the results for the exchange rate data. The p-values clearly show that the ARCH(1) is outperformed by other models, although the MSE2 criterion is a possible exception. However, there is no evidence that the GARCH(1,1) is outperformed and a closer inspection of the models reveals that the GARCH(1,1) has one of the best sample performances.

Panels B and C contain the results from the IBM return data, based on the SPA test and the RC, respectively. From Panel B it is evident that both the ARCH(1) and the GARCH(1,l) are significantly outperformed by other volatility models in terms of all loss functions, with the possible exception of the R2LOG loss function. Thus there is strong evidence that the GARCH(1,1) is inferior to alternative models. The p-values in Panel C are based on the (non-standardized) test statistic equation image. The results in Panel C are alarmingly different from those in Panel B, because these p-values suggest the exact opposite conclusion in most cases. Panel C suggests that the GARCH(1,1) is not significantly outperformed, and even the ARCH(1) cannot be rejected as being superior to all other models for three of the six loss functions. The contradicting results are explained by the fact that the equation image is not properly standardized, and this causes the tests RCl, RCc and RCu to be sensitive to erratic models. The problem is that a model with a relatively large var(k) has a disproportional effect on the distribution of equation image, in particular the right tail which defines the critical values, see Hansen (2003). The p-values in the right-most column (boldface) are those of the original RC by White (2000), and these provide little evidence against the two benchmarks. So the results in Table II confirm that the RC is less powerful than the SPA test.

The realized variance can be constructed in many ways and different measures of the realized variance could lead to different results. To verify that our results are not sensitive to our choice of RV measure we repeat the empirical analysis of the IBM returns data using six other measures. These measures include: one based on a different spline method and sampling frequency; one based on the Fourier method by Barucci and Reno (2002); two based on the previous-tick method; and two based on the linear interpolation method. The p-values of the SPAc test for the seven different measures of the realized variance are presented in Table III. Fortunately, the p-values do not differ much across the various measures of the realized variance, although most of the alternative measures provide slightly stronger evidence that the GARCH(1,1) is outperformed in terms of the R2LOG loss function, and slightly weaker evidence in terms of the MAE1 and MAE2 loss functions.

Table III. Results for different measures of realized variance
CriterionMethod for estimating realized variance
Spl-50 3 minSpl-250 2 minFourier M = 85Linear 5 minPrevious 5 minLinear 1 minPrevious 1 min
  1. Notes: This table reports p-values of the SPAc test from the analysis of IBM returns where the GARCH(1,1) is used as the benchmark. The p-values are obtained for seven different measures of the realized variance that are constructed with different techniques (and sampling frequencies). Spl-50 and Spl-250 refer to a cubic spline method that use 50 and 250 knot points, respectively; the third measure is based on the Fourier method; and the last four measures are based on the linear interpolation and previous-tick methods.

MSE10.02710.02300.01340.01250.01330.01110.0103
MSE20.02800.02130.01350.01680.01810.00820.0082
QLIKE0.04570.03500.01660.01780.01750.01120.0118
R2LOG0.06510.09980.04620.04090.05050.03750.0340
MAE10.00390.06350.04760.06900.06620.09600.0881
MAE20.00560.08880.07240.05100.06000.07070.0749

Figures 1–4 show the ‘population’ of model performances for various loss functions (and the two data sets).14 The plots provide information about how similar/different the models' sample performances were, and show the location of the ARCH(1) and GARCH(1,1) relative to the full set of models. The x-axis is the (negative value of) average sample loss, such that the right tail represents the model with the best sample performance. Each figure contains four panels. The upper left panel is the model density of all the models, whereas the last three panels show the performance densities for different ‘types’ of models. The models are divided into groups according to their type: Gaussian vs. t-distributed specification; models with and without a leverage effect; and the three mean specifications.

thumbnail image

Figure 1. Population of model performance: exchange rate data and MSE2 loss function. The x-axis is the negative value of average sample loss

Download figure to PowerPoint

thumbnail image

Figure 2. Population of model performance: exchange rate data and MAE2 loss function. The x-axis is the negative value of average sample loss

Download figure to PowerPoint

thumbnail image

Figure 3. Population of model performance: IBM data and MSE2 loss function. The x-axis is the negative value of average sample loss

Download figure to PowerPoint

thumbnail image

Figure 4. Population of model performance: IBM data and MAE2 loss function. The x-axis is the negative value of average sample loss

Download figure to PowerPoint

Figures 1 and 2, which display the results for the exchange rate data, show that the GARCH(1,1) is one of the best performing models, whereas the ARCH(1) has one of the worst sample performances. There are no major differences between the various types of models, although there is a small tendency that the t-distributed specification leads to a better performance than a Gaussian specification in Figure 2.

The results for the IBM return data are displayed in Figures 3 and 4. From the SPA test we concluded that the GARCH(1,1) was significantly outperformed by other models, and the two figures also show that the GARCH(1,1) is ranked much lower in this sample. It now seems that the Gaussian specification does better than the t-distributed specification, on average. However, the very best performing model in terms of the MAE2 loss function is a model with a t-distributed specification. From our analysis of the IBM data it is evident that models that can accommodate a leverage effect are superior to those that cannot, particularly in Figure 4.

Although the conditional mean µt = E(rt|t−1) is likely to be small, it cannot ex ante be ruled out that a more sophisticated specification for µt, such as the GARCH-in-mean, leads to better forecasts of volatility than the zero-mean specification. However, the performance is almost identical across the three mean specifications, as can be seen from Figures 1–4.

7. CONCLUSIONS

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE GARCH UNIVERSE
  5. 3. FORECAST EVALUATION
  6. 4. REALIZED VARIANCE
  7. 5. TEST FOR SUPERIOR PREDICTIVE ABILITY
  8. 6. DATA AND EMPIRICAL RESULTS
  9. 7. CONCLUSIONS
  10. Acknowledgements
  11. REFERENCES
  12. Supporting Information

We have compared a large number of volatility models, in terms of their ability to forecast the conditional variance in an out-of-sample setting.

Our analysis was limited to DM–$ exchange rates and IBM stock returns and a universe of models that consisted of 330 different ARCH-type models. The main findings are that there is no evidence that the GARCH(1,1) model is outperformed by other models, when the models are evaluated using the exchange rate data. This cannot be explained by the SPA test lacking power because the ARCH(1) model is clearly rejected and found to be inferior to other models. In the analysis of IBM stock returns we found conclusive evidence that the GARCH(1,1) is inferior, and our results strongly suggested that good out-of-sample performance requires a specification that can accommodate a leverage effect.

The performances of the volatility models were measured out-of-sample using six loss functions, where realized variance was used to construct an estimate of the unobserved conditional variance. The significance of relative performance was evaluated with the test for superior predictive ability of Hansen (2001) and the reality check for data snooping of White (2000). Our empirical analysis illustrated the usefulness of the SPA test and showed that the SPA test is more powerful than the RC.

The SPA test and the RC are not model selection criteria and therefore not designed to identify the best volatility model (in population). It is also unlikely that our data contain sufficient information to conclude that the model with the best sample performance is significantly better than all other models. Nevertheless, the use of a significance test, such as the SPA test, has clear advantages over model selection criteria, because it allows us to make strong conclusions. In our setting, the SPA test provided conclusive evidence that the GARCH(1,1) is inferior to other models in our analysis of IBM returns. However, in the analysis of the exchange rate data, there was no evidence against the claim that: ‘nothing beats a GARCH(1,1)’.

Acknowledgements

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE GARCH UNIVERSE
  5. 3. FORECAST EVALUATION
  6. 4. REALIZED VARIANCE
  7. 5. TEST FOR SUPERIOR PREDICTIVE ABILITY
  8. 6. DATA AND EMPIRICAL RESULTS
  9. 7. CONCLUSIONS
  10. Acknowledgements
  11. REFERENCES
  12. Supporting Information

Financial support from the Danish Research Agency, grant no 24-00-0363, and the Salomon Research Award at Brown University is gratefully acknowledged. We thank Professor M. Hashem Pesaran (editor) and two anonymous referees for many suggestions that improved our paper, and we thank Tim Bollerslev for sharing the realized variance data for the exchange rate and Roberto Renò for constructing some of the realized variance data for the IBM returns. We also thank Tim Bollerslev, Frank Diebold, Rob Engle and seminar participants at Aarhus School of Business, Aarhus University, Brown University, Cornell University, University of Pennsylvania and ESEM 2002 for valuable comments. We are responsible for all remaining errors.

REFERENCES

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE GARCH UNIVERSE
  5. 3. FORECAST EVALUATION
  6. 4. REALIZED VARIANCE
  7. 5. TEST FOR SUPERIOR PREDICTIVE ABILITY
  8. 6. DATA AND EMPIRICAL RESULTS
  9. 7. CONCLUSIONS
  10. Acknowledgements
  11. REFERENCES
  12. Supporting Information
  • 1

    However, we have by no means included all the volatility models that have been proposed in the literature. For a comparison of a smaller set of models that also includes stochastic volatility models and fractionally integrated GARCH models, see Hansen et al. (2003).

  • 2

    We do not restrict υ to be an integer.

  • 3

    See Taylor (1986) and Schwert (1990).

  • 4

    Engle and Patton (2001) also point out that heteroskedastic returns imply (even more) heteroskedasticity in the squared returns, equation image. So parameters are estimated inefficiently and the usual standard errors are misleading.

  • 5

    Provided that a = 0 and b = 1, which essentially requires the forecasts to be unbiased.

  • 6

    In practice, m must be chosen moderately large, to avoid that intraday returns become correlated due to market microstructure effects. In the technical appendix (Hansen and Lunde, 2001), we list the R2 values from two MZ regressions, equation image and equation image, where the realized variance, equation image, is defined in the next section. The R2 of the former typically lies between 2 and 4%, whereas the R2 of the latter lies between 35 and 45%. This strongly suggests that equation image is a far more precise estimate of equation image than is equation image.

  • 7

    For other DGPs the equation image is consistent for the integrated variance, see Meddahi (2002) and Barndorff-Nielsen and Shephard (2001), which need not equal equation image. However, this does not change our main argument for using the realized variance, which is that equation image is a more precise estimator of equation image than is equation image.

  • 8

    Even the IGARCH model produces a stationary returns series {rt}, see Nelson (1990).

  • 9

    This procedure involves a dependence parameter, q, that serves to preserve possible time-dependence in Xt. We used q = 0.5 and generated B = 10, 000 bootstrap resamples in our empirical analysis.

  • 10

    The bootstrap variables are constructed such that equation image, and typically we have equation image.

  • 11

    The true p-value is defined as equation image, where t is the observed value of the test statistic and the probability is evaluated using the true (but unknown) values of λ and Ω.

  • 12

    This is done by applying the Splus routine called smooth.spline, which is a one-dimensional cubic smoothing spline that has a basis of B-splines, as discussed in chapters 1–3 of Green and Silverman (1994).

  • 13

    Additional results are given in a technical appendix (Hansen and Lunde, 2001).

  • 14

    To save space, we have only included the figures that correspond to the MSE2 and MAE2 loss functions. The figures for all six loss functions are given in Hansen and Lunde (2001).

Supporting Information

  1. Top of page
  2. Abstract
  3. 1. INTRODUCTION
  4. 2. THE GARCH UNIVERSE
  5. 3. FORECAST EVALUATION
  6. 4. REALIZED VARIANCE
  7. 5. TEST FOR SUPERIOR PREDICTIVE ABILITY
  8. 6. DATA AND EMPIRICAL RESULTS
  9. 7. CONCLUSIONS
  10. Acknowledgements
  11. REFERENCES
  12. Supporting Information

The JAE Data Archive directory is available at http://qed.econ.queensu.ca/jae/datasets/hansen002/ .

Please note: Wiley Blackwell is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.