SEARCH

SEARCH BY CITATION

Summary

  1. Top of page
  2. Summary
  3. 1 Introduction
  4. 2 Data
  5. 3 Models
  6. 4 Results
  7. 5 Conclusions
  8. Acknowledgements
  9. References
  10. Appendix
  11. Supporting Information

This paper compares alternative models of time-varying volatility on the basis of the accuracy of real-time point and density forecasts of key macroeconomic time series for the USA. We consider Bayesian autoregressive and vector autoregressive models that incorporate some form of time-varying volatility, precisely random walk stochastic volatility, stochastic volatility following a stationary AR process, stochastic volatility coupled with fat tails, GARCH and mixture of innovation models. The results show that the AR and VAR specifications with conventional stochastic volatility dominate other volatility specifications, in terms of point forecasting to some degree and density forecasting to a greater degree. Copyright © 2014 John Wiley & Sons, Ltd.

1 Introduction

  1. Top of page
  2. Summary
  3. 1 Introduction
  4. 2 Data
  5. 3 Models
  6. 4 Results
  7. 5 Conclusions
  8. Acknowledgements
  9. References
  10. Appendix
  11. Supporting Information

A growing number of studies have provided evidence of time-varying volatility in the economies of many industrialized nations. Regarding this point, most available evidence, based on data through the early to mid 2000s, has highlighted the Great Moderation (e.g. Stock and Watson 2003, 2007; Cogley and Sargent, 2005; Primiceri, 2005; Koop and Potter, 2007; Benati, 2008; Justiniano and Primiceri, 2008; Giordani and Villani, 2010). Some more recent studies have shown that, following the Great Moderation, volatility rose sharply during the severe recession of 2007–2009 (e.g. Clark, 2009, 2011; Curdia et al., 2013).

Modeling the apparently significant time variation in macroeconomic volatility is important to the accuracy of a range of types of inference. In general, of course, least squares estimates of vector autoregressive (VAR) coefficients remain consistent in the face of conditional heteroskedasticity, but ordinary least squares (OLS) variance estimates do not. Moreover, modeling the conditional heteroskedasticity can yield more efficient generalized least squares (GLS) estimates of VAR coefficients; Sims and Zha (2006) have emphasized the value of volatility modeling for improving efficiency. Accordingly, in both dimensions, taking account of time variation in volatility should improve the VAR-based estimation and inference common in macroeconomic analysis. In particular, in VAR-based analysis of impulse responses, variance decompositions, and historical decompositions—used, for instance, to assess the effects of alternative monetary policies—modeling time variation in conditional volatilities is likely to be important for accurate inferences. In addition, some recent dynamic stochastic general equilibrium (DSGE) model research (e.g. Fernandez-Villaverde and Rubio-Ramirez, 2007; Justiniano and Primiceri, 2008) has emphasized the importance of modeling time variation in volatility for explaining the sources of the Great Moderation and other changes in volatility.

Modeling changes in volatility should also help to improve the accuracy of density forecasts from VARs. Shifts in volatility have the potential to result in forecast densities that are either far too wide or too narrow. For instance, in light of the Great Moderation, density forecasts for gross domestic product (GDP) growth in 2006 based on time series models assuming constant variances over a sample such as 1960–2005 would probably be far too wide, with inflated confidence intervals and probabilities of tail events such as recession. As another example, in late 2008, density forecasts for 2009 based on time series models assuming constant variances for 1985–2008 would have been too narrow. Results in Giordani and Villani (2010), Jore et al. (2010) and Clark (2011) support this intuition on the gains to point and density forecasts of modeling shifts in conditional volatilities. D'Agostino et al. (2013) show that the combination of time-varying parameters and stochastic volatility improves the accuracy of point and density forecasts. These benefits to allowing time-varying volatility could prove useful to central banks that provide density information in the form of forecast fan charts and qualitative assessments of forecast uncertainty.

In most recent macro-econometric studies of time-varying volatility (e.g. Stock and Watson, 2003, 2007; Cogley and Sargent, 2005; Primiceri, 2005; Benati, 2008), the time variation in volatility has been captured with a single model: stochastic volatility, in which log volatility follows a random walk process. In Bayesian estimation algorithms, the stochastic volatility specification is computationally tractable. In addition, studies such as Clark (2011) and Carriero et al. (2012) have shown that it is effective for improving the accuracy of density forecasts from AR and VAR models. However, there are alternatives that could also be effective for capturing changes in macroeconomic volatility. 1 Studies such as Koop and Potter (2007), Giordani and Villani (2010) and Groen et al. (2013) have used models in which volatility is subject to potentially many discrete breaks; others, such as Jore et al. (2010), have used models with a small number of discrete breaks. Yet another model of time-varying volatility would be a generalized autoregressive conditional heteroskedasticity (GARCH) specification. While the pioneering development of ARCH (Engle, 1982) and GARCH (Bollerslev, 1986) included applications to inflation, these models seem to have become rare in macro-econometric modeling, with the exception of a few studies, such as Canarella et al. (2010) and Chung et al. (2012).

While a number of studies in the finance literature have compared alternative models of time-varying volatility of asset returns (e.g. Hansen and Lunde, 2005; Geweke and Amisano, 2010; and Nakajima, 2012), no such broad comparison yet exists for macroeconomic variables. 2 Accordingly, this paper compares alternative models of time-varying macroeconomic volatility, included within autoregressive and vector autoregressive specifications for key macroeconomic indicators and estimated using Bayesian inference. We base our comparison on real-time out-of-sample forecast accuracy, for both point and density forecasts of US data on GDP growth, the unemployment rate, inflation in the GDP deflator and the 3-month Treasury bill rate. 3

The set of univariate AR models includes the following volatility specifications: constant volatility; stochastic volatility (with both constant coefficients in the conditional mean portion of the model and time-varying coefficients in the conditional mean); stochastic volatility following a stationary AR process; stochastic volatility coupled with fat tails; GARCH; and a mixture of innovations model. The set of VARs includes the same volatility specifications, except for the mixture of innovations model (for reasons of computational tractability).

Our results indicate that the AR and VAR specifications with stochastic volatility dominate models with alternative volatility specifications, in terms of point forecasting to some degree and density forecasting to a greater degree. Therefore, at least from a macroeconomic forecasting perspective, these alternative volatility specifications seem to have no advantage over the now widely used stochastic volatility specification.

The paper proceeds as follows. Section 2 describes the data. Section 3 presents the models and estimation methodology; details on priors are provided in the Appendix; and details of estimation algorithms and some addition results are provided as supporting information in a supplementary Appendix. Section 4 presents the results. Section 5 concludes.

2 Data

  1. Top of page
  2. Summary
  3. 1 Introduction
  4. 2 Data
  5. 3 Models
  6. 4 Results
  7. 5 Conclusions
  8. Acknowledgements
  9. References
  10. Appendix
  11. Supporting Information

We use quarterly data to estimate models for growth of real GDP, inflation in the GDP price index or deflator (henceforth, GDP inflation), unemployment rate and 3-month Treasury bill rate. We compute GDP growth as 100 times the log difference of real GDP and inflation as 100 times the log difference of the GDP price index, to put them into units of percentage point changes. The unemployment rate and interest rate are also defined in units of percentage points (annualized in the case of the interest rate).

We obtained quarterly real-time data on GDP and the GDP price index from the Federal Reserve Bank of Philadelphia's Real Time Dataset for Macroeconomists. For simplicity, we use ‘GDP’ and ‘GDP price index’ to refer to the output and price series, even though the measures are based on gross national product (GNP) and a fixed weight deflator for much of the sample. As described in Croushore and Stark (2001), the vintages of the RTDSM (Real-Time Data Set for Macroeconomists) are dated to reflect the information available around the middle of each quarter. In vintage t, the available data run through period t − 1.

In the case of unemployment and interest rates, for which real-time revisions are small to essentially non–existent, we simply abstract from real-time aspects of the data and use currently available time series. We obtained monthly data on the unemployment rate and 3-month Treasury bill rate from the FAME database of the Federal Reserve Board of Governors and formed the quarterly unemployment and interest rate as simple within-quarter averages of the monthly data.

As discussed in such sources as Romer and Romer (2000), Sims (2002) and Croushore (2006), evaluating the accuracy of real-time forecasts requires a difficult decision on what to take as the actual data in calculating forecast errors. 4 We follow studies such as Romer and Romer (2000) and Faust and Wright (2009) and use the second available estimates of GDP/GNP and the GDP/GNP deflator as actuals in evaluating forecast accuracy. In the case of h-quarter-ahead forecasts made for period t + H with vintage t data ending in period t − 1, the second available estimate is taken from the vintage t + h + 2 data set. In light of our abstraction from real-time revisions in the unemployment and interest rates, for these series the real-time data correspond to the final vintage data.

We evaluate forecasts from 1975:Q1 to 2011:Q2, which requires real-time data vintages from 1975:Q1 to 2011:Q4. For each forecast origin t starting with 1975:Q1, we use the real-time data vintage t to estimate the forecast models and construct forecasts of quarterly values of all variables for periods t and beyond. 5 We report results for forecast horizons of 1, 2, 4, and 8 quarters ahead. In light of the time t − 1 information actually incorporated in the models used for forecasting at t, the 1-quarter-ahead forecast is a current quarter (t) forecast, while the 2-quarter-ahead forecast is a next-quarter (t + 1) forecast, etc. For most models, the starting point of the model estimation sample is 1955:Q1; in some of these specifications, we use data for the 1948–1954 period to set the priors on some parameters, as detailed in the Appendix. For the VAR-TVP-SV and AR-TVP-SV specifications, to permit the use of a longer training sample for setting the prior on the initial VAR or AR coefficients, the starting point of the model estimation sample is 1961:Q1 and we use data for the 1948–1960 period to set the priors on some parameters.

3 Models

  1. Top of page
  2. Summary
  3. 1 Introduction
  4. 2 Data
  5. 3 Models
  6. 4 Results
  7. 5 Conclusions
  8. Acknowledgements
  9. References
  10. Appendix
  11. Supporting Information

This section provides the specifications of our models and an overview of the estimation methods. 6 Because in most cases the AR models are simplifications of corresponding VAR models, we present the VAR models first and then provide a much briefer presentation of AR models. The priors are detailed in the Appendix, and the estimation algorithms (which are very similar to those in other studies) are detailed in the supplementary Appendix.

The volatility specifications we include reflect a range of considerations. While other studies have provided strong evidence of time-varying volatility in macroeconomic variables, the prevalence of VAR and AR models with constant volatility leads us to include these specifications in our analysis. These models are of course easy to use for forecasting, and for point forecasting their performance should not be affected much by conditional heteroskedasticity (we return to this point in our analysis of results).

Among possible specifications of time-varying volatility, we treat random-walk stochastic volatility (simply ‘stochastic volatility’ in what follows) as a baseline because, since the pioneering work of Cogley and Sargent (2005) and Primiceri (2005), it has become the dominant approach in recent macro-econometric modeling. However, there are reasons to think this particular specification within the class of stochastic volatility models could be too restrictive in some dimensions. First, log volatility could follow a stationary, low-order AR process rather than a random walk. In the recent literature on DSGE models with stochastic volatility, some studies model log volatility as a random walk (e.g. Justiniano and Primiceri, 2008), while others model it as a stationary AR(1) process (e.g. Fernandez-Villaverde and Rubio-Ramirez, 2007). In forecasting, the random walk specification might adversely affect density forecast performance by allowing volatility to blow up (becoming either unduly high or low) over the forecast horizon. Second, in light of the dramatic movements in macroeconomic variables over the recent Great Recession, the standard stochastic volatility specification could miss fat tails in the underlying shock distribution. Curdia et al. (2013) find that adding fat tails to stochastic volatility improves the fit of a DSGE model for the US economy. Motivated by these considerations, we examine the forecasting performance of VAR and AR models with (random walk) stochastic volatility, stationary AR(1) stochastic volatility, and both fat tails and stochastic volatility. 7 In light of other evidence of time variation in the regression coefficients of VAR and AR models (e.g. Cogley and Sargent, 2005), we also consider VAR and AR models with both stochastic volatility and time-varying (regression) parameters.

Despite the prevalence of stochastic volatility in recent macro-econometric modeling, VAR and AR models with GARCH could reasonably be considered as alternatives. While GARCH has become prevalent over time in finance modeling (e.g. Geweke and Amisano, 2010) and not common in macro modeling, Engle (1982) and Bollerslev (1986) had inflation in mind with their development of AR and GARCH models. Moreover, there are some recent macroeconomic analyses that have used GARCH formulations (e.g. Canarella et al., 2010; Chung et al., 2012). In DSGE-based modeling, while Fernandez-Villaverde-and-Rubio-Ramirez (2010) describe reasons to prefer stochastic volatility to GARCH (primarily because GARCH makes it more difficult to separate volatility shocks from levels shocks), there are some DSGE applications that consider GARCH (Andreasen, 2012).

Finally, within the set of AR models, we consider a specification in which volatility is subject to potentially many discrete breaks, rather than to the continuous breaks implied by stochastic volatility or GARCH specifications. For reasons described in Koop and Potter (2007), for example, a specification with potentially large, discrete breaks may be conceptually preferable to models with small, frequent breaks. Our particular specification is much more readily applied to AR models than VAR models, so we only consider an AR specification: a model that takes the mixture of innovations form, developed in studies such as Giordani et al. (2007), Koop and Potter (2007) and Groen et al. (2013).

While the mixture model we consider is similar to a Markov switching model, for computational reasons we omit a direct comparison to switching models, leaving such a comparison for future research. Switching can be difficult to use with vector autoregressions, which has limited their use (see, for example, discussions in Bognanni, 2013, and Hubrich and Tetlow, 2012). Given the current state of the art, a VAR with Markov switching would pose a significant computational challenge in a real-time forecasting evaluation spanning more than 140 quarters.

3.1 VAR Models

While we write out the models for a general lag order p, all of the VAR models include four lags, except that, to streamline computations, the VAR-TVP-SV model includes two lags, following studies such as Cogley and Sargent (2005) and D'Agostino et al. (2013).

3.1.1 Constant Volatility

Let yt denote the k × 1 vector of model variables, B0 = a k × 1 vector of intercepts and Bi,i = 1, … ,p a k × k matrix of coefficients on lag i. For our set of k = 4 variables, we consider a VAR(p) model with a constant variance–covariance matrix of shocks:

  • display math(1)
3.1.2 Stochastic Volatility

The VAR-SV model includes the conventional macro-econometric formulation of a random walk process for log volatility:

  • display math(2)
  • display math

where A = a lower triangular matrix with ones on the diagonal and non-zero coefficients below the diagonal, and the diagonal matrix Λt contains the time-varying variances of underlying structural shocks. 8 This model implies that the reduced form variance–covariance matrix of innovations to the VAR is var(vt) ≡ Σt = A − 1ΛtA − 1 ′ . Note that, as in Primiceri's (2005) implementation, innovations to log volatility are allowed to be correlated across variables (not only in this baseline stochastic volatility specification but also the other specifications detailed below). 9 Thus Φ is not restricted to be diagonal.

3.1.3 Stationary ar(1) Stochastic Volatility

The VAR-stationary SV specification treats log volatility as following an AR(1) process, which we presume to be stationary: 10

  • display math(3)
3.1.4 Stochastic Volatility with Fat Tails

The VAR-SVt model augments the (random walk) stochastic volatility specification to include fat tails, similarly to the DSGE specification considered in Curdia et al. (2013), which follows the stochastic volatility with fat tails formulation of Jacquier et al. (2004):

  • display math(4)

where di denotes the degrees of freedom of the Student-t distribution that is the marginal distribution of inline image. Fat tails arise due to the qi,t, which are assumed to be independent over time and across variables. We consider two treatments of the degrees of freedom of the fat tails component. In the first (used with what we identify as the ‘VAR-SVt’ model), di is a parameter to be estimated (for each variable). In the second (the ‘VAR-SVt, 5 df’ specification), di is simply fixed at 5, to ensure fat tails.

3.1.5 Time-Varying Parameters and Stochastic Volatility

Letting Xt denote the collection of right-hand-side variables of each equation of the VAR and Bt denote the period t value of the vector of all VAR coefficients (of dimension k(kp + 1) × 1), the VAR-TVP-SV model takes the form given in Cogley and Sargent (2005):

  • display math(5)

where A = a lower triangular matrix with ones on the diagonal and non-zero coefficients below the diagonal. The VAR coefficients follow random-walk processes, with innovations that are allowed to be correlated across coefficients. The volatility portion of the VAR-TVP-SV model is the same as that of the VAR-SV specification.

3.1.6 GARCH

The VAR-GARCH model incorporates a standard GARCH(1,1) process (as in Chung et al., 2012, for example) for the orthogonalized error of each VAR equation: 11

  • display math(6)

where A = a lower triangular matrix with ones on the diagonal and non-zero coefficients below the diagonal. In this GARCH formulation, each variable is treated independently, with the conditional variance hi,t a function of one lag of itself and one lag of the squared error from the VAR equation. We impose conditions to ensure positivity and stationarity of each volatility process.

3.2 AR Models

All of our AR models include two lags for GDP growth and four lags for unemployment, inflation and the T-bill rate. 12 We consider AR models with constant volatility (AR), stochastic volatility (AR-SV), stationary stochastic volatility (AR-stationary SV), stochastic volatility with fat tails (AR-SVt, using estimated degrees of freedom and a fixed 5 degrees of freedom), stochastic volatility with time-varying parameters (AR-TVP-SV) and GARCH (AR-GARCH). For all of these models, the AR specification is the same as the corresponding VAR specification, simplified to a univariate setting (k = 1), which eliminates the A matrix and makes Φ, Λt, Qt and Ht scalars instead of matrices. Accordingly, in the interest of brevity, for these models we omit the AR specification details and instead refer the reader to the VAR specification details given above. The only model for which we spell out details is the AR-mixture specification.

3.2.1 Mixture of Innovations

The AR-mixture model is specified as follows, for each scalar time series yt:

  • display math(7)

In this model, the constants πj are the probabilities of breaks in each period, for each parameter j (either a coefficient or the log volatility). If a break occurs to parameter j, the parameter shifts by an innovation nj, which has variance qj.

3.3 Estimation Algorithms and Sampling of Forecasts

We estimate all of the models described above using Bayesian Markov chain Monte Carlo (MCMC) methods. This section provides a brief overview of our methods. The supplementary Appendix and the studies cited below provide additional detail on algorithms and priors.

For the AR and VAR models with constant variances, we use the Normal-diffuse prior and posterior detailed in such sources as Kadiyala and Karlsson (1997) and estimate the models by Gibbs sampling.

To estimate the AR-SV, AR-stationary SV, AR-TVP-SV, VAR-SV, VAR-stationary SV and VAR-TVP-SV models, we use Gibbs samplers. Stochastic volatility is estimated with the algorithm of Kim et al. (1998), as detailed in Primiceri (2005). 13 For the VAR-TVP-SV and AR-TVP-SV models, our algorithm is the same as Primiceri's (2005), but simplified to treat the A matrix as constant as in Cogley and Sargent (2005). For the VAR-SV model, the volatility portion of the model is handled as it is in the case of the VAR-TVP-SV specification. The VAR coefficients are drawn from a conditional posterior distribution that is multivariate normal, with a GLS-based mean and variance given in Clark (2011). For the VAR-stationary SV specification, we add to the VAR-SV algorithm a step to draw the coefficients of the AR(1) process for each variable's log volatility from a conditional posterior distribution that (like the prior) is multivariate normal. Because the innovations to log volatility can be correlated across variables, the conditional posterior of these coefficients corresponds to that for a seemingly unrelated regression system of equations.

To estimate the AR-SVt and VAR-SVt models, we extend the Gibbs sampling algorithm used for the VAR-SV specification to accommodate fat tails, following Jacquier et al. (2004). The key extension is the addition of a step to draw, for each variable, the time series of qi,t from an inverse Gamma distribution. The other steps are the same as those of the VAR-SV algorithm, but for a few normalizations of data or innovations to reflect the qi,t terms. For the case in which we estimate the degrees of freedom, we rely on an exponential prior for the degrees of freedom and conditional posterior that requires a Metropolis step (for this, we use the implementation of Koop, 2003), treating each variable independently.

For the AR-GARCH and VAR-GARCH models, we use a Metropolis-within-Gibbs MCMC algorithm, combining Gibbs sampling steps for model coefficients with a random walk Metropolis–Hastings (MH) algorithm to draw the GARCH parameters. Our MH algorithm for the GARCH parameters is similar to those in Vrontos et al. (2000) and So et al. (2005). Specifically, we employ an adaptive MH-MCMC algorithm that combines a random-walk Metropolis (RW-M) and an independent kernel (IK) MH algorithm. In the case of the VAR-GARCH model, the Choleski matrix A is handled in the same way as it is in the VARs with stochastic volatility.

Finally, our approach to estimating the AR-mixture model is taken from Groen et al. (2013). The steps in their Gibbs sampler include: using the algorithm of Gerlach et al. (2000) to sample the latent states κj,t that indicate the timing of breaks in the coefficients and variance; using the simulation smoother of Carter and Kohn (1994) to sample the regression parameters; and using the algorithm of Kim et al. (1998) to draw the time-varying volatility and the variance of innovations to volatility.

All of our reported results are based on samples of 5000 posterior draws, retained from larger samples of draws. However, we use different burn periods and thinning intervals for different models, depending on the mixing properties of the algorithms (drawing on our own results on mixing properties and others in the literature, such as Carriero et al., 2012). Details on the burn samples and thinning intervals are given in Table 1 of the supplementary Appendix.

The posterior distributions of forecasts reflect the uncertainty due to all parameters of each model and shocks occurring over the forecast horizon. For example, to simulate the predictive density of the VAR-TVP-SV specification, we follow the approach of Cogley et al. (2005). From a forecast origin of period t, for each retained draw of the time series of Bt up through t, Λt up through t, A, Q and Φ, we: (i) draw innovations to coefficients for periods t + 1 to t + H (H = the maximum forecast horizon considered) from a normal distribution with variance–covariance matrix Q and use the random-walk structure to compute BT + 1, … ,BT + H; (ii) draw innovations to log volatilities for periods t + 1 to t + H from a multivariate normal distribution with variance–covariance matrix Φ and use the random-walk model of logλt + h to compute λT + 1, … ,λT + H; (iii) draw innovations to Yt + h, h = 1, … ,H, from a normal distribution with variance ΣT + h = A − 1ΛT + hA − 1 ′ , and use the vector autoregressive structure of the model along with the time series of coefficients BT + h to obtain draws of Yt + h, h = 1, … ,H. The draws of Yt + h are used to compute the forecast statistics of interest. To take another example, for the VAR-SV model, we use the same approach to simulating the predictive distribution, except that the steps for simulating time series of the VAR coefficients are eliminated. 14

4 Results

  1. Top of page
  2. Summary
  3. 1 Introduction
  4. 2 Data
  5. 3 Models
  6. 4 Results
  7. 5 Conclusions
  8. Acknowledgements
  9. References
  10. Appendix
  11. Supporting Information

In light of the key role that time-varying volatility will play in our results, it is useful to begin with a review of macroeconomic volatility over time. 15 For that purpose, we use our last vintage of data to estimate the VAR and VAR-SV models over a sample of 1955:Q1–2011:Q3. As a simple baseline, for the VAR model, we compute the residuals at the posterior mean of the coefficients and estimate rolling windows (41-quarter centered moving averages) of standard deviations of the reduced-form residuals. For the VAR-SV model, we obtain time series of the standard deviations of reduced-form shocks to each variable (from the diagonals of Σt = A − 1ΛtA − 1 ′ ) by computing the standard deviations at each draw and then computing the posterior medians and 70% credible sets.

The volatility estimates reported in Figure 1 display considerable variation over time, with some fairly significant co-movement. The simple rolling window estimates of residual standard deviations from the VAR show the volatility of GDP growth and unemployment declining through the 1960s, rising sharply until about 1980, plunging with the Great Moderation, and then rising again at the end of the moving-average sample window, reflecting the Great Recession of 2007–2009. The rolling window estimates of volatility in inflation and the T-bill rate trend up from early in the sample through roughly 1980 and then follow a pattern similar to that for growth and unemployment. The estimates of standard deviations from the VAR-SV display broadly similar patterns, but with sharper and somewhat more frequent movements than are evident in the rolling window estimates from the VAR. In either case, the co-movement of macroeconomic volatility appears to be high; the correlations of the volatility estimates from the VAR-SV range (across variables) from 0.77 to 0.96. Overall, this full-sample evidence points to important variation and co-movement in volatility that models likely need to capture to succeed in density forecasting.

image

Figure 1. Full-sample estimates of standard deviations of reduced form residuals, from VAR-SV and VAR. The black line in each chart is the posterior median of the standard deviation of the reduced-form residual of the VAR-SV model (i.e. the posterior median of the square root of one of the diagonal elements of Σt); the blue lines present the 70% credible set. The green line is the standard deviation of the reduced-form residual of the VAR, for which the residuals were computed at the posterior mean of the VAR coefficients obtained for the full sample of 1955:Q1–2011:Q3, and then standard deviations were computed over 41-quarter centered moving average windows.

Download figure to PowerPoint

4.1 Forecast Metrics

Turning to the real-time out-of-sample forecast comparison that is the focus of the paper, for the purpose of assessing the efficacy of alternative models of time-varying volatility, we separate our comparisons of AR models from our comparisons of VAR models. We use a recursive forecasting scheme, expanding the model estimation sample as forecasting moves forward in time. We provide results for a full sample of 1975:Q1–2011:Q2 and for a Great Moderation sample period of 1985:Q1–2007:Q4 (included in part for comparability to samples used by other studies). Because the results are broadly similar across these samples, we cover them jointly, rather than separately, in our discussion. The section proceeds by detailing our approaches to comparing forecasts and then presenting the results.

We first consider the accuracy of point forecasts (defined as posterior medians), using root mean square errors (RMSEs). We then consider density forecasts, using both the average log predictive score and the average continuous ranked probability score (CRPS). The predictive score, motivated and described in such sources as Geweke and Amisano (2010), is commonly viewed as the broadest measure of density accuracy. We compute the log predictive score using the quadratic approximation of Adolfson et al. (2007):

  • display math(8)

where inline image denotes the observed outcome, inline image denotes the posterior mean of the forecast distribution, and Vt + h | t denotes the posterior variance of the forecast distribution.

As indicated in Gneiting and Raftery (2007) and Gneiting and Ranjan (2011), some researchers view the continuous ranked probability score as having advantages over the log score. 16 In particular, the CRPS does a better job of rewarding values from the predictive density that are close to but not equal to the outcome, and it is less sensitive to outlier outcomes. The CRPS, defined such that a lower number is a better score, is given by

  • display math(9)

where f denotes the cumulative distribution function associated with the predictive density f, inline image denotes an indicator function taking value 1 if inline image and 0 otherwise, and Yt + h and inline image are independent random draws from the posterior predictive density.

To facilitate the reading of results from tables, we present the RMSEs, log scores and CRPS for benchmark models and relative-to-baseline measures of RMSEs, scores and CRPS for other models. In light of existing research findings that show random walk stochastic volatility typically improves the accuracy of forecasts from AR and VAR models (e.g. Clark, 2011; D'Agostino et al., 2013), we take models with random walk stochastic volatility (AR-SV for univariate specifications and VAR-SV for VAR specifications) as baselines and compare models with constant volatility or other time-varying volatility specifications to the baselines. More specifically, in our tables, for the baseline AR and VAR models with stochastic volatility, we report the RMSEs, average log scores and average CRPS. For the other AR (VAR) models, we report: ratios of each model's RMSE to the baseline AR-SV (VAR-SV) model, such that entries less than 1 indicate that the given model yields forecasts more accurate than those from the baseline; differences in score relative to the AR-SV (VAR-SV) baseline, such that a positive number indicates a model beats the baseline; and ratios of each model's average CRPS relative to the baseline AR-SV (VAR-SV) model, such that entries less than 1 indicate that the given model performs better.

To provide a rough gauge of whether the differences in forecast accuracy are significant, we apply Diebold and Mariano (1995) t-tests for equality of the average loss (with loss defined as squared error, log score,or CRPS). 17 In the tables, differences in accuracy that are statistically different from zero are denoted by one, two or three asterisks, corresponding to significance levels of 10%, 5% and 1%, respectively. The underlying p-values are based on t-statistics computed with a serial correlation-robust variance, using the pre-whitened quadratic spectral estimator of Andrews and Monahan (1992). Our use of the Diebold–Mariano test with forecasts from models that are, in many cases, nested is a deliberate choice. Monte Carlo evidence in Clark and McCracken (2011a,b) indicates that, with nested models, the Diebold–Mariano test compared against normal critical values can be viewed as a somewhat conservative (in the sense of tending to have size modestly below nominal size) test for equal accuracy in the finite sample. Since the AR-SV (VAR-SV) model nests the AR (VAR) model, for the comparison of the AR (VAR) model to the AR-SV (VAR-SV) specification, we report p-values based on one-sided tests, taking the AR (VAR) as the null and the AR-SV (VAR-SV) as the alternative. Because the AR-SV and AR-GARCH (VAR-SV and VAR-GARCH) models are not nested, for these model comparisons we report p-values based on two-sided tests. Since the other models considered nest the AR-SV and VAR-SV baselines, for the remaining comparisons we treat each model as nesting the baseline, and we report p-values based on one-sided tests, taking the AR-SV (VAR-SV) as the null and the other model in question as the alternative.

4.2 Point Forecasts: RMSEs

The results in Table 1 indicate that, for AR models, a model with stochastic volatility yields point forecasts that, in general, are either about as accurate or more accurate than forecasts from a model with constant volatility. For inflation and the interest rate, forecasts from the AR-SV model are more accurate than forecasts from the AR model, with differences that are usually statistically significant. In the case of GDP growth and unemployment, the AR-SV forecasts are about as good as or a little less accurate than the AR model forecasts. As examples, at the 4-quarter-ahead horizon in the 1975–2011 sample, the ratio of the AR RMSE to the AR-SV RMSE is 0.997 for GDP growth and 0.962 for unemployment, while the corresponding ratios are 1.041 for inflation and 1.061 for the interest rate.

Table 1. Real-time forecast RMSEs (RMSEs for AR-SV and VAR-SV benchmarks, RMSE ratios in all others)
 GDP growth, 1975:Q1–2011:Q2GDP growth, 1985:Q1–2007:Q4
h = 1Qh = 2Qh = 4Qh = 8Qh = 1Qh = 2Qh = 4Qh = 8Q
  1. Notes:

    1. In each quarter t from 1975:Q1 to 2011:Q2, vintage t data (which end in t − 1) are used to form forecasts for periods t to t + 7, corresponding to horizons of 1–8 quarters ahead. The forecast errors are calculated using the second-available (real-time) estimates of growth and inflation and currently available measures of unemployment and the short-term interest rate as the actuals. All variables are defined in percentage points (annualized in the case of the interest rate).

  2. 2. The models are detailed in Section 3. The notation ‘VAR-stationary SV’ refers to a VAR model in which log volatility follows an AR(1) process. The notation ‘VAR-SVt’ refers to a VAR model with stochastic volatility and fat tails, in which d denotes the degrees of freedom of the Student-t distribution that is the marginal distribution of innovations to the model; the degrees of freedom are estimated in the case of the ‘VAR-SVt’ specification and fixed at 5 in the case of the ‘VAR-SVt, 5 df’ specification. All forecasts are produced with recursive estimation of the models.

  3. 3. For the baseline AR-SV and VAR-SV models, the table reports the RMSEs (first row of each panel). For all other AR models, the table reports the ratio of each model relative to the RMSE of the AR-SV baseline. For the VAR models, the table reports the ratio of each model relative to the RMSE of the VAR-SV baseline. Entries less than 1 indicate that forecasts from the indicated model are more accurate than forecasts from the associated baseline model.

  4. 4. To provide a rough gauge of whether the RMSE ratios are significantly different from 1, we use the Diebold–Mariano t-statistic for equal MSE. Differences in accuracy that are statistically different from zero are denoted by asterisks, corresponding to significance levels of *10%, **5% and ***1%. The underlying p-values are based on t-statistics computed with a serial correlation-robust variance, using the pre-whitened quadratic spectral estimator of Andrews and Monahan (1992). Since the AR-SV (VAR-SV) model nests the AR (VAR) model, for the comparison of the AR (VAR) model to the AR-SV (VAR-SV) specification, we report p-values based on one-sided tests, taking the AR (VAR) as the null and the AR-SV (VAR-SV) as the alternative. Because the AR-SV and AR-GARCH (VAR-SV and VAR-GARCH) models are not nested, for these model comparisons we report p-values based on two-sided tests. Since the other models considered nest the AR-SV and VAR-SV baselines, for the remaining comparisons we treat each model as nesting the AR-SV or VAR-SV baseline, and we report p-values based on one-sided tests, taking the AR-SV (VAR-SV) as the null and the other model in question as the alternative.

Univariate models
AR-SV0.7420.7480.7450.7470.4480.4610.4880.491
AR1.0031.0060.9970.9950.9860.9900.9760.974
AR-GARCH0.982**1.0131.0031.0050.9900.9920.983**1.004
AR-mixture1.0211.0731.0621.0711.0201.0291.0651.133
AR-stationary SV1.0021.0020.9960.9961.0000.9950.987**0.988**
AR-SVt, 5 df1.0041.0101.0061.0061.0121.0161.0101.016
AR-SVt1.0021.0041.0021.0041.0061.0071.0071.005
AR-TVP-SV1.0241.0441.0241.0191.0081.0061.0211.050
Multivariate models
VAR-SV0.7380.7170.7200.7320.4810.4950.4920.473
VAR1.069**1.078*1.0480.9891.066***1.059**0.9970.973
VAR-GARCH1.138***1.265***1.195***1.082***1.132**1.139**1.1441.016
VAR-stationary SV1.0181.0141.0120.9891.0191.0190.9980.994
VAR-SVt, 5 df1.0080.9981.0001.0061.0071.0040.979***0.993*
VAR-SVt1.0020.9981.0031.0021.0001.0010.992**0.998
VAR-TVP-SV0.9760.9750.9751.0000.9920.951**0.9911.005
 Unemployment, 1975:Q1–2011:Q2Unemployment, 1985:Q1–2007:Q4
h = 1Qh = 2Qh = 4Qh = 8Qh = 1Qh = 2Qh = 4Qh = 8Q
Univariate models
AR-SV0.3140.5531.0181.7040.1640.2920.5370.932
AR0.9610.9640.9620.9100.9850.9920.9980.978
AR-GARCH0.9050.9270.9780.9291.0201.0151.0140.939
AR-mixture0.9621.1001.3021.5281.1121.1771.2831.366
AR-stationary SV1.0221.0271.0201.0171.0161.0231.0231.015
AR-SVt, 5 df0.993**0.9980.9990.994*1.0011.0031.0020.998
AR-SVt0.997*1.0000.9990.997*1.0021.0001.0000.999
AR-TVP-SV0.892**0.912*0.9530.9380.9830.9820.9810.993
Multivariate models
VAR-SV0.2860.5090.9221.3870.1530.2700.4950.752
VAR1.0101.0311.0501.0041.049**1.072*1.0820.986
VAR-GARCH1.071**1.142***1.149**1.120*1.100*1.1471.1401.043
VAR-stationary SV1.0071.0121.0181.0151.0171.0171.0180.995
VAR-SVt, 5 df1.0081.0141.0201.0071.0161.0151.0201.001
VAR-SVt1.0051.0061.0091.0051.0071.0051.0091.000
VAR-TVP-SV0.9470.9690.9840.9930.9710.9610.9721.051
 Inflation, 1975:Q1–2011:Q2Inflation, 1985:Q1–2007:Q4
h = 1Qh = 2Qh = 4Qh = 8Qh = 1Qh = 2Qh = 4Qh = 8Q
Univariate models
AR-SV0.3080.3390.3720.4780.2460.2590.2590.317
AR1.035***1.035***1.041**1.046*1.016**1.038***1.065***1.119***
AR-GARCH1.059**1.061***1.090**1.1021.0271.0221.072**1.096**
AR-mixture1.1371.1421.1631.1211.0360.9871.0190.993
AR-stationary SV1.0081.0050.9971.0071.0071.0141.0291.049
AR-SVt, 5 df1.0131.0010.9991.0161.0101.0111.0121.022
AR-SVt1.0041.0020.9981.0031.0051.0041.0091.006
AR-TVP-SV1.0061.0080.9650.9731.0010.9730.9910.930
Multivariate models
VAR-SV0.3090.3520.4130.5920.2440.2620.2670.342
VAR1.022*1.040**1.0030.9861.032***1.043***1.080***1.108***
VAR-GARCH1.0811.0631.0251.0651.151**1.0301.1131.026
VAR-stationary SV1.0031.0041.0061.0201.0061.0041.0201.040
VAR-SVt, 5 df1.0041.0061.0271.0191.0021.0011.0081.017
VAR-SVt1.0051.0051.0111.0080.9981.0000.9981.001
VAR-TVP-SV0.9901.0370.858**0.783***1.0000.9790.9390.838**
 Interest rate, 1975:Q1–2011:Q2Interest rate, 1985:Q1–2007:Q4
h = 1Qh = 2Qh = 4Qh = 8Qh = 1Qh = 2Qh = 4Qh = 8Q
Univariate models
AR-SV0.8101.2511.7722.6220.3990.7341.2521.882
AR1.053***1.072***1.0611.0731.158***1.137*1.0891.085
AR-GARCH0.9921.0700.9971.0190.9510.9830.944**0.971
AR-mixture1.0471.2041.2411.5250.9681.0531.1611.323
AR-stationary SV0.9990.9990.9970.9991.0031.0001.0001.003
AR-SVt, 5 df0.9970.9991.0031.0051.0031.0061.0081.020
AR-SVt0.9981.0000.9991.0021.0021.0021.0021.009
AR-TVP-SV0.9511.0250.9580.9960.885**0.932*0.9621.038
Multivariate models
VAR-SV0.7871.2061.7352.5910.3990.7281.2231.885
VAR1.039**1.056**1.031*1.0211.072**1.049**0.9950.970
VAR-GARCH1.0651.1651.0681.1561.0011.0570.9640.959
VAR-stationary SV1.0051.0000.9981.0061.0021.0051.0031.013
VAR-SVt, 5 df1.0081.0081.0091.0091.0111.0151.0061.007
VAR-SVt1.0041.0061.0051.0041.0031.0061.0021.000
VAR-TVP-SV1.0051.0621.0091.0120.853**0.901**0.9690.981*

Among AR models with time-varying volatility, none of the alternative volatility specifications considered yield any consistent, sizable advantage over our stochastic volatility baseline. In some cases, some of these alternative models—GARCH and the AR-mixture in particular—are significantly less accurate than the AR-SV baseline. An example is the AR-GARCH performance in inflation forecasting in the 1975–2011 sample. That said, there are some variable-horizon combinations for which GARCH, the mixture formulation, stationary SV or fat tails improve on the AR-SV baseline. However, these advantages are not consistent (across variables and horizons), and they are typically small. As an example, allowing fat tails very slightly improves the accuracy of unemployment rate forecasts in the 1975–2011 sample. Finally, note that extending the AR-SV model to include time-varying parameters (AR-TVP-SV) improves forecast accuracy in some cases (e.g. unemployment) and reduces it in others (e.g. GDP growth).

Turning to the results for VAR models, the specification with stochastic volatility (VAR-SV) fairly consistently and significantly improves on the accuracy of the model with constant volatility (VAR). For example, at shorter horizons, the RMSEs of the VAR forecasts of GDP growth are roughly 6% higher than the RMSEs of the VAR-SV forecasts. The advantage of the VAR-SV model over the VAR tends to be larger and more significant for inflation and the interest rate than for GDP growth and unemployment. Among VAR models with time-varying volatility, none of the alternative volatility specifications considered yield any consistent, sizable advantage over the stochastic volatility baseline. In fact, on balance, the VAR with GARCH is almost always dominated, often significantly, by the VAR-SV baseline, with RMSEs that often exceed baseline by 15% or more. But, as in the results for AR models, while there are some variable-horizon combinations in which specifications including stationary volatility or fat tails improve on the VAR-SV baseline, these advantages are not consistent (across variables and horizons), and they are typically small. Finally, extending the VAR-SV model to include time-varying parameters (VAR-TVP-SV) improves forecast accuracy in many, although not all, cases, consistent with the findings in D'Agostino et al. (2013).

Overall, for point forecasting, including stochastic volatility in autoregressive models seems to help forecast accuracy more often than it harms accuracy, while among models with time-varying volatility, none of the alternatives we consider offer any advantage over the (random walk) stochastic volatility baseline.

In these point forecasts, the gains in forecast accuracy provided by modeling time-varying volatility are not easily explained. In a very large sample, in the presence of time-varying volatility, the VAR (AR) and VAR-SV (AR-SV) models should yield the same coefficient estimates. However, in our finite sample of data, the coefficient estimates differ. For each model, the posterior medians of the forecast distributions are very similar to (unreported) point forecasts obtained using just the posterior means of coefficients. Accordingly, the differences in VAR and VAR-SV forecasts are due to differences in (posterior mean) coefficient estimates, not some other effect of stochastic volatility on the predictive density. However, it is difficult to pinpoint large differences in coefficients across the two models that clearly drive the differences in forecasts. Instead, there seems to be a fairly large number of small to modest differences in coefficients that together lead to some differences in forecasts. Some of the larger differences in coefficients across the VAR and VAR-SV specifications seem to be in estimates of the interest rate equation, particularly in the coefficients on inflation and the interest rate. One more easily identified pattern is that the difference in accuracy of inflation forecasts is largely due to lower bias of forecasts from the VAR-SV model, due to a lower implied mean of inflation for the VAR-SV model than the VAR model. However, the difference in mean inflation is also not easily linked to particular coefficient differences. Consequently, to explain the gains to point forecast accuracy that come with modeling time-varying volatility, we are left to speculate that, in the finite sample, in the presence of sharp movements in volatility and persistent movements in variables such as unemployment, inflation and the interest rate, including stochastic volatility can help to reduce some adverse effects of volatility changes on constant-volatility parameter estimates.

4.3 Density Forecasts: Log Predictive Scores

The results in Table 2 for log predictive scores indicate that, for AR models, including stochastic volatility significantly improves the accuracy of density forecasts relative to models with constant volatility, although more so at shorter horizons than longer horizons. At shorter horizons, the gains in average predictive scores are typically much bigger than the differences in RMSEs associated with stochastic volatility models. As an example, in the 1985–2011 sample, for 1-quarter-ahead forecasts of GDP growth, the AR model improves on the RMSE of the AR-SV baseline by 1.4%, while the AR-SV model has a predictive score that is 26% higher than that of the AR specification.

Table 2. Average log predictive scores (scores for AR-SV and VAR-SV benchmarks, score differences in all others)
 GDP growth, 1975:Q1–2011:Q2GDP growth, 1985:Q1–2007:Q4
h = 1Qh = 2Qh = 4Qh = 8Qh = 1Qh = 2Qh = 4Qh = 8Q
  1. Notes:

    1. In each quarter t from 1975:Q1 to 2011:Q2, vintage t data (which end in t − 1) are used to form forecasts for periods t to t + 7, corresponding to horizons of 1–8 quarters ahead. The forecast errors are calculated using the second-available (real-time) estimates of growth and inflation and currently available measures of unemployment and the short-term interest rate as the actuals. All variables are defined in percentage points (annualized in the case of the interest rate).

  2. 2. The models are detailed in Section 3. The notation ‘VAR-stationary SV’ refers to a VAR model in which log volatility follows an AR(1) process. The notation ‘VAR-SVt’ refers to a VAR model with stochastic volatility and fat tails, in which d denotes the degrees of freedom of the Student-t distribution that is the marginal distribution of innovations to the model; the degrees of freedom are estimated in the case of the ‘VAR-SVt’ specification and fixed at 5 in the case of the ‘VAR-SVt, 5 df’ specification. All forecasts are produced with recursive estimation of the models.

  3. 3. For the baseline AR-SV and VAR-SV models, the table reports (first row of each panel) the average values of log predictive density scores, computed with the Gaussian (quadratic) approximation given in equation (8), defined so that a higher score implies a better model. For all other AR models, the table reports the average score of each model less the average score of the AR-SV baseline. For the VAR models, the table reports the average score of each model less the average score of the VAR-SV baseline. Entries greater than 0 indicate that forecasts from the indicated model are more accurate than forecasts from the associated baseline model.

  4. 4. To provide a rough gauge of whether the average scores are significantly different, we use a Diebold–Mariano t-statistic for equal average score. Differences in accuracy that are statistically different from zero are denoted by asterisks, corresponding to significance levels of *10%, **5% and ***1%. The underlying p-values are based on t-statistics computed with a serial correlation-robust variance, using the pre-whitened quadratic spectral estimator of Andrews and Monahan (1992). Since the AR-SV (VAR-SV) model nests the AR (VAR) model, for the comparison of the AR (VAR) model to the AR-SV (VAR-SV) specification, we report p-values based on one-sided tests, taking the AR (VAR) as the null and the AR-SV (VAR-SV) as the alternative. Because the AR-SV and AR-GARCH (VAR-SV and VAR-GARCH) models are not nested, for these model comparisons we report p-values based on two-sided tests. Since the other models considered nest the AR-SV and VAR-SV baselines, for the remaining comparisons we treat each model as nesting the AR-SV or VAR-SV baseline, and we report p-values based on one-sided tests, taking the AR-SV (VAR-SV) as the null and the other model in question as the alternative.

Univariate models
AR-SV − 1.017 − 1.088 − 1.146 − 1.218 − 0.695 − 0.739 − 0.790 − 0.790
AR − 0.134*** − 0.090 − 0.039 0.027 − 0.260*** − 0.249*** − 0.219*** − 0.227***
AR-GARCH − 0.037 − 0.068** − 0.065 − 0.052* − 0.078*** − 0.100*** − 0.085 − 0.120**
AR-mixture − 0.389 − 0.420 − 0.388 − 0.333 − 0.106 − 0.110 − 0.165 − 0.173
AR-stationary SV − 0.044 − 0.024 − 0.009 0.025 − 0.088 − 0.079 − 0.067 − 0.093
AR-SVt, 5 df − 0.022 − 0.020 − 0.004 0.013 − 0.047 − 0.044 − 0.043 − 0.054
AR-SVt − 0.010 − 0.005 − 0.006 0.008 − 0.015 − 0.013 − 0.016 − 0.019
AR-TVP-SV − 0.010 − 0.009 0.000 0.010 − 0.015 − 0.018 − 0.031 − 0.068
Multivariate models
VAR-SV − 0.999 − 1.068 − 1.102 − 1.184 − 0.714 − 0.769 − 0.761 − 0.777
VAR − 0.179*** − 0.119** − 0.084 0.002 − 0.243*** − 0.204*** − 0.214*** − 0.227***
VAR-GARCH − 0.179*** − 0.231*** − 0.275*** − 0.216 − 0.193*** − 0.236*** − 0.407*** − 0.477***
VAR-stationary SV − 0.065 − 0.034 − 0.032 0.017 − 0.129 − 0.116 − 0.153 − 0.179
VAR-SVt, 5 df − 0.036 − 0.022 − 0.002 − 0.003 − 0.046 − 0.038 − 0.028 − 0.039
VAR-SVt − 0.017 − 0.012 0.002 − 0.003 − 0.015 − 0.016 − 0.005 − 0.003
VAR-TVP-SV 0.027 0.053** 0.021 0.005 0.012 0.049 0.006 − 0.014
 Unemployment, 1975:Q1–2011:Q2Unemployment, 1985:Q1–2007:Q4
h = 1Qh = 2Qh = 4Qh = 8Qh = 1Qh = 2Qh = 4Qh = 8Q
Univariate models
AR-SV − 0.094 − 0.790 − 1.921 − 3.100 0.324 − 0.231 − 0.979 − 1.774
AR − 0.109* 0.008 0.483 1.069 − 0.212*** − 0.174 0.046 0.400
AR-GARCH 0.056 0.112 0.500 1.123 − 0.049 − 0.123 − 0.028 0.298
AR-mixture − 0.095 0.092 0.449 0.779 − 0.090 − 0.112 − 0.163 − 0.324
AR-stationary SV 0.028 0.066 0.243 0.453 − 0.014 0.012 0.119 0.299*
AR-SVt, 5 df − 0.035 − 0.024 0.088 0.109 − 0.032 − 0.026 0.038 0.118*
AR-SVt − 0.020 − 0.029 0.030 0.051 − 0.022 − 0.013 0.024 0.054
AR-TVP-SV 0.046 0.081 0.333 0.772 0.006 − 0.024 0.063 0.300
Multivariate models
VAR-SV 0.025 − 0.596 − 1.447 − 2.447 0.413 − 0.125 − 0.729 − 1.225
VAR − 0.178*** − 0.168*** − 0.032 0.370 − 0.229*** − 0.224*** − 0.137 0.076
VAR-GARCH − 0.515*** − 0.430* − 0.101 0.246 − 0.791*** − 0.747*** − 0.510*** − 0.183
VAR-stationary SV − 0.032 − 0.014 0.065 0.326 − 0.050 − 0.054 − 0.020 0.081
VAR-SVt, 5 df − 0.034 − 0.039 − 0.024 0.079 − 0.046 − 0.039 − 0.031 0.016
VAR-SVt − 0.017 − 0.023 − 0.041 − 0.011 − 0.015 − 0.010 − 0.018 − 0.000
VAR-TVP-SV 0.025 0.038 0.071 0.269 0.020 0.004 − 0.010 − 0.013
 Inflation, 1975:Q1–2011:Q2Inflation, 1985:Q1–2007:Q4
h = 1Qh = 2Qh = 4Qh = 8Qh = 1Qh = 2Qh = 4Qh = 8Q
Univariate models
AR-SV − 0.192 − 0.272 − 0.355 − 0.573 − 0.001 − 0.078 − 0.112 − 0.322
AR − 0.078*** − 0.090*** − 0.108*** − 0.117*** − 0.078** − 0.105*** − 0.174*** − 0.166***
AR-GARCH − 0.119 − 0.100 − 0.137** − 0.116*** − 0.185 − 0.142 − 0.196 − 0.137**
AR-mixture − 0.336 − 0.368 − 0.230 − 0.341 − 0.356 − 0.312 − 0.239 − 0.507
AR-stationary SV − 0.015 − 0.015 − 0.043 − 0.057 − 0.034 − 0.044 − 0.092 − 0.077
AR-SVt, 5 df − 0.010 − 0.017 − 0.036 − 0.039 − 0.016 − 0.025 − 0.056 − 0.055
AR-SVt 0.000 0.000 − 0.005 − 0.007 − 0.002 − 0.004 − 0.015 − 0.009
AR-TVP-SV 0.017 0.019 0.029 0.052 0.014 0.038* 0.034* 0.078*
Multivariate models
VAR-SV − 0.203 − 0.302 − 0.424 − 0.718 − 0.011 − 0.104 − 0.146 − 0.404
VAR − 0.041* − 0.072** − 0.085** − 0.098** − 0.069** − 0.092*** − 0.169*** − 0.147***
VAR-GARCH − 0.386*** − 0.367*** − 0.374*** − 0.376*** − 0.526*** − 0.506*** − 0.582*** − 0.510***
VAR-stationary SV − 0.007 − 0.003 − 0.036 − 0.048 − 0.029 − 0.030 − 0.082 − 0.065
VAR-SVt, 5 df − 0.014 − 0.018 − 0.028 − 0.015 − 0.005 − 0.005 − 0.039 − 0.038
VAR-SVt − 0.006 0.000 − 0.004 0.003 0.001 0.005 − 0.009 − 0.005
VAR-TVP-SV 0.029 0.020 0.076* 0.150*** 0.003 0.022 0.072 0.169**
 Interest rate, 1975:Q1–2011:Q2Interest rate, 1985:Q1–2007:Q4
h = 1Qh = 2Qh = 4Qh = 8Qh = 1Qh = 2Qh = 4Qh = 8Q
Univariate models
AR-SV − 0.902 − 1.570 − 2.324 − 2.966 − 0.598 − 1.279 − 1.974 − 2.264
AR − 0.420*** − 0.246 0.139 0.092 − 0.285*** − 0.045 0.224 0.115
AR-GARCH 0.056 0.121 0.400** 0.471 0.137* 0.171 0.339** 0.206
AR-mixture − 0.136 − 0.021 0.299 0.294 0.024 0.029 0.098 − 0.193
AR-stationary SV 0.032 0.079* 0.234** 0.236 0.048 0.121* 0.219** 0.159**
AR-SVt, 5 df − 0.011 0.039 0.119* 0.123* − 0.009 0.043 0.089 0.008
AR-SVt 0.005 0.020 0.060** 0.077* 0.009 0.026 0.042* − 0.001
AR-TVP-SV 0.093** 0.111* 0.332* 0.449 0.136*** 0.161** 0.241** 0.126
Multivariate models
VAR-SV − 0.834 − 1.501 − 2.204 − 2.921 − 0.471 − 1.145 − 1.853 − 2.263
VAR − 0.434*** − 0.253 0.090 0.200 − 0.350*** − 0.108 0.195 0.226
VAR-GARCH − 0.326*** − 0.160 0.192 0.368 − 0.378*** − 0.176 0.115 0.109
VAR-stationary SV − 0.004 0.039 0.139** 0.193* − 0.000 0.039 0.124* 0.107
VAR-SVt, 5 df − 0.040 0.003 0.036 0.015 − 0.042 − 0.015 0.011 − 0.039
VAR-SVt − 0.023 − 0.006 − 0.000 − 0.025 − 0.017 − 0.012 − 0.014 − 0.048
VAR-TVP-SV 0.068** 0.079* 0.190* 0.324* 0.100*** 0.118* 0.170 0.164

As in the point forecasts from the set of AR models, none of the alternative volatility specifications yield any consistent, sizable advantage over our stochastic volatility baseline. Some models offer an occasional advantage, but only occasionally. 18 However, the results for the unemployment rate yield some more notable, although not significant, differences. The outcomes for the unemployment rate during the 2007–2009 recession fell further in the extremes of the tails of the distribution than is the case for the outcomes of other variables. 19 This is especially true with stochastic volatility, which implied at the time that forecast uncertainty was low by historical norms. The log score for AR-SV forecasts of the unemployment rate declines very sharply as the forecast horizon rises, more so in the sample that goes through 2011 than in the sample that ends in 2007. As a consequence, at the longer forecast horizons, models such as the AR and the AR-GARCH yield a much higher score than the AR-SV baseline in the 1975–2011 sample. However, these gains are not statistically significant, despite their size.

The patterns are broadly similar in results for the set of VAR models. With the multivariate specifications, including stochastic volatility with the VAR-SV model often improves on the log scores of the constant volatility VAR model, more so at shorter horizons than longer horizons. At the 1 quarter horizon, the score advantage of the VAR-SV model over the VAR is roughly 20% for growth and unemployment forecasts, less than 10% for inflation and 40% for the interest rate. The VAR-SV model dominates the VAR with GARCH, with the exception of unemployment and interest rates at longer horizons. Making volatility stationary as in the VAR-stationary SV model improves scores in some cases (e.g. unemployment and interest rates at longer horizons) and lowers them in others (e.g. growth and inflation at most horizons). 20 Adding fat tails to stochastic volatility typically lowers the log scores by a small amount. 21 Finally, consistent with the results of D'Agostino et al. (2013), adding TVP to the VAR-SV model typically (not always) improves density forecast accuracy, by amounts that are sometimes small and other times sizable enough to be statistically significant.

Overall, these results show that including stochastic volatility in autoregressive models typically yields sizable gains in density accuracy as measured by log scores, whereas among models with time-varying volatility none of the alternatives we consider offer any advantage over the stochastic volatility baseline.

4.4 Density Forecasts: CRPS

In the CRPS results for AR models shown in Table 3, including stochastic volatility consistently improves, often significantly, the accuracy of density forecasts relative to models with constant volatility, typically more so at shorter horizons than longer horizons (recall that a lower CRPS indicates better performance). For example, in interest rate forecasts over the 1985–2011 sample, the ratio of the CRPS of the AR model relative to the AR-SV model is 1.305 at the 1-quarter horizon, 1.165 at the 2-quarter horizon and 1.052 at the 4-quarter horizon. Among models with time-varying volatility, none of the alternatives offers any consistent advantage over the AR-SV baseline. The AR-mixture model is almost always worse than the baseline. The AR-GARCH specification is usually less accurate than the AR-SV baseline, except in interest rate forecasting. Allowing fat tails or making volatility stationary typically reduces density forecast accuracy by a small amount, although in a few cases these model enhancements yield small improvements in forecast accuracy (e.g. the AR-stationary SV model offers small, statistically significant reductions in CRPS in longer horizon forecasts of interest rates). One other finding worth noting is that, in the case of the unemployment rate forecasts, the CRPS-based performance of the baseline model with stochastic volatility does not deteriorate as rapidly with the forecast horizon as did the log score-based performance of the same model. As a result, as the forecast horizon increases, the model with constant volatility does not improve as much in relative terms under the CRPS measure as it did under the log score measure. This pattern reflects the fact that the CRPS is less sensitive to outlier outcomes.

Table 3. Average CRPS (CRPS for AR-SV and VAR-SV benchmarks, CRPS ratios in all others)
 GDP growth, 1975:Q1–2011:Q2GDP growth, 1985:Q1–2007:Q4
h = 1Qh = 2Qh = 4Qh = 8Qh = 1Qh = 2Qh = 4Qh = 8Q
  1. Notes:

    1. In each quarter t from 1975:Q1 to 2011:Q2, vintage t data (which end in t − 1) are used to form forecasts for periods t to t + 7, corresponding to horizons of 1–8 quarters ahead. The forecast errors are calculated using the second-available (real-time) estimates of growth and inflation and currently available measures of unemployment and the short-term interest rate as the actuals. All variables are defined in percentage points (annualized in the case of the interest rate).

  2. 2. The models are detailed in Section 3. The notation ‘VAR-stationary SV’ refers to a VAR model in which log volatility follows an AR(1) process. The notation ‘VAR-SVt’ refers to a VAR model with stochastic volatility and fat tails, in which d denotes the degrees of freedom of the Student-t distribution that is the marginal distribution of innovations to the model; the degrees of freedom are estimated in the case of the ‘VAR-SVt’ specification and fixed at 5 in the case of the ‘VAR-SVt, 5 df’ specification. All forecasts are produced with recursive estimation of the models.

  3. 3. For the baseline AR-SV and VAR-SV models, the table reports (first row of each panel) the average cumulative ranked probability score (CRPS), computed with the formula given in equation (9), defined so that a lower CPRS implies a better model. For all other AR models, the table reports the ratio of the average CRPS for the model relative to the average CRPS of the AR-SV baseline. For the VAR models, the table reports the ratio of the average CRPS for the model relative to the average CRPS of the VAR-SV baseline. Entries less than 1 indicate that forecasts from the indicated model are more accurate than forecasts from the associated baseline model.

  4. 4. To provide a rough gauge of whether the average scores are significantly different, we use a Diebold–Mariano t-statistic for equal average CRPS. Differences in accuracy that are statistically different from zero are denoted by asterisks, corresponding to significance levels of *10%, **5% and ***1%. The underlying p-values are based on t-statistics computed with a serial correlation-robust variance, using the pre-whitened quadratic spectral estimator of Andrews and Monahan (1992). Since the AR-SV (VAR-SV) model nests the AR (VAR) model, for the comparison of the AR (VAR) model to the AR-SV (VAR-SV) specification, we report p-values based on one-sided tests, taking the AR (VAR) as the null and the AR-SV (VAR-SV) as the alternative. Because the AR-SV and AR-GARCH (VAR-SV and VAR-GARCH) models are not nested, for these model comparisons we report p-values based on two-sided tests. Since the other models considered nest the AR-SV and VAR-SV baselines, for the remaining comparisons we treat each model as nesting the AR-SV or VAR-SV baseline, and we report p-values based on one-sided tests, taking the AR-SV (VAR-SV) as the null and the other model in question as the alternative.

Univariate models
AR-SV0.3810.3880.3940.3970.2610.2670.2820.283
AR1.059***1.057***1.042**1.036*1.136***1.140***1.114***1.120***
AR-GARCH1.0071.027**1.0221.0211.039**1.053***1.0451.063**
AR-mixture1.0701.0761.0821.0861.0251.0231.0741.132
AR-stationary SV1.0171.0151.0091.0031.0401.0421.0301.041
AR-SVt, 5 df1.0041.0081.0061.0041.0071.0131.0171.019
AR-SVt1.0021.0001.0040.9991.0000.9991.0101.007
AR-TVP-SV1.0211.0271.0321.0291.0101.0111.0331.058
Multivariate models
VAR-SV0.3880.3860.3890.3950.2750.2810.2790.276
VAR1.111***1.108***1.080**1.033*1.150***1.145***1.121***1.121***
VAR-GARCH1.160***1.269***1.257***1.205***1.162***1.213***1.315***1.349***
VAR-stationary SV1.0281.0251.0211.0051.0471.0601.0561.065
VAR-SVt, 5 df1.0101.0020.9961.0031.0091.0140.987**0.994
VAR-SVt1.0041.0021.0021.0001.0001.0060.9970.994*
VAR-TVP-SV0.9640.955*0.9771.0090.9910.958**0.9951.011
 Unemployment, 1975:Q1–2011:Q2Unemployment, 1985:Q1–2007:Q4
h = 1Qh = 2Qh = 4Qh = 8Qh = 1Qh = 2Qh = 4Qh = 8Q
Univariate models
AR-SV0.1540.2740.5260.9550.0930.1580.2930.530
AR1.0191.0090.9770.8911.131**1.1401.0961.005
AR-GARCH0.9590.9800.9950.9001.0631.1111.1230.999
AR-mixture1.0451.1041.2041.2761.1511.2121.3241.457
AR-stationary SV1.0081.0131.0090.9921.0101.0141.0080.978*
AR-SVt, 5 df0.9981.0011.0000.9891.0081.0141.0110.995
AR-SVt0.9991.0031.0020.9921.0021.0051.0050.999
AR-TVP-SV0.9470.9640.9720.9401.0181.0471.0471.046
Multivariate models
VAR-SV0.1450.2550.4640.7190.0880.1520.2820.448
VAR1.060***1.077**1.0610.9731.137***1.155***1.1150.966
VAR-GARCH1.340***1.340***1.246***1.146**1.742***1.632***1.384**1.102
VAR-stationary SV1.0131.0151.0120.9971.0271.0241.0100.974
VAR-SVt, 5 df1.0161.0241.0201.0051.0221.0211.0251.005
VAR-SVt1.0091.0091.0101.0041.0151.0091.0101.003
VAR-TVP-SV0.9510.9550.9731.0160.9840.9690.9701.017
 Inflation, 1975:Q1–2011:Q2Inflation, 1985:Q1–2007:Q4
h = 1Qh = 2Qh = 4Qh = 8Qh = 1Qh = 2Qh = 4Qh = 8Q
Univariate models
AR-SV0.1690.1830.2030.2600.1380.1460.1490.186
AR1.039***1.037***1.061***1.076***1.038**1.066***1.115***1.127***
AR-GARCH1.1031.0881.118*1.112*1.1681.1451.1881.132*
AR-mixture1.1221.1031.0921.0471.0711.0231.0320.964
AR-stationary SV1.0021.0051.0151.0261.0121.0231.0561.055
AR-SVt, 5 df1.0061.0051.0071.0221.0101.0131.0191.022
AR-SVt0.9991.0021.0001.0081.0041.0061.0061.003
AR-TVP-SV0.9970.9890.9660.9580.9960.9700.9810.924
Multivariate models
VAR-SV0.1710.1880.2170.3090.1390.1510.1530.200
VAR1.027**1.039**1.0341.0271.039**1.057***1.125***1.120***
VAR-GARCH1.251***1.249***1.228***1.179**1.401***1.394**1.518***1.381***
VAR-stationary SV0.9980.9941.0111.0281.0061.0061.0421.032
VAR-SVt, 5 df1.0021.0001.0171.0140.9990.9981.0111.015
VAR-SVt0.9991.0021.0041.0040.9970.9990.9990.999
VAR-TVP-SV0.9761.0100.893**0.800***0.9890.9750.9440.839**
 Interest rate, 1975:Q1–2011:Q2Interest rate, 1985:Q1–2007:Q4
h = 1Qh = 2Qh = 4Qh = 8Qh = 1Qh = 2Qh = 4Qh = 8Q
Univariate models
AR-SV0.3650.6271.0001.5940.2200.4160.7371.134
AR1.162***1.138***1.0471.0351.305***1.165***1.0521.033
AR-GARCH0.9711.0190.930***0.957*0.9520.9670.903**0.926
AR-mixture1.0211.1021.1031.2730.9771.0571.1411.319
AR-stationary SV0.9971.0020.983***0.977***1.0000.9930.9840.978**
AR-SVt, 5 df1.0031.0030.9980.9971.0081.0071.0041.009
AR-SVt1.0001.0040.9950.9981.0011.0071.0031.007
AR-TVP-SV0.9330.9850.927*0.9740.896**0.926*0.9381.001
Multivariate models
VAR-SV0.3610.6150.9751.5750.2160.4060.7101.135
VAR1.135***1.096**1.0100.9711.249***1.095**0.9800.924
VAR-GARCH1.136*1.161*1.0251.0411.240***1.122*0.9920.960
VAR-stationary SV1.0041.0010.9930.9850.9990.9980.9940.996
VAR-SVt, 5 df1.0131.0131.0101.0021.0121.0131.0121.009
VAR-SVt1.0101.0041.0051.0051.0061.0051.0071.004
VAR-TVP-SV0.940*0.9940.9810.9890.872***0.905**0.9510.956*

With VAR models, the patterns are broadly similar. In most cases, compared to a VAR with constant volatility, a VAR including stochastic volatility improves density accuracy as measured by the CRPS, again more so at shorter horizons than longer horizons. Moreover, at horizons of 1 and 2 quarters, the gains in accuracy associated with stochastic volatility are statistically significant. Among models with time-varying volatility, no other specification offers consistent improvement over the VAR-SV baseline. Notably, the VAR-GARCH model performs significantly worse for almost every variable and horizon combination. Making stochastic volatility stationary or adding fat tails does not have much effect on CRPS-based density accuracy; these extensions slightly reduce accuracy in some cases (e.g. the performance of the VAR-stationary SV specification with GDP growth forecasts) and improve it in others (e.g. the performance of the VAR-SVt model in longer horizon GDP growth forecasts in the 1985–2007 sample). Once again, though, adding TVP to the VAR with stochastic volatility typically improves forecast accuracy, especially for inflation and the interest rate.

5 Conclusions

  1. Top of page
  2. Summary
  3. 1 Introduction
  4. 2 Data
  5. 3 Models
  6. 4 Results
  7. 5 Conclusions
  8. Acknowledgements
  9. References
  10. Appendix
  11. Supporting Information

This paper compares, from a forecasting perspective, alternative models of time-varying macroeconomic volatility, included within AR and VAR specifications for key macroeconomic indicators. The set of models includes constant volatility; random walk stochastic volatility; stochastic volatility following a stationary AR process; stochastic volatility coupled with fat tails; GARCH; and a mixture of innovations model. The forecast comparisons cover GDP growth, the unemployment rate, inflation in the GDP deflator and a short-term interest rate from 1975 to 2011. Our results indicate that the AR and VAR specifications with stochastic volatility dominate models with alternative volatility specifications, in terms of point forecasting to some degree and density forecasting to a greater degree, in particular when using proper scoring rules such as the CRPS. We conclude that, from a macroeconomic forecasting perspective, these alternative volatility specifications seem to have no advantage over the now widely used random walk stochastic volatility specification.

While this paper has focused on economic forecasting, we suggest the results have implications for macroeconomic modeling. There has been considerable effort over the last several years to enable DSGE models to account for time-varying volatility, to be able to explain the sources of the Great Moderation and other changes in volatility (examples include Justiniano and Primiceri, 2008; Fernandez-Villaverde and Rubio-Ramirez, 2007; Fernandez-Villaverde et al., 2010; and Curdia et al., 2013). These studies have examined questions including which shocks drive time-varying volatility, the roles of different shocks in business cycle transmission and the roles of coefficients compared to shock sizes. While some work with DSGE models has considered GARCH (e.g. Andreasen, 2012) and Markov switching (e.g. Bianchi, 2013), most work on DSGE models with time-varying volatility has focused on random walk stochastic volatility. Our finding that stochastic volatility is, in general, in forecasting, at least as good as most other readily tractable (if sometimes more complicated) volatility specifications that might be considered supports the focus of structural modeling on stochastic volatility.

Acknowledgements

  1. Top of page
  2. Summary
  3. 1 Introduction
  4. 2 Data
  5. 3 Models
  6. 4 Results
  7. 5 Conclusions
  8. Acknowledgements
  9. References
  10. Appendix
  11. Supporting Information

We gratefully acknowledge helpful comments from referees, Editor Frank Diebold, Andrea Carriero, Lutz Kilian, Dimitris Korobilis, Massimiliano Marcellino, Michael McCracken, and seminar participants at the 21th Symposium of the Society for Nonlinear Dynamics and Econometrics and the 7th Rimini Bayesian Econometrics Workshop. The views expressed herein are solely those of the authors and do not necessarily reflect the views of the Federal Reserve Bank of Cleveland, Federal Reserve System or Norges Bank.

  1. 1

    Karapanagiotidis (2012) considers yet another approach, using autoregressive Wishart processes to capture time-varying volatility in macroeconomic BVARs for forecasting. Koop and Korobilis (2013) show that a computational shortcut for allowing time-varying volatility, using a form of exponential smoothing of volatility, improves the accuracy of point and density forecasts from larger VARs.

  2. 2

    In the finance literature, some studies compare volatility models for their efficacy in modeling returns (e.g. Geweke and Amisano, 2010; Nakajima, 2012), while others compare volatility models for their efficacy in modeling volatility (e.g. Hansen and Lunde, 2005).

  3. 3

    In an earlier version of this paper (Clark and Ravazzolo, 2012), we obtained similar results in real-time forecasts for the UK.

  4. 4

    The GDP data available today for, say, 1985, represent the best available estimates of output in 1985. However, output as defined and measured today is quite different from output as defined and measured in 1970. For example, today we have available chain-weighted GDP; in the 1980s, output in the USA was measured with fixed-weight GNP. Forecasters in 1985 could not have foreseen such changes.

  5. 5

    In light of the practice of institutions such as the Federal Reserve to report growth and inflation rates in the form of four-quarter average growth rates, we have also examined the accuracy of GDP growth and inflation forecasts transformed into four-quarter averages. These results, provided in Tables 3–5 of the supplementary Appendix, are qualitatively similar to the quarterly forecast results we provide herein.

  6. 6

    In light of the inflation forecasting success Stock and Watson (2007) demonstrated with an unobserved components–stochastic volatility model, some might consider their model to be a natural candidate for inclusion in the inflation forecasting comparisons. In the interest of brevity, we focus results in the paper on models in the autoregressive class. However, we include inflation forecasting results for the UC-SV model of Stock and Watson (2007) in the supplementary Appendix, in Table 2. In our real-time data on GDP inflation, the UC-SV model performs comparably to our AR-SV baseline. In the 1985–2007 period, at longer forecast horizons, the UC-SV specification is a little more accurate than the AR-SV model, but not significantly. In the 1975–2011 period, the AR-SV model is a little more accurate than the UC-SV specification.

  7. 7

    In results not reported in the interest of brevity, we also considered a VAR with fat tails but not stochastic volatility. Forecasts from this model were clearly less accurate than forecasts from the model with stochastic volatility. We also tried a version of the VAR-GARCH with Student-t residuals where the degrees of freedom τ are estimated. Results were worse than the normal case and we do not report them.

  8. 8

    In results not reported in the interest of brevity, we also considered a version of the VAR-SV model in which the elements of A are time-varying, random walks as in Primiceri (2005), for example. Allowing A to be time-varying did not offer any consistent or material improvement in forecast accuracy.

  9. 9

    In our estimates, the correlations implied by Φ are sizable in many cases.

  10. 10

    Under the prior we use, detailed in the Appendix, we never draw AR(1) coefficients that exceed a value of 1. However, our algorithm includes an accept–reject step for ruling out explosive draws.

  11. 11

    We also tried a version with Student-t residuals where the degrees of freedom τ are estimated. Results were worse than the normal case and we do not report them.

  12. 12

    We include fewer lags for GDP growth because its persistence is relatively low and other studies have found AR(1) or AR(2) models to forecast GDP well.

  13. 13

    However, we modify the algorithm of Primiceri (2005) to reflect the correction to the ordering of steps detailed in Del Negro and Primiceri (2013).

  14. 14

    In the simpler case of the VAR with constant volatilities, to generate draws of forecasts, for each draw of the VAR coefficients and error covariance matrix, we generate shocks from t + 1 to t + H, using the given draw of Φ. We use the shocks, the autoregressive structure of the VAR and the draw of coefficients to compute the draw of Yt + h, h = 1, … ,H.

  15. 15

    Figure 1 of the supplementary Appendix provides time series of correlations among the reduced-form innovations of the variables considered. These correlations show some variation over time, but less significant variation than is evident in the volatilities. In addition, the correlations across residuals implied by the VAR-SV model show somewhat less variation over time than do the simple rolling window estimates of correlations among innovations (however, the rolling window correlations are also noisy over time). With the VAR-SV model treating the matrix A as constant, any movement over time in residual correlations has to be due to changes in the volatilities contained in Λt. One might worry that the assumption of a constant A matrix limits the ability of the model to capture true variation in correlations, but we obtained very similar results in a version of the model that allows the elements of A to be time varying, following random walks. We take these findings as evidence of limited movement over time in the correlations of reduced-form VAR innovations. However, we leave as a subject for further research the development of a tractable, alternative VAR specification with time-varying volatility that might better pick up time variation in correlations.

  16. 16

    See Ravazzolo and Vahey (2013) for an application to disaggregate inflation.

  17. 17

    Amisano and Giacomini (2007) extend the results of Giacomini and White (2006) for point forecasts to density forecasts, developing a set of weighted likelihood ratio tests. In our application to tests for density forecasts, we do not employ a weighting scheme (or put another way, we use equal weights).

  18. 18

    The exception to this rule is that, with AR models, GARCH seems to work better than stochastic volatility for the interest rate. For example, with 1-quarter-ahead forecasts in the 1985–2011 sample, the AR-GARCH model improves upon the score of the AR-SV baseline by 13.7%.

  19. 19

    To better understand the broad influences of the crisis on density forecast performance, for the VAR and VAR-SV models we have taken a closer look at one-step-ahead predictive scores over the 2006–2010 period. This analysis indicates the performance of the stochastic volatility specification briefly deteriorates relative to the performance of the constant volatility VAR for the following reasons. Before the crisis, the VAR-SV specification generally scores better than the VAR because the VAR-SV model better picks up the effects of the Great Moderation on volatility. Some of the extreme outcomes of the crisis period are more unusual by the standards of the VAR-SV-estimated (narrower) predictive densities than by the standards of the VAR-estimated (wider) predictive densities. Consequently, for a few quarters, the VAR tends to score better than the VAR-SV models. But after a few quarters, the VAR-SV model has picked up enough of a rise in volatility that it resumes yielding predictive scores better than the scores from the VAR.

  20. 20

    The estimates of the VAR-stationary SV model imply volatility to be persistent but stationary, with some differences across variables. The posterior means of the AR(1) coefficients of the volatility processes range from 0.6 for some variables in some samples to 0.9 for others.

  21. 21

    The posterior mean of the degrees of freedom with the AR-SVt and VAR-SVt models typically falls between about 15 and 20, but rises as high as 25 for inflation, with a posterior standard deviation of roughly 10.

  22. 22

    For the handful of the data vintages that do not start until 1959, we set the mean and variance of the initial value of coefficients and mean of the initial value of log volatility at their values from the most recent (earlier) vintage for which data were available back to 1948.

References

  1. Top of page
  2. Summary
  3. 1 Introduction
  4. 2 Data
  5. 3 Models
  6. 4 Results
  7. 5 Conclusions
  8. Acknowledgements
  9. References
  10. Appendix
  11. Supporting Information
  • Adolfson M, Linde J, Villani M. 2007. Forecasting performance of an open economy DSGE model. Econometric Reviews 26: 289328.
  • Amisano G, Giacomini R. 2007. Comparing density forecasts via weighted likelihood ratio tests. Journal of Business and Economic Statistics 25: 177190.
  • Andreasen M. 2012. On the effects of rare disasters and uncertainty shocks for risk premia in non-linear DSGE models. Review of Economic Dynamics 15: 295316.
  • Andrews D, Monahan JC. 1992. An improved heteroskedasticity and autocorrelation consistent covariance matrix estimator. Econometrica 60: 953966.
  • Benati L. 2008. The ‘Great Moderation’ in the United Kingdom. Journal of Money, Credit, and Banking 40: 121147.
  • Bianchi F. 2013. Regime switches, agents' beliefs, and post-World War II U.S. macroeconomic dynamics. Review of Economic Studies 80: 463490.
  • Bognanni M. 2013. An empirical analysis of time-varying fiscal multipliers. Working Paper. University of Pennsylvania.
  • Bollerslev T. 1986. Generalized autoregressive conditional heteroskedasticity. Journal of Econometrics 31: 307327.
  • Canarella G, Fang W, Miller S, Pollard S. 2010. Is the Great Moderation ending? UK and US evidence. Modern Economy 1: 1742.
  • Carriero A, Clark T, Marcellino M. 2012. Common drifting volatility in large Bayesian VARs. Working Paper. Federal Reserve Bank of Cleveland.
  • Carter C, Kohn R. 1994. On Gibbs sampling for state space models. Biometrika 81: 541553.
  • Chung H, Laforte J-P, Reifschneider D, Williams J. 2012. Have we underestimated the likelihood and severity of zero lower bound events? Journal of Money, Credit, and Banking 44: 4782.
  • Clark T. 2009. Is the Great Moderation over? An empirical analysis. Federal Reserve Bank of Kansas City Economic Review 94: 542.
  • Clark T. 2011. Real-time density forecasts from BVARs with stochastic volatility. Journal of Business and Economic Statistics 29: 327341.
  • Clark T, McCracken M. 2011a. Nested forecast model comparisons: a new approach to testing equal accuracy. Working Paper. Federal Reserve Bank of St Louis.
  • Clark T, McCracken M. 2011b. Testing for unconditional predictive ability. In Oxford Handbook of Economic Forecasting, Clements MP, Hendry DF (eds). Oxford University Press: Oxford; 415440.
  • Clark T, Ravazzolo F. 2012. The macroeconomic forecasting performance of autoregressive models with alternative specifications of time-varying volatility. Working Paper 2012-09. Norges Bank.
  • Cogley T, Sargent T. 2005. Drifts and volatilities: monetary policies and outcomes in the post-World War II U.S. Review of Economic Dynamics 8: 262302.
  • Cogley T, Morozov S, Sargent T. 2005. Bayesian fan charts for U.K. inflation: forecasting and sources of uncertainty in an evolving monetary system. Journal of Economic Dynamics and Control 29: 18931925.
  • Croushore D. 2006. Forecasting with real-time macroeconomic data. In Handbook of Economic Forecasting, Elliott G, Granger C, Timmermann A (eds). North-Holland: Amsterdam; 961982.
  • Croushore D, Stark T. 2001. A real-time data set for macroeconomists. Journal of Econometrics 105: 111130.
  • Curdia V, Del Negro M, Greenwald D. 2013. Rare shocks, great recessions. Working Paper. Federal Reserve Bank of New York.
  • D'Agostino A, Gambetti L, Giannone D. 2013. Macroeconomic forecasting and structural change. Journal of Applied Econometrics 28: 82101.
  • Del Negro M, Primiceri G. 2013. Time varying structural vector autoregressions and monetary policy: a corrigendum. Working Paper. Northwestern University.
  • Diebold F, Mariano R. 1995. Comparing predictive accuracy. Journal of Business and Economic Statistics 13: 253263.
  • Engle R. 1982. Autoregressive conditional heteroscedasticity with estimates of the variance of United Kingdom inflation. Econometrica 50: 9871007.
  • Faust J, Wright J. 2009. Comparing Greenbook and reduced form forecasts using a large realtime dataset. Journal of Business and Economic Statistics 27: 468479.
  • Fernandez-Villaverde J, Rubio-Ramierez J. 2007. Estimating macroeconomic models: a likelihood approach. Review of Economic Studies 74: 10591087.
    Direct Link:
  • Fernandez-Villaverde J, Rubio-Ramierez J. 2010. Macroeconomics and volatility: data, models, and estimation. Working Paper. University of Pennsylvania.
  • Fernandez-Villaverde J, Guerron-Quintana P, Rubio-Ramierez J. 2010. Fortune or virtue: time-variant volatilities versus parameter drifting in U.S. data. Working Paper. University of Pennsylvania.
  • Gerlach R, Carter C, Kohn R. 2000. Efficient Bayesian inference for dynamic mixture models. Journal of the American Statistical Association 95: 819828.
  • Geweke J, Amisano G. 2010. Comparing and evaluating Bayesian predictive distributions of asset returns. International Journal of Forecasting 26: 216230.
  • Giacomini R, White H. 2006. Tests of conditional predictive ability. Econometrica 74: 15451578.
  • Giordani P, Villani M. 2010. Forecasting macroeconomic time series with locally adaptive signal extraction. International Journal of Forecasting 26: 312325.
  • Giordani P, Kohn R, van Dijk D. 2007. A unified approach to nonlinearity, outliers and structural breaks. Journal of Econometrics 137: 112137.
  • Gneiting T, Raftery A. 2007. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102: 359378.
  • Gneiting T, Ranjan R. 2011. Comparing density forecasts using threshold and quantile weighted proper scoring rules. Journal of Business and Economic Statistics 29: 411422.
  • Groen J, Paap R, Ravazzolo F. 2013. Real-time inflation forecasting in a changing world. Journal of Business and Economic Statistics 31: 2944.
  • Hansen P, Lunde A. 2005. A forecast comparison of volatility models: does anything beat a GARCH(1,1)? Journal of Applied Econometrics 20: 873889.
  • Hubrich K, Tetlow R. 2012. Financial stress and economic dynamics: the transmission of crises. FEDS Working Paper 2012-82. Federal Reserve Board of Governors.
  • Jacquier E, Polson N, Rossi P. 2004. Bayesian analysis of stochastic volatility models with fat-tails and correlated errors. Journal of Econometrics 122: 185212.
  • Jore A, Mitchell J, Vahey S. 2010. Combining forecast densities from VARs with uncertain instabilities. Journal of Applied Econometrics 25: 621634.
  • Justiniano A, Primiceri G. 2008. The time-varying volatility of macroeconomic fluctuations. American Economic Review 98: 604641.
  • Kadiyala K, Karlsson S. 1997. Numerical methods for estimation and inference in Bayesian VAR-models. Journal of Applied Econometrics 12: 99132.
  • Karapanagiotidis P. 2012. Improving Bayesian VAR density forecasts through autoregressive Wishart stochastic volatility. Working Paper. University of Toronto.
  • Kim S, Shephard N, Chib S. 1998. Stochastic volatility: Likelihood inference and comparison with ARCH models. Review of Economic Studies 65: 361393.
    Direct Link:
  • Koop G. 2003. Bayesian Econometrics. Wiley: Chichester.
  • Koop G, Korobilis D. 2013. Large time-varying parameter VARs. Journal of Econometrics 177: 185198.
  • Koop G, Potter S. 2007. Estimation and forecasting in models with multiple breaks. Review of Economic Studies 74: 763789.
    Direct Link:
  • Nakajima J. 2012. Bayesian analysis of generalized autoregressive conditional heteroskedasticity and stochastic volatility: modeling leverage, jumps and heavy-tails for financial time series. Japanese Economic Review 63: 81103.
  • Primiceri G. 2005. Time varying structural vector autoregressions and monetary policy. Review of Economic Studies 72: 821852.
    Direct Link:
  • Ravazzolo F, Vahey S. 2013. Forecast densities for economic aggregates from disaggregate ensembles. Studies in Nonlinear Dynamics and Econometrics. (forthcoming).
  • Romer C, Romer D. 2000. Federal Reserve information and the behavior of interest rates. American Economic Review 90: 429457.
  • Sims C. 2002. The role of models and probabilities in the monetary policy process. Brookings Papers on Economic Activity 2: 140.
  • Sims C, Zha T. 2006. Were there regime switches in US monetary policy? American Economic Review 96: 5481.
  • So M, Chen C, Chen M-T. 2005. A Bayesian threshold nonlinearity test for financial time series. Journal of Forecasting 24: 6175.
  • Stock J, Watson M. 2003. Has the business cycle changed? Evidence and explanations. Monetary Policy and Uncertainty: adapting to a Changing Economy, a symposium sponsored by the Federal Reserve Bank of Kansas City.
  • Stock J, Watson M. 2007. Has U.S. inflation become harder to forecast? Journal of Money, Credit, and Banking 39: 333.
  • Vrontos ID, Dellaportas P, Politis DN. 2000. Full Bayesian inference for GARCH and EGARCH models. Journal of Business and Economic Statistics 18: 187198.

Appendix

  1. Top of page
  2. Summary
  3. 1 Introduction
  4. 2 Data
  5. 3 Models
  6. 4 Results
  7. 5 Conclusions
  8. Acknowledgements
  9. References
  10. Appendix
  11. Supporting Information

This Appendix details the priors we use. In the interest of brevity, in most cases we provide details for the VAR specifications and omit details on AR specifications, which are just simplifications of the VAR specifications.

VAR with Constant Volatility

For the VAR coefficients, we use a conventional Minnesota prior, without cross-variable shrinkage (note that i and j refer to the row and column of Bl):

  • display math(A.1)
  • display math(A.2)

Following common settings, we set θ = 0.2, ϵ = 1000, and the scale parameters inline image at estimates of residual variances from AR(p) models from the estimation sample. With all of the variables of our VAR models defined so that they should be stationary, we set the prior mean of all the VAR coefficients to 0.

VAR with TVP and Stochastic Volatility

Our prior specification is patterned after Primiceri (2005), except that the prior for the (constant rather than time-varying) A matrix is taken from Cogley and Sargent (2005). More specifically, following Primiceri (2005), we use OLS estimates of the VAR over a training sample of 1949–1960 to set the mean and variance of the initial value of the vector of VAR coefficients (setting the initial variance at four times the OLS-estimated variance). 22 The prior for Q follows an inverted Wishart distribution, with a mean of inline image times the training sample OLS variance matrix and degrees of freedom μQ = 10 for AR models and μQ = k(kp + 1) + 1 for VARs.

In the prior for the volatility-related components of the model, we use an approach to setting them similar to that of such studies as Cogley and Sargent (2005), Primiceri (2005) and Clark (2011). The prior for A is uninformative, with a mean and variance for each row vector of inline image, i = 2, … ,k. We make the priors on the volatility-related parameters loosely informative. The prior for Φ is inverted Wishart, with mean of 0.01 × Ik and k + 1 degrees of freedom. For the initial value of the log volatility of each equation i, we use a mean of inline image and variance of 4. To obtain inline image, we use the residuals from the VAR(2) estimated over the 1949–1960 training sample. For each j = 2, … ,k, we regress the residual from the VAR model for j on the residuals associated with variables 1 to j − 1 and compute the error variance inline image (this filters out covariance as reflected in A). We set the prior mean of log volatility in period 0 at inline image.

VAR with Stochastic Volatility

For the VAR-SV specification, we use a normal prior for the VAR coefficients and a prior like that of the VAR-TVP-SV specification for the volatility components of the model. More specifically, for the VAR coefficients, the prior mean and variance are specified as described above for the VAR with constant volatility. The prior for A is uninformative. The prior for Φ is inverted Wishart, with mean of 0.01 × Ik and k + 1 degrees of freedom. For the initial value of the log volatility of each equation i, we use a mean of inline image and variance of 4. To obtain inline image, we use the residuals from AR(4) models estimated over a training sample of 1949–1954. For each j = 2, … ,k, we regress the residual from the AR model for j on the residuals associated with variables 1 to j − 1 and compute the error variance inline image. We set the prior mean of log volatility in period 0 at inline image. Since a handful of the data vintages do not start until 1959, we use the same prior mean on initial volatility for all vintages (forecast origins), computed using the last available vintage of data, with a training sample of 24 observations.

In the case of the model with stationary AR(1) stochastic volatility, we set the prior mean and standard deviation of each volatility equation's intercept at 0 and 0.5, respectively, and we set the prior mean of the AR(1) coefficient at 0.8, with a standard deviation of 0.2.

In the case of the model with fat tails in which the degrees of freedom are estimated, we follow the implementation of Koop (2003) in using an exponential distribution prior (with a prior mean of degrees of freedom of 10) and conditional posterior that requires a Metropolis step (specifically, a random walk chain algorithm with a normally distributed increment random variable), under assumptions that imply independence across the VAR's variables.

VAR with GARCH Volatility

For the VAR-GARCH model, we use normal priors and posteriors for the VAR coefficients. The prior mean and variance are set to OLS estimates over a training sample of 1949–1954. In the case of the GARCH parameters, we use uniform distribution priors that satisfy the restrictions a0,i > 0, aj,i ≥ 0, j = 1, 2, a1 + a2 < 1. Therefore, the prior for a0,i spans the positive real space, with 0 excluded; the prior for a1,i is bounded in [0,1); and the prior for a2,i is bounded in [0, 1 − a1,i). Finally, the prior for A is uninformative.

AR mixture of Innovation Models

In the case of the structural break probability parameters, we use Beta distributions for the priors, of the form πj ∼ Beta(aj,bj), j = 0, … ,p + 1. The parameters aj and bj are set according to prior beliefs about the occurrence of structural breaks. The expected prior probability of a break is aj ∕ (aj + bj). We set aj = 0.8 and bj = 30 for j = 0, … ,p and ap + 1 = 0.5 and bp + 1 = 2. For the variance parameters, which reflect prior beliefs about the size of the structural breaks, we use an inverted Gamma-2 prior that depends on scale parameter inline image and degrees of freedom parameter νj: inline image, with inline image. The expected prior break size therefore equals the square root of (ωjνj) ∕ (νj − 2) for νj > 2. We set ωj equal to OLS estimates of the variance of the autoregressive parameters and residual variance divided by 100 over a training sample of 1949–1954. Similarly to the VAR-TVP-SV approach, we use OLS estimates of the AR models over the training sample to set the mean and variance of the initial value of the vector of AR coefficients (setting the variance at 4 times the OLS-estimated variance), and the variance of the residuals, inline image, to set the initial values of the log variance inline image.

Supporting Information

  1. Top of page
  2. Summary
  3. 1 Introduction
  4. 2 Data
  5. 3 Models
  6. 4 Results
  7. 5 Conclusions
  8. Acknowledgements
  9. References
  10. Appendix
  11. Supporting Information

The JAE Data Archive directory is available at http://qed.econ.queensu.ca/jae/datasets/clark004/

FilenameFormatSizeDescription
jae2379-sup-0001-supinfo.pdfPDF document474KSupporting info item

Please note: Wiley Blackwell is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.