You have free access to this content

# Bayesian measures of model complexity and fit

Article first published online: 23 OCT 2002

DOI: 10.1111/1467-9868.00353

Issue

## Journal of the Royal Statistical Society: Series B (Statistical Methodology)

Volume 64, Issue 4, pages 583–639, October 2002

Additional Information

#### How to Cite

Spiegelhalter, D. J., Best, N. G., Carlin, B. P. and Van Der Linde, A. (2002), Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64: 583–639. doi: 10.1111/1467-9868.00353

#### Publication History

- Issue published online: 23 OCT 2002
- Article first published online: 23 OCT 2002
- [
*Read before*The Royal Statistical Society*at a meeting organized by the*Research Section*on Wednesday, March 13th, 2002*, Professor D. Firth*in the Chair]*

### References

- 1973) Information theory and an extension of the maximum likelihood principle. In Proc. 2nd Int. Symp. Information Theory (eds B. N.Petrov and F.Csáki ), pp. 267–281. Budapest: Akadémiai Kiadó. (
- 1974) Scale mixtures of normal distributions. J. R. Statist. Soc. B, 36, 99–102. and (
- 1966) Limiting behaviour of posterior distributions when the model is incorrect. Ann. Math. Statist., 37, 51–58. (
- 1979) Expected information as expected utility. Ann. Statist., 7, 686–690. (
- 1994) Bayesian Theory. Chichester: Wiley. and (
- 1974) Spatial interaction and the statistical analysis of lattice systems (with discussion). J. R. Statist. Soc. B, 36, 192–236. (
- 2001) Bayesian varying-coefficient modelsusing adaptive regression splines. Statist. Modlng, 1, 195–211. and (
- 1976) Science and statistics. J. Am. Statist. Ass., 71, 791–799. (
- 1993) Approximate inference in generalized linear mixed models. J. Am. Statist. Ass., 88, 9–25. and (
- 1965) Statistical Theory and Methodology in Science and Engineering. New York: Wiley. (
- 1998) Asymptotic behaviour of Bayes estimates under possibly incorrect models. Ann. Statist., 26, 617–644. and (
- 1998) Model Selection and Inference. New York: Springer. and (
- 2000) Bayes and Empirical Bayes Methods for Data Analysis, 2nd edn. Boca Raton: Chapman and Hall–CRC Press. and (
- 1998) Analysis of multivariate probit models. Biometrika, 85, 347–361. and (
- 1987) Empirical Bayes estimates of age-standardised relative risks for use in disease mapping. Biometrics, 43, 671–681. and (
- 1974) The direct use of likelihood for significance testing. In Proc. Conf. Foundational Questions in Statistical Inference (eds O.Barndorff-Nielsen, P.Blaesild and G.Schou), pp. 335–352. Aarhus: University of Aarhus. (
- (1997a) The direct use of likelihood for significance testing. Statist. Comput., 7, 247–252.
- (1997b) Commentary on the paper by Murray Aitkin, and on discussion by Mervyn Stone. Statist. Comput., 7, 265–269.
- 1986) How biased is the apparent error rate of a prediction rule? J. Am. Statist. Ass., 81, 461–470. (
- 2001) Bayesian analyses of longitudinal binary data using markov regression models of unknown order. Statist. Med., 20, 755–770. , and (
- 1999) Bayesian inference for prevalence in longitudinal two-phase studies. Biometrics, 55, 1145–1150. , and (
- 1985) Diagnostics for smoothing splines. J. R. Statist. Soc. B, 47, 332–341. (
- 1986) Diagnostics for penalized least-squares estimators. Statist. Probab. Lett., 4, 265–272. and (
- 1993) A likelihood-based method for analysing longitudinal binary responses. Biometrika, 80, 141–151. and (
- 1994) Bayesian model choice: asymptotics and exact calculations. J. R. Statist. Soc. B, 56, 501–514. and (
- 2000) Conditional categorical response models with application to treatment of acute myocardial infarction. Appl. Statist., 49, 171–186. , , , and (
- 1998) Model choice: a minimum posterior predictive loss approach. Biometrika, 85, 1–11. and (
- 2002) Inequalities between expected marginal log likelihoods with implications for likelihood-based model comparison. Technical Report. Department of Statistics, University of Connecticut, Storrs. and (
- Gilks, W. R., Richardson, S. and Spiegelhalter, D. J. (eds) (1996) Markov Chain Monte Carlo in Practice. New York: Chapman and Hall.
- 1993) Random-effects models for longitudinal data using Gibbs sampling. Biometrics, 49, 441–453. , , and (
- 1956) The surprise index for the multivariate normal distribution. Ann. Math. Statist., 27, 1130–1135. (
- 2002) Hidden Markov models and disease mapping. J. Am. Statist. Ass., to be published. and (
- 2001) MCMC methods for computing Bayes factors: a comparative review. J. Am. Statist. Ass., 96, 1122–1132. and (
- 1990) Generalized Additive Models. London: Chapman and Hall. and (
- 2001) Counting degrees of freedom in hierarchical and other richly-parameterised models. Biometrika, 88, 367–379. and (
- 1967) The behaviour of maximum likelihood estimates under non-standard conditions. In Proc. 5th Berkeley Symp. Mathematical Statistics and Probability (eds L. M.LeCam and J.Neyman), vol. 1, pp. 221–233. Berkeley: University of California Press. (
- 1995) Bayes factors and model uncertainty. J. Am. Statist. Ass., 90, 773–795. and (
- 1999) Bayesian model choice: what and why? In Bayesian Statistics 6 (eds J. M.Bernardo, J. O.Berger, A. P.Dawid and A. F. M.Smith ), pp. 343–370. Oxford: Oxford University Press. , and (
- 1970) A correspondence between Bayesian estimation on stochastic processes and smoothing by splines. Ann. Math. Statist., 41, 495–502. and (
- 1951) On information and sufficiency. Ann. Math. Statist., 22, 79–86. and (
- 1982) Random effects models for longitudinal data. Biometrics, 38, 963–974. and (
- 1995) Predictive model selection. J. R. Statist. Soc. B, 57, 247–262. and (
- 1996) Hierarchical generalized linear models (with discussion). J. R. Statist. Soc. B, 58, 619–678. and (
- 1995) Splines from a Bayesian point of view. Test, 4, 63–81. (
- (2000) Reference priors for shrinkage and smoothing parameters. J. Statist. Planng Inf., 90, 245–274.
- 1972) Bayes estimates for the linear model (with discussion). J. R. Statist. Soc. B, 34, 1–44. and (
- 1992) Bayesian interpolation. Neur. Computn, 4, 415–447. (
- (1995) Probable networks and plausible predictions—a review of practical Bayesian methods for supervised neural networks. Netwrk Computn Neur. Syst., 6, 469–505.
- 1989) Generalized Linear Models, 2nd edn. London: Chapman and Hall. and (
- 1992) Performing likelihood ratio tests with multiply imputed data sets. Biometrika, 79, 103–112. and (
- 1992) The
*effective*number of parameters: an analysis of generalization and regularization in nonlinear learning systems. In Advances in Neural Information Processing Systems 4 (eds J. E.Moody, S. J.Hanson and R. P.Lippmann ), pp. 847–854. San Mateo: Morgan Kaufmann. ( - 1994) Network information criterion—determining the number of hidden units for artificial neural network models. IEEE Trans. Neur. Netwrks, 5, 865–872. , and (
- 2000) Reference Bayesian methods for generalised linear mixed models. J. Am. Statist. Ass., 95, 227–237. and (
- 1988) A Bayesian model selection criterion. Technical Report. University of Washington, Seattle. (
- 1999) The Bayesian analysis of a pivotal pharmacokinetic study. Statist. Meth. Med. Res., 8, 195–216. , , and (
- 1997) On Bayesian analysis of mixtures with an unknown number of components (with discussion). J. R. Statist. Soc. B, 59, 731–792. and (
- 1996) Pattern Recognition and Neural Networks. Cambridge: Cambridge University Press. (
- 1978) Information criteria for choice of regression models: a comment. Econometrica, 46, 1273–1291. (
- 1978) Estimating the dimension of a model. Ann. Statist., 6, 461–466. (
- 1994) Parameterizations for natural exponential-families with quadratic variance functions. J. Am. Statist. Ass., 89, 1471–1482. (
- 2000) WinBUGS Version 1.3 User Manual. Cambridge: Medical Research Council Biostatistics Unit. (Available from http://www.mrcbsu.cam.ac.uk/bugs.) , and (
- 1996) BUGS Examples Volume 1, Version 0.5 (Version ii). Cambridge: Medical Research Council Biostatistics Unit. , , and (
- 1977) An asymptotic equivalence of choice of model by cross-validation and Akaike's criterion. J. R. Statist. Soc. B, 39, 44–47. (
- 1976) Distribution of informational statistics and a criterion for model fitting (in Japanese). Suri-Kagaku, 153, 12–18. (
- 1999) Bayesian neural networks with correlated residuals. In IJCNN’99: Proc. 1999. Int. Joint Conf. Neural Networks. New York: Institute of Electrical and Electronic Engineers. and (
- 1978) Improper priors, spline smoothing and the problem of guarding against model errors in regressions. J. R. Statist. Soc. B, 40, 364–372. (
- (1983) Bayesian ‘‘confidence intervals’’ for the cross-validated smoothing spline. J. R. Statist. Soc. B, 45, 133–150.
- (1990) Spline Models for Observational Data. Philadelphia: Society for Industrial and Applied Mathematics.
- 1998) On measuring and correcting the effects of data mining and model selection. J. Am. Statist. Ass., 93, 120–131. (
- 1998) Evaluation of highly complex modeling procedures with binomial and Poisson data. Technical Report. Graduate School of Business, University of Chicago, Chicago. and (
- 1991) Generalised linear models with random effects; a Gibbs sampling approach. J. Am. Statist. Ass., 86, 79–86. and (
- 2000) Comparing hierarchical models for spatio-temporally misaligned data using the deviance information criterion. Statist. Med., 19, 2265–2278. and (

### References in the discussion

- 1991) Posterior Bayes factors (with discussion). J. R. Statist. Soc. B, 53, 111–142. (
- 1973) Information theory and an extension of the maximum likelihood principle. In Proc. 2nd Int. Symp. Information Theory (eds B. N.Petrov and F.Csáki), pp. 267–281. Budapest: Akadémiai Kiadó. (
- 1980) A note on the generalized information criterion for choice of a model. Biometrika, 67, 413–418. (
- 2000) Robust Diagnostic Regression Analysis. New York: Springer. and (
- — (2002) Forward search added variable
*t*tests and the effect of masked outliers on model selection and transformation. Technical Report LSERR73. London School of Economics and Political Science, London. - Expected information as expected utility. Ann. Statist., 7, 686–690. (1979)
- — (1999) Nested hypothesis testing: the Bayesian reference criterion (with discussion). In Bayesian Statistics 6 (eds J. M.Bernardo, J. O.Berger, A. P.Dawid and A. F. M.Smith), pp. 101–130. Oxford: Oxford University Press.
- 1994) Bayesian Theory. New York: Wiley. and (
- 2002) Intrinsic estimation. 7th Valencia Int. Meet. Bayesian Statistics, Tenerife, June. and (
- 1998) Model Selection and Inference: a Practical Information-theoretic Approach. New York: Springer. and (
- — (2002) Model Selection and Multimodel Inference: a Practical Information-theoretical Approach, 2nd edn. New York: Springer.
- 2000) Mixture models, latent variables and partitioned importance sampling.
*Technical**Report*. Paris. , and ( - 2000) Computational and inferential difficulties with mixtures posterior distribution. J. Am. Statist. Ass., 95, 957–979. , and (
- 1991) Experts in Uncertainty. Oxford: Oxford University Press. (
- 1999) Probabilistic Networks and Expert Systems. New York: Springer. , , and (
- 1999) Nonconjugate Bayesian estimation of covariance matrices and its use in hierarchical models. J. Am. Statist. Ass., 94, 1254–1263. and (
- — (2001) Shrinkage estimators for covariance matrices. Biometrics, 57, 1173–1184.
- 1984) Statistical theory: the prequential approach. J. R. Statist. Soc. A, 147, 278–292. (
- — (1986) Probability forecasting. In Encyclopedia of Statistical Sciences, vol. 7 (eds S.Kotz, N. L.Johnson and C. B.Read), pp. 210–218. New York: Wiley-Interscience.
- — (1991) Fisherian inference in likelihood and prequential frames of reference (with discussion). J. R. Statist. Soc. B, 53, 79–109.
- — (1992a) Prequential analysis, stochastic complexity and Bayesian inference (with discussion). In Bayesian Statistics 4 (eds J. M.Bernardo, J. O.Berger, A. P.Dawid and A. F. M.Smith), pp. 109–125. Oxford: Oxford University Press.
- — (1992b) Prequential data analysis. In Current Issues in Statistical Inference: Essays in Honor of D. Basu (eds M.Ghosh and P. K.Pathak), pp. 113–126. Hayward: Institute of Mathematical Statistics.
- 1999) Discussion on ‘Decision models in screening for breast cancer’ (by G. Parmigiani). In Bayesian Statistics 6 (eds J. M.Bernardo, J. O.Berger, A. P.Dawid and A. F. M.Smith), pp. 541–543. Oxford: Oxford University Press. (
- 2000) A case study of stochastic optimization in health policy: problem formulation and preliminary results. J. Global Optimzn, 18, 399–416. and (
- 2002) Model choice in qualitative regression models. J. Statist. Planng Inf., to be published. and (
- 1986) How biased is the apparent error rate of a prediction rule? J. Am. Statist. Ass., 81, 461–470. (
- 2002) Stochastic optimization: a review. Int. Statist. Rev.,to be published. and (
- 2002) Cluster modelling for disease rate mapping. In Spatial Cluster Modelling (eds A. B.Lawson and D.Denison), ch. 8. New York: CRC Press. and (
- 1996) Model determination using sampling-based methods. In Markov Chain Monte Carlo in Practice (eds W. R.Gilks, S.Richardson and D. J.Spiegelhalter), pp. 145–162. London: Chapman and Hall. (
- 1992) Model determination using predictive distributions with implementation via sampling-based methods (with discussion). In Bayesian Statistics 4 (eds J. M.Bernardo, J. O.Berger, A. P.Dawid and A. F. M.Smith), pp. 147–167. Oxford: Oxford University Press. , and (
- 1996) Posterior predictive assessment of model fitness via realized discrepancies (with discussion). Statist. Sin., 6, 733–807. , and (
- 1952) Rational decisions. J. R. Statist. Soc. B, 14, 107–114. (
- 2002) Hidden Markov models and disease mapping. J. Am. Statist. Ass., to be published. and (
- Counting degrees of freedom in hierarchical and other richly-parameterised models. Biometrika, 88, 367–379. and (2001)
- 1999) Bayesian wavelet analysis with a model complexity prior. In Bayesian Statistics 6 (eds J. M.Bernardo, J. O.Berger, A. P.Dawid and A. F. M.Smith), pp. 769–776. Oxford: Oxford University Press. and (
- Bayes factors and model uncertainty. J. Am. Statist. Ass., 90, 773–795. and (1995)
- 1999) Bayesian model choice: what and why? In Bayesian Statistics 6 (eds J. M.Bernardo, J. O.Berger, A. P.Dawid and A. F. M.Smith), pp. 343–370. Oxford: Oxford University Press. , and (
- 2001) Bayesian model discrimination in the analysis of capture-recapture and related data.
*PhD Thesis*. School of Mathematics, University of Bristol, Bristol. ( - 2001) Bayesian estimation of census undercount. Biometrika, 88, 317–336. and (
- 1996) Generalised information criteria in model selection. Biometrika, 83, 875–890. and (
- 1988) Local computations with probabilities on graphical structures and their application to expert systems (with discussion). J. R. Statist. Soc. B, 50, 157–224. and (
- 2000) Cluster modelling of disease incidence via rjmcmc methods: a comparative evaluation. Statist. Med., 19, 2361–2376. (
- 1996) Hierarchical generalized linear models (with discussion). J. R. Statist. Soc. B, 58, 619–678. and (
- — (2001a) Hierarchical generalized linear models: a synthesis of generalized linear models, random effect models and structured dispersions. Biometrika, 88, 987–1006.
- — (2001b) Modelling and analysing correlated non-normal data. Statist. Modlng, 1, 3–16.
- 2003) Choosing a model selection strategy. Scand. J. Statist., to be published. and (
- 1991) Model selection and accounting for model uncertainty in graphical models using Occam's window.
*Technical Report 213*. Department of Statistics, University of Washington, Seattle. and ( - 2002) Perfect sampling for point process cluster modelling. In Spatial Cluster Modelling (eds A. B.Lawson and D.Denison), ch. 5. New York: CRC Press. and (
- Performing likelihood ratio tests with multiply imputed data sets. Biometrika, 79, 103–112. and (1992)
- 1998) A robust Bayesian look at the theory of precise measurement. In Decision Research from Bayesian Approaches to Normative Systems (eds J.Shantan
*et al*.). Boston: Kluwer. , and ( - 1996) Applied Linear Statistical Models, 4th edn. New York: McGraw-Hill. , , and (
- 1991) Robust Bayesian credible intervals and prior ignorance. Int. Statist. Rev., 58, 1–23. and (
- 2002) Some criteria for Bayesian model choice.
*Preprint*. (Available from http://calvin.iarc.fr/martyn/papers/.) ( - 1981) Spectral Analysis and Time Series. London: Academic Press. (
- 1996) Intrinsic loss functions. Theory Decsn, 40, 191–214. (
- 1997) An asymptotic theory for linear model selection. Statist. Sin., 7, 221–264. (
- 1999) On efficient probability forecasting systems. Biometrika, 86, 765–784. and (
- — (2000) Consistency in misspecified models. Research Report 218. Department of Statistical Science, University College London, London. (Available from: http://www.ucl.ac.uk/Stats/research/abs00.html218.)
- 1996) Plausible Bayesian games. In Bayesian Statistics 5 (eds J. M.Bernardo, J. O.Berger, A. P.Dawid and A. F. M.Smith), pp. 387–406. Oxford: Oxford University Press. (
- 1974) Cross-validatory choice and assessment of statistical predictions (with discussion). J. R. Statist. Soc. B, 36, 111–147. (
- — (1977) An asymptotic equivalence of choice of model by cross-validation and Akaike's criterion. J. R. Statist. Soc. B, 36, 44–47.
- 2001) Bayesian model assessment and selection using expected utilities. DSc Dissertation. Helsinki University of Technology, Helsinki. (Available from http://lib.hut.fi/Diss/2001/isbn9512257653/.) (
- 2002a) Bayesian model assessment and comparison using cross-validation predictive densities. Neur. Computn, 14, in the press. and (
- — (2002b) Cross-validation, information criteria, expected utilities and the effective number of parameters. To be published.
- 2000) Bayesian information criterion for censored survival models. Biometrics, 56, 256–262. and (
- 1981) A statistic for allocating
*C*_{p}to individual cases. Technometrics, 23, 27–31. ( - On measuring and correcting the effects of data mining and model selection. J. Am. Statist. Ass., 93, 120–131. (1998)
- Comparing hierarchical models for spatio-temporally misaligned data using the deviance information criterion. Statist. Med., 19, 2265–2278. and (2000)