A Score Regression Approach to Assess Calibration of Continuous Probabilistic Predictions
Article first published online: 26 MAR 2010
© 2010, The International Biometric Society
Volume 66, Issue 4, pages 1295–1305, December 2010
How to Cite
Held, L., Rufibach, K. and Balabdaoui, F. (2010), A Score Regression Approach to Assess Calibration of Continuous Probabilistic Predictions. Biometrics, 66: 1295–1305. doi: 10.1111/j.1541-0420.2010.01406.x
- Issue published online: 26 MAR 2010
- Article first published online: 26 MAR 2010
- Received January 2009. Revised November 2009. Accepted January 2010.
- Continuous predictions;
- Scoring rules
Summary Calibration, the statistical consistency of forecast distributions and the observations, is a central requirement for probabilistic predictions. Calibration of continuous forecasts is typically assessed using the probability integral transform histogram. In this article, we propose significance tests based on scoring rules to assess calibration of continuous predictive distributions. For an ideal normal forecast we derive the first two moments of two commonly used scoring rules: the logarithmic and the continuous ranked probability score. This naturally leads to the construction of two unconditional tests for normal predictions. More generally, we propose a novel score regression approach, where the individual scores are regressed on suitable functions of the predictive variance. This conditional approach is applicable even for certain nonnormal predictions based on the Dawid–Sebastiani score. Two case studies illustrate that the score regression approach has typically more power in detecting miscalibrated forecasts than the other approaches considered, including a recently proposed technique based on conditional exceedance probability curves.