Two Criteria for Evaluating Risk Prediction Models
Article first published online: 14 DEC 2010
© 2010, The International Biometric Society No claim to original US Federal works
Volume 67, Issue 3, pages 1057–1065, September 2011
How to Cite
Pfeiffer, R. M. and Gail, M. H. (2011), Two Criteria for Evaluating Risk Prediction Models. Biometrics, 67: 1057–1065. doi: 10.1111/j.1541-0420.2010.01523.x
- Issue published online: 14 SEP 2011
- Article first published online: 14 DEC 2010
- Received January 2010. Revised August 2010. Accepted September 2010.
- Discriminatory accuracy;
- Lorenz curve;
- Risk models
Summary We propose and study two criteria to assess the usefulness of models that predict risk of disease incidence for screening and prevention, or the usefulness of prognostic models for management following disease diagnosis. The first criterion, the proportion of cases followed PCF (q), is the proportion of individuals who will develop disease who are included in the proportion q of individuals in the population at highest risk. The second criterion is the proportion needed to follow-up, PNF (p), namely the proportion of the general population at highest risk that one needs to follow in order that a proportion p of those destined to become cases will be followed. PCF (q) assesses the effectiveness of a program that follows 100q% of the population at highest risk. PNF (p) assess the feasibility of covering 100p% of cases by indicating how much of the population at highest risk must be followed. We show the relationship of those two criteria to the Lorenz curve and its inverse, and present distribution theory for estimates of PCF and PNF. We develop new methods, based on influence functions, for inference for a single risk model, and also for comparing the PCFs and PNFs of two risk models, both of which were evaluated in the same validation data.