• Open Access

Predictive accuracy of risk factors and markers: a simulation study of the effect of novel markers on different performance measures for logistic regression models

Authors

  • Peter C. Austin,

    Corresponding author
    1. Institute for Clinical Evaluative Sciences, Toronto, ON, Canada
    2. Department of Health Management, Policy and Evaluation, University of Toronto, Toronto, ON, Canada
    3. Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada
    • Correspondence to: Peter C. Austin, Institute for Clinical Evaluative Sciences, G1 06, 2075 Bayview Avenue, Toronto, ON, M4N 3M5, Canada.

      E-mail: peter.austin@ices.on.ca

    Search for more papers by this author
  • Ewout W. Steyerberg

    1. Department of Public Health, Erasmus Medical Centre, The Netherlands
    Search for more papers by this author

Abstract

The change in c-statistic is frequently used to summarize the change in predictive accuracy when a novel risk factor is added to an existing logistic regression model. We explored the relationship between the absolute change in the c-statistic, Brier score, generalized R2, and the discrimination slope when a risk factor was added to an existing model in an extensive set of Monte Carlo simulations. The increase in model accuracy due to the inclusion of a novel marker was proportional to both the prevalence of the marker and to the odds ratio relating the marker to the outcome but inversely proportional to the accuracy of the logistic regression model with the marker omitted. We observed greater improvements in model accuracy when the novel risk factor or marker was uncorrelated with the existing predictor variable compared with when the risk factor has a positive correlation with the existing predictor variable. We illustrated these findings by using a study on mortality prediction in patients hospitalized with heart failure. In conclusion, the increase in predictive accuracy by adding a marker should be considered in the context of the accuracy of the initial model. Copyright © 2012 John Wiley & Sons, Ltd.

Ancillary