Probability estimation with machine learning methods for dichotomous and multicategory outcome: Theory

Authors

  • Jochen Kruppa,

    1. Institut für Medizinische Biometrie und Statistik, Universität zu Lübeck, Universitätsklinikum Schleswig-Holstein, Campus Lübeck, 23562 Lübeck, Germany
    Search for more papers by this author
    • These authors contributed equally to the work.

  • Yufeng Liu,

    1. Department of Statistics and Operations Research, Carolina Center for Genome Sciences, The University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
    Search for more papers by this author
    • These authors contributed equally to the work.

  • Gérard Biau,

    1. Université Pierre et Marie Curie, France
    2. Département de Mathématiques et Applications, Ecole Normale Supérieure, 45 rue d'Ulm, France
    Search for more papers by this author
  • Michael Kohler,

    1. Fachbereich Mathematik, TU Darmstadt, 64289 Darmstadt, Germany
    Search for more papers by this author
  • Inke R. König,

    1. Institut für Medizinische Biometrie und Statistik, Universität zu Lübeck, Universitätsklinikum Schleswig-Holstein, Campus Lübeck, 23562 Lübeck, Germany
    Search for more papers by this author
  • James D. Malley,

    1. Center for Information Technology, National Institutes of Health, Bethesda, USA
    Search for more papers by this author
  • Andreas Ziegler

    Corresponding author
    1. Institut für Medizinische Biometrie und Statistik, Universität zu Lübeck, Universitätsklinikum Schleswig-Holstein, Campus Lübeck, 23562 Lübeck, Germany
    2. Zentrum für Klinische Studien Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
    Search for more papers by this author

Abstract

Probability estimation for binary and multicategory outcome using logistic and multinomial logistic regression has a long-standing tradition in biostatistics. However, biases may occur if the model is misspecified. In contrast, outcome probabilities for individuals can be estimated consistently with machine learning approaches, including k-nearest neighbors (k-NN), bagged nearest neighbors (b-NN), random forests (RF), and support vector machines (SVM). Because machine learning methods are rarely used by applied biostatisticians, the primary goal of this paper is to explain the concept of probability estimation with these methods and to summarize recent theoretical findings. Probability estimation in k-NN, b-NN, and RF can be embedded into the class of nonparametric regression learning machines; therefore, we start with the construction of nonparametric regression estimates and review results on consistency and rates of convergence. In SVMs, outcome probabilities for individuals are estimated consistently by repeatedly solving classification problems. For SVMs we review classification problem and then dichotomous probability estimation. Next we extend the algorithms for estimating probabilities using k-NN, b-NN, and RF to multicategory outcomes and discuss approaches for the multicategory probability estimation problem using SVM. In simulation studies for dichotomous and multicategory dependent variables we demonstrate the general validity of the machine learning methods and compare it with logistic regression. However, each method fails in at least one simulation scenario. We conclude with a discussion of the failures and give recommendations for selecting and tuning the methods. Applications to real data and example code are provided in a companion article (doi:10.1002/bimj.201300077).

Ancillary