Volume 61, Issue 1

Survival Model Predictive Accuracy and ROC Curves

Patrick J. Heagerty

Department of Biostatistics, University of Washington, P.O. Box 357232, Seattle, Washington 98195‐7232, U.S.A.

Search for more papers by this author
Yingye Zheng

Fred Hutchinson Cancer Research Center, 1100 Fairview Avenue North, MP 702, P.O. Box 19024, Seattle, Washington 98109‐1024, U.S.A.

Search for more papers by this author
First published: 28 February 2005
Citations: 618
Email: heagerty@u.washington.edu

Abstract

Summary The predictive accuracy of a survival model can be summarized using extensions of the proportion of variation explained by the model, or R2, commonly used for continuous response models, or using extensions of sensitivity and specificity, which are commonly used for binary response models. In this article we propose new time‐dependent accuracy summaries based on time‐specific versions of sensitivity and specificity calculated over risk sets. We connect the accuracy summaries to a previously proposed global concordance measure, which is a variant of Kendall's tau. In addition, we show how standard Cox regression output can be used to obtain estimates of time‐dependent sensitivity and specificity, and time‐dependent receiver operating characteristic (ROC) curves. Semiparametric estimation methods appropriate for both proportional and nonproportional hazards data are introduced, evaluated in simulations, and illustrated using two familiar survival data sets.

Number of times cited according to CrossRef: 618

  • Big Data, Personalized Medicine and Network Pharmacology: Beyond the Current Paradigms, Approaching Complex Diseases, 10.1007/978-3-030-32857-3_5, (95-110), (2020).
  • Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networks, Medical Image Analysis, 10.1016/j.media.2020.101789, (101789), (2020).
  • Combining tumor response and personalized risk assessment: potential for adaptation of concurrent chemotherapy in locoregionally advanced nasopharyngeal carcinoma in the intensity-modulated radiotherapy era, Radiotherapy and Oncology, 10.1016/j.radonc.2020.10.005, (2020).
  • Oblique Survival Trees in Discrete Event Time Analysis, IEEE Journal of Biomedical and Health Informatics, 10.1109/JBHI.2019.2908773, 24, 1, (247-258), (2020).
  • Prospective Associations between BOLD Markers of Response Inhibition and the Transition to Frequent Binge Drinking, Alcoholism: Clinical and Experimental Research, 10.1111/acer.14261, 44, 2, (463-469), (2020).
  • Radiomics of Coronary Artery Calcium in the Framingham Heart Study, Radiology: Cardiothoracic Imaging, 10.1148/ryct.2020190119, 2, 1, (e190119), (2020).
  • Surgically Resected Esophageal Squamous Cell Carcinoma: Patient Survival and Clinicopathological Prognostic Factors, Scientific Reports, 10.1038/s41598-020-62028-5, 10, 1, (2020).
  • Predictive accuracy of markers or risk scores for interval censored survival data, Statistics in Medicine, 10.1002/sim.8547, 39, 18, (2437-2446), (2020).
  • Magnetic resonance imaging features of tumor and lymph node to predict clinical outcome in node-positive cervical cancer: a retrospective analysis, Radiation Oncology, 10.1186/s13014-020-01502-w, 15, 1, (2020).
  • Smoothed time‐dependent receiver operating characteristic curve for right censored survival data, Statistics in Medicine, 10.1002/sim.8671, 39, 24, (3373-3396), (2020).
  • Utility of the “omics” in kidney disease: Methods of analysis, sampling considerations, and technical approaches in renal biomarkers, Kidney Biomarkers, 10.1016/B978-0-12-815923-1.00002-X, (19-153), (2020).
  • Dielectric behavior of thin films of unsaturated polyester‐resin/carbon nanotube semiconductor composites, Polymers for Advanced Technologies, 10.1002/pat.4938, 31, 10, (2172-2184), (2020).
  • Power of a dual‐use SNP panel for pedigree reconstruction and population assignment, Ecology and Evolution, 10.1002/ece3.6645, 10, 17, (9522-9531), (2020).
  • Dynamic-DeepHit: A Deep Learning Approach for Dynamic Survival Analysis With Competing Risks Based on Longitudinal Data, IEEE Transactions on Biomedical Engineering, 10.1109/TBME.2019.2909027, 67, 1, (122-133), (2020).
  • A Retrospective Cohort Study of Risk of Gallbladder Disease in Acute Cerebral Infarction Patients, Journal of the Korean Neurological Association, 10.17340/jkna.2020.1.3, 38, 1, (16-24), (2020).
  • Testing longitudinal data for prognostication in ambulatory heart failure patients with reduced ejection fraction. A proof of principle from the GISSI-HF database, International Journal of Cardiology, 10.1016/j.ijcard.2020.03.064, (2020).
  • A covariate‐specific time‐dependent receiver operating characteristic curve for correlated survival data, Statistics in Medicine, 10.1002/sim.8550, 39, 19, (2477-2489), (2020).
  • Nine glycolysis-related gene signature predicting the survival of patients with endometrial adenocarcinoma, Cancer Cell International, 10.1186/s12935-020-01264-1, 20, 1, (2020).
  • Predicting survival and longevity of sows using purebred and crossbred data1, Translational Animal Science, 10.1093/tas/txaa073, 4, 2, (2020).
  • A Practical Perspective on the Concordance Index for the Evaluation and Selection of Prognostic Time-to-Event Models, Journal of Biomedical Informatics, 10.1016/j.jbi.2020.103496, (103496), (2020).
  • Establishment of a novel cell cycle-related prognostic signature predicting prognosis in patients with endometrial cancer, Cancer Cell International, 10.1186/s12935-020-01428-z, 20, 1, (2020).
  • Developing an Improved Statistical Approach for Survival Estimation in Bone Metastases Management: The Bone Metastases Ensemble Trees for Survival (BMETS) Model, International Journal of Radiation Oncology*Biology*Physics, 10.1016/j.ijrobp.2020.05.023, (2020).
  • Evolutionary Dendritic Neural Model for Classification Problems, Complexity, 10.1155/2020/6296209, 2020, (1-13), (2020).
  • Metabolic correlates of prevalent mild cognitive impairment and Alzheimer's disease in adults with Down syndrome, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12028, 12, 1, (2020).
  • Pseudodementia, pseudo‐pseudodementia, and pseudodepression, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12027, 12, 1, (2020).
  • Plasma pyroglutamate‐modified amyloid beta differentiates amyloid pathology, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12029, 12, 1, (2020).
  • Disease severity at the time of initial cognitive assessment is related to prior health‐care resource use burden, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12093, 12, 1, (2020).
  • Clinical and biomarker trajectories in sporadic Alzheimer's disease: A longitudinal study, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12095, 12, 1, (2020).
  • Subjective cognitive and non‐cognitive complaints and brain MRI biomarkers in the MEMENTO cohort, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12051, 12, 1, (2020).
  • Elevated plasma neurofilament light in aging reflects brain white‐matter alterations but does not predict cognitive decline or Alzheimer's disease, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12050, 12, 1, (2020).
  • Cerebrospinal fluid biomarkers of Alzheimer's disease in a cohort of adults with Down syndrome, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12057, 12, 1, (2020).
  • Down syndrome: Distribution of brain amyloid in mild cognitive impairment, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12013, 12, 1, (2020).
  • Blood DNA methylation signatures to detect dementia prior to overt clinical symptoms, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12056, 12, 1, (2020).
  • Commentary on Composite cognitive and functional measures for early stage Alzheimer's disease trials, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12012, 12, 1, (2020).
  • Reduced cortical thickness in World Trade Center responders with cognitive impairment, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12059, 12, 1, (2020).
  • Response to Schneider and Goldberg “Composite cognitive and functional measures for early stage Alzheimer's disease trials”, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12015, 12, 1, (2020).
  • Collective effects of age, sex, genotype, and cognitive status on fitness outcomes, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12058, 12, 1, (2020).
  • Clearance of interstitial fluid (ISF) and CSF (CLIC) group—part of Vascular Professional Interest Area (PIA), Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12053, 12, 1, (2020).
  • Obesity impacts brain metabolism and structure independently of amyloid and tau pathology in healthy elderly, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12052, 12, 1, (2020).
  • Lower practice effects as a marker of cognitive performance and dementia risk: A literature review, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12055, 12, 1, (2020).
  • Dual sensory impairment in older adults and risk of dementia from the GEM Study, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12054, 12, 1, (2020).
  • The Intergenerational Impact of a Slow Pandemic: HIV and Children, New Directions for Child and Adolescent Development, 10.1002/cad.20358, 2020, 171, (139-150), (2020).
  • Composite cognitive and functional measures for early stage Alzheimer's disease trials, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12017, 12, 1, (2020).
  • In vivo network models identify sex differences in the spread of tau pathology across the brain, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12016, 12, 1, (2020).
  • NTAL is associated with treatment outcome, cell proliferation and differentiation in acute promyelocytic leukemia, Scientific Reports, 10.1038/s41598-020-66223-2, 10, 1, (2020).
  • Immune-related long non-coding RNA signature identified prognosis and immunotherapeutic efficiency in bladder cancer (BLCA), Cancer Cell International, 10.1186/s12935-020-01362-0, 20, 1, (2020).
  • Novel gene signatures for prognosis prediction in ovarian cancer, Journal of Cellular and Molecular Medicine, 10.1111/jcmm.15601, 24, 17, (9972-9984), (2020).
  • Small nucleolar RNAs signature (SNORS) identified clinical outcome and prognosis of bladder cancer (BLCA), Cancer Cell International, 10.1186/s12935-020-01393-7, 20, 1, (2020).
  • A cluster of metabolism-related genes predict prognosis and progression of clear cell renal cell carcinoma, Scientific Reports, 10.1038/s41598-020-67760-6, 10, 1, (2020).
  • ROC curves for clinical prediction models part 1. ROC plots showed no added value above the AUC when evaluating the performance of clinical prediction models, Journal of Clinical Epidemiology, 10.1016/j.jclinepi.2020.01.028, 126, (207-216), (2020).
  • Estimation of incident dynamic AUC in practice, Computational Statistics & Data Analysis, 10.1016/j.csda.2020.107095, (107095), (2020).
  • Cu and Zn isotope ratio variations in plasma for survival prediction in hematological malignancy cases, Scientific Reports, 10.1038/s41598-020-71764-7, 10, 1, (2020).
  • Ranking-Based Survival Prediction on Histopathological Whole-Slide Images, Medical Image Computing and Computer Assisted Intervention – MICCAI 2020, 10.1007/978-3-030-59722-1_41, (428-438), (2020).
  • A numerical strategy to evaluate performance of predictive scores via a copula‐based approach, Statistics in Medicine, 10.1002/sim.8566, 39, 20, (2671-2684), (2020).
  • A Gleason score-related outcome model for human prostate cancer: a comprehensive study based on weighted gene co-expression network analysis, Cancer Cell International, 10.1186/s12935-020-01230-x, 20, 1, (2020).
  • Inference about time-dependent prognostic accuracy measures in the presence of competing risks, BMC Medical Research Methodology, 10.1186/s12874-020-01100-0, 20, 1, (2020).
  • Incorporating longitudinal biomarkers for dynamic risk prediction in the era of big data: A pseudo‐observation approach, Statistics in Medicine, 10.1002/sim.8687, 39, 26, (3685-3699), (2020).
  • The Alzheimer's Biomarker Consortium‐Down Syndrome: Rationale and methodology, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring, 10.1002/dad2.12065, 12, 1, (2020).
  • Cardiac and Stress Biomarkers and Chronic Kidney Disease Progression: The CRIC Study, Clinical Chemistry, 10.1373/clinchem.2019.305797, 65, 11, (1448-1457), (2020).
  • Practical guidance for P2Y12 inhibitors in acute myocardial infarction undergoing percutaneous coronary intervention, European Heart Journal - Cardiovascular Pharmacotherapy, 10.1093/ehjcvp/pvaa005, (2020).
  • Progressive Desmoid Tumor: Radiomics Compared With Conventional Response Criteria for Predicting Progression During Systemic Therapy—A Multicenter Study by the French Sarcoma Group, American Journal of Roentgenology, 10.2214/AJR.19.22635, (1-10), (2020).
  • Approval policies for modifications to machine learning‐based software as a medical device: A study of bio‐creep, Biometrics, 10.1111/biom.13379, 0, 0, (2020).
  • ROC‐guided survival trees and ensembles, Biometrics, 10.1111/biom.13213, 0, 0, (2020).
  • Can radiomics improve the prediction of metastatic relapse of myxoid/round cell liposarcomas?, European Radiology, 10.1007/s00330-019-06562-5, (2020).
  • smoothROCtime: an R package for time-dependent ROC curve estimation, Computational Statistics, 10.1007/s00180-020-00955-7, (2020).
  • MR image phenotypes may add prognostic value to clinical features in IDH wild-type lower-grade gliomas, European Radiology, 10.1007/s00330-020-06683-2, (2020).
  • Identification of core biomarkers associated with pathogenesis and prognostic outcomes of laryngeal squamous-cell cancer using bioinformatics analysis, European Archives of Oto-Rhino-Laryngology, 10.1007/s00405-020-05856-5, (2020).
  • Validation of Novel Prognostic Biomarkers for Early-Stage Clear-Cell, Endometrioid and Mucinous Ovarian Carcinomas Using Immunohistochemistry, Frontiers in Oncology, 10.3389/fonc.2020.00162, 10, (2020).
  • DeepSurvNet: deep survival convolutional network for brain cancer survival rate classification based on histopathological images, Medical & Biological Engineering & Computing, 10.1007/s11517-020-02147-3, (2020).
  • Cumulative/dynamic ROC curve estimation under interval censorship, Journal of Statistical Computation and Simulation, 10.1080/00949655.2020.1736071, (1-21), (2020).
  • Machine learning and radiomic phenotyping of lower grade gliomas: improving survival prediction, European Radiology, 10.1007/s00330-020-06737-5, (2020).
  • Assessing the accuracy of predictive models with interval-censored data, Biostatistics, 10.1093/biostatistics/kxaa011, (2020).
  • Quantifying the Predictive Accuracy of a Polygenic Risk Score for Predicting Incident Cancer Cases : Application to the CARTaGENE Cohort, Frontiers in Genetics, 10.3389/fgene.2020.00408, 11, (2020).
  • Prognostic nomogram based on immune scores for laryngeal squamous cell cancer, European Archives of Oto-Rhino-Laryngology, 10.1007/s00405-020-06189-z, (2020).
  • Application of the Minority Stress Theory: Understanding the Mental Health of Undocumented Latinx Immigrants, American Journal of Community Psychology, 10.1002/ajcp.12455, 0, 0, (2020).
  • Weighed Gene Coexpression Network Analysis of Features That Control Cancer Stem Cells Reveals Prognostic Biomarkers in Lung Adenocarcinoma, Frontiers in Genetics, 10.3389/fgene.2020.00311, 11, (2020).
  • Association Between Lymph Node Ratio and Survival in Patients with Pathological Stage II/III Gastric Cancer, Annals of Surgical Oncology, 10.1245/s10434-020-08616-1, (2020).
  • A field study of the impact of psychotropic medication on delinquency and juvenile justice system involvement among a high risk sample of children and adolescents, Journal of Offender Rehabilitation, 10.1080/10509674.2020.1767258, (1-20), (2020).
  • ASSOCIATION OF DIAGNOSIS CODE-BASED AND LABORATORY RESULTS-BASED KIDNEY FUNCTION WITH DEVELOPMENT OF VISION THREATENING DIABETIC RETINOPATHY, Ophthalmic Epidemiology, 10.1080/09286586.2020.1773869, (1-6), (2020).
  • Bioinformatics Analysis of Prognostic miRNA Signature and Potential Critical Genes in Colon Cancer, Frontiers in Genetics, 10.3389/fgene.2020.00478, 11, (2020).
  • Comprehensive Characterization of the Prognosis Value of Alternative Splicing Events in Acute Myeloid Leukemia, DNA and Cell Biology, 10.1089/dna.2020.5534, (2020).
  • A Multi-State Approach to Modelling Intermediate Events and Multiple Mortgage Loan Outcomes, Risks, 10.3390/risks8020064, 8, 2, (64), (2020).
  • Tumour microenvironment (TME) characterization identified prognosis and immunotherapy response in muscle-invasive bladder cancer (MIBC), Cancer Immunology, Immunotherapy, 10.1007/s00262-020-02649-x, (2020).
  • Measuring the temporal prognostic utility of a baseline risk score, Lifetime Data Analysis, 10.1007/s10985-020-09503-3, (2020).
  • Radiomics risk score may be a potential imaging biomarker for predicting survival in isocitrate dehydrogenase wild-type lower-grade gliomas, European Radiology, 10.1007/s00330-020-07089-w, (2020).
  • The Positive Lymph Node Ratio Predicts Survival in T1−4N1−3M0 Non-Small Cell Lung Cancer: A Nomogram Using the SEER Database, Frontiers in Oncology, 10.3389/fonc.2020.01356, 10, (2020).
  • Multi-Parametric Deep Learning Model for Prediction of Overall Survival after Postoperative Concurrent Chemoradiotherapy in Glioblastoma Patients, Cancers, 10.3390/cancers12082284, 12, 8, (2284), (2020).
  • Long-Term Bleeding Risk Prediction with Dual Antiplatelet Therapy After Acute Coronary Syndromes Treated Without Revascularization, Circulation: Cardiovascular Quality and Outcomes, 10.1161/CIRCOUTCOMES.120.006582, (2020).
  • Hepatocellular Carcinoma Within the Milan Criteria: A Novel Inflammation-Based Nomogram System to Assess the Outcomes of Ablation, Frontiers in Oncology, 10.3389/fonc.2020.01764, 10, (2020).
  • The Impact of Churn on Client Value in Health Insurance, Evaluation Using a Random Forest Under Various Censoring Mechanisms, Journal of the American Statistical Association, 10.1080/01621459.2020.1764364, (1-12), (2020).
  • Hematopoietic Gene Expression Regulation Through m6A Methylation Predicts Prognosis in Stage III Colorectal Cancer, Frontiers in Oncology, 10.3389/fonc.2020.572708, 10, (2020).
  • Baseline and Dynamic Risk Predictors of Appropriate Implantable Cardioverter Defibrillator Therapy, Journal of the American Heart Association, 10.1161/JAHA.120.017002, (2020).
  • Two-Stage Adaptive Design for Prognostic Biomarker Signatures with a Survival Endpoint, Statistics in Biopharmaceutical Research, 10.1080/19466315.2020.1835710, (1-27), (2020).
  • Validation of discrete time‐to‐event prediction models in the presence of competing risks, Biometrical Journal, 10.1002/bimj.201800293, 62, 3, (643-657), (2019).
  • Comparison of pre‐ and post‐bronchodilator lung function as predictors of mortality: The HUNT Study, Respirology, 10.1111/resp.13648, 25, 4, (401-409), (2019).
  • An EMT‐related gene signature for the prognosis of human bladder cancer, Journal of Cellular and Molecular Medicine, 10.1111/jcmm.14767, 24, 1, (605-617), (2019).
  • Assessing predictive accuracy of survival regressions subject to nonindependent censoring, Statistics in Medicine, 10.1002/sim.8420, 39, 4, (469-480), (2019).
  • Bioinformatic profiling identifies a platinum‐resistant–related risk signature for ovarian cancer, Cancer Medicine, 10.1002/cam4.2692, 9, 3, (1242-1253), (2019).
  • An outcome model for human bladder cancer: A comprehensive study based on weighted gene co‐expression network analysis, Journal of Cellular and Molecular Medicine, 10.1111/jcmm.14918, 24, 3, (2342-2355), (2019).
  • Synthesis and Characterization of an Epidermal Growth Factor Receptor‐Selective RuII Polypyridyl–Nanobody Conjugate as a Photosensitizer for Photodynamic Therapy, ChemBioChem, 10.1002/cbic.201900419, 21, 4, (531-542), (2019).
  • See more

The full text of this article hosted at iucr.org is unavailable due to technical difficulties.