Volume 32, Issue 14
Research Article

A unified inference procedure for a class of measures to assess improvement in risk prediction systems with survival data

Hajime Uno

Corresponding Author

Department of Biostatistics and Computational Biology, Dana Farber Cancer Institute, Boston, MA, U.S.A.

Department of Biostatistics, Harvard University, Boston, MA, U.S.A.

Correspondence to: Hajime Uno, Department of Biostatistics and Computational Biology, Dana Farber Cancer Institute, 450 Brookline Avenue, Boston, MA 02115, U.S.A.

E‐mail: huno@jimmy.harvard.edu

Search for more papers by this author
Lu Tian

Department of Health Research and Policy, Stanford University School of Medicine, Stanford, CA, U.S.A.

Search for more papers by this author
Tianxi Cai

Department of Biostatistics, Harvard University, Boston, MA, U.S.A.

Search for more papers by this author
Isaac S. Kohane

Division of Health Sciences and Technology, Harvard University and Massachusetts Institute of Technology, Cambridge, MA, U.S.A.

Search for more papers by this author
L.J. Wei

Department of Biostatistics, Harvard University, Boston, MA, U.S.A.

Search for more papers by this author
First published: 05 October 2012
Citations: 165

Abstract

Risk prediction procedures can be quite useful for the patient's treatment selection, prevention strategy, or disease management in evidence‐based medicine. Often, potentially important new predictors are available in addition to the conventional markers. The question is how to quantify the improvement from the new markers for prediction of the patient's risk in order to aid cost–benefit decisions. The standard method, using the area under the receiver operating characteristic curve, to measure the added value may not be sensitive enough to capture incremental improvements from the new markers. Recently, some novel alternatives to area under the receiver operating characteristic curve, such as integrated discrimination improvement and net reclassification improvement, were proposed. In this paper, we consider a class of measures for evaluating the incremental values of new markers, which includes the preceding two as special cases. We present a unified procedure for making inferences about measures in the class with censored event time data. The large sample properties of our procedures are theoretically justified. We illustrate the new proposal with data from a cancer study to evaluate a new gene score for prediction of the patient's survival. Copyright © 2012 John Wiley & Sons, Ltd.

Number of times cited according to CrossRef: 165

  • Biomarkers in patients with heart failure and central sleep apnoea: findings from the SERVE‐HF trial, ESC Heart Failure, 10.1002/ehf2.12521, 7, 2, (503-511), (2020).
  • Data on the power of high betatrophin to predict cardiovascular deaths in coronary patients, Data in Brief, 10.1016/j.dib.2019.104989, 28, (104989), (2020).
  • Validation of cardiac damage classification and addition of albumin in a large cohort of patients undergoing transcatheter aortic valve replacement, International Journal of Cardiology, 10.1016/j.ijcard.2020.01.031, (2020).
  • A Novel Risk Score for Type 2 Diabetes Containing Sleep Duration: A 7-Year Prospective Cohort Study among Chinese Participants, Journal of Diabetes Research, 10.1155/2020/2969105, 2020, (1-13), (2020).
  • Una puntuación de riesgo genético predice recurrencias en pacientes jóvenes con infarto agudo de miocardio, Revista Española de Cardiología, 10.1016/j.recesp.2019.08.006, (2020).
  • The Synergic Association of hs-CRP and Serum Amyloid P Component in Predicting All-Cause Mortality in Patients With Type 2 Diabetes, Diabetes Care, 10.2337/dc19-2489, 43, 5, (1025-1032), (2020).
  • A modified T categorization for part-solid lesions in Chinese patients with clinical stage I Non–small cell lung cancer, Lung Cancer, 10.1016/j.lungcan.2020.04.028, (2020).
  • Unsupervised Cluster Analysis of Patients With Aortic Stenosis Reveals Distinct Population With Different Phenotypes and Outcomes, Circulation: Cardiovascular Imaging, 10.1161/CIRCIMAGING.119.009707, 13, 5, (2020).
  • A nomogram for predicting overall survival in patients with low‐grade endometrial stromal sarcoma: A population‐based analysis, Cancer Communications, 10.1002/cac2.12067, 40, 7, (301-312), (2020).
  • Prognostic Value of Dynamic Changes in Pulmonary Congestion During Exercise Stress Echocardiography in Heart Failure With Preserved Ejection Fraction, Circulation: Heart Failure, 10.1161/CIRCHEARTFAILURE.119.006769, 13, 6, (2020).
  • An independent validation of the kidney failure risk equation in an Asian population, Scientific Reports, 10.1038/s41598-020-69715-3, 10, 1, (2020).
  • Safety and Prognostic Value of Vasodilator Stress Cardiovascular Magnetic Resonance in Patients With Heart Failure and Reduced Ejection Fraction, Circulation: Cardiovascular Imaging, 10.1161/CIRCIMAGING.120.010599, 13, 9, (2020).
  • Interpatient Similarities in Cardiac Function, JACC: Cardiovascular Imaging, 10.1016/j.jcmg.2019.12.018, (2020).
  • Circulating trimethyllysine and risk of acute myocardial infarction in patients with suspected stable coronary heart disease, Journal of Internal Medicine, 10.1111/joim.13067, 288, 4, (446-456), (2020).
  • Cardiorespiratory Fitness is an Independent Predictor of Cardiovascular Morbidity and Mortality, and Improves Accuracy of Prediction Models, Canadian Journal of Cardiology, 10.1016/j.cjca.2020.05.017, (2020).
  • Global Right Heart Assessment with Speckle-Tracking Imaging Improves the Risk Prediction of a Validated Scoring System in Pulmonary Arterial Hypertension, Journal of the American Society of Echocardiography, 10.1016/j.echo.2020.05.020, (2020).
  • Panel of novel urine biomarkers for incident microalbuminuria in people with type 2 diabetes mellitus, Diabetic Medicine, 10.1111/dme.14280, 37, 11, (1910-1918), (2020).
  • Cardiac biomarkers and risk of mortality in chronic kidney disease (the CRIC Study), Kidney International Reports, 10.1016/j.ekir.2020.08.028, (2020).
  • Prognostic impact of plasma volume estimated from hemoglobin and hematocrit in heart failure with preserved ejection fraction, Clinical Research in Cardiology, 10.1007/s00392-020-01639-4, (2020).
  • Prognostic value for long-term graft survival of estimated glomerular filtration rate and proteinuria quantified at 3 months after kidney transplantation, Clinical Kidney Journal, 10.1093/ckj/sfaa044, (2020).
  • Higher Activation of the Rostromedial Prefrontal Cortex during Mental Stress Predicts Major Cardiovascular Disease Events in Individuals with Coronary Artery Disease, Circulation, 10.1161/CIRCULATIONAHA.119.044442, (2020).
  • Preoperative computed tomography‐guided disease‐free survival prediction in gastric cancer: a multicenter radiomics study, Medical Physics, 10.1002/mp.14350, 0, 0, (2020).
  • Revised 15-item MDS-specific frailty scale maintains prognostic potential, Leukemia, 10.1038/s41375-020-01026-y, (2020).
  • A Novel Prognostic Scoring System Integrating Gene Expressions and Clinicopathological Characteristics to Predict Very Early Relapse in Node-Negative Estrogen Receptor-Positive/HER2-Negative Breast Cancer, Frontiers in Oncology, 10.3389/fonc.2020.01335, 10, (2020).
  • Prognostic nomogram for predicting survival in patients with high grade endometrial stromal sarcoma: a Surveillance Epidemiology, and End Results database analysis, International Journal of Gynecologic Cancer, 10.1136/ijgc-2020-001409, (ijgc-2020-001409), (2020).
  • Plasma Biomarkers of Tubular Injury and Inflammation Are Associated with Chronic Kidney Disease Progression in Children, Journal of the American Society of Nephrology, 10.1681/ASN.2019070723, (ASN.2019070723), (2020).
  • Plasma Lipidome and Prediction of Type 2 Diabetes in the Population-Based Malmö Diet and Cancer Cohort, Diabetes Care, 10.2337/dc19-1199, 43, 2, (366-373), (2019).
  • Validation of the hypertrophic cardiomyopathy risk-sudden cardiac death calculator in Asians, Heart, 10.1136/heartjnl-2019-315160, 105, 24, (1892-1897), (2019).
  • Body Composition Predicts Mortality and Decompensation in Compensated Cirrhosis Patients: A Prospective Cohort Study, JHEP Reports, 10.1016/j.jhepr.2019.11.005, (2019).
  • High betatrophin in coronary patients protects from cardiovascular events, Atherosclerosis, 10.1016/j.atherosclerosis.2019.11.011, (2019).
  • Prognostic value of visit-to-visit systolic blood pressure variability related to diabetic kidney disease among patients with type 2 diabetes, Journal of Hypertension, 10.1097/HJH.0000000000002038, 37, 7, (1411-1418), (2019).
  • Defining Subclinical Myocardial Dysfunction and Implications for Patients with Diabetes Mellitus and Preserved Ejection Fraction, The American Journal of Cardiology, 10.1016/j.amjcard.2019.06.011, (2019).
  • Soluble low-density lipoprotein receptor-related protein 1 as a biomarker of coronary risk: Predictive capacity and association with clinical events, Atherosclerosis, 10.1016/j.atherosclerosis.2019.06.904, (2019).
  • Left ventricular ejection fraction and adjudicated, cause-specific hospitalizations after myocardial infarction complicated by heart failure or left ventricular dysfunction, American Heart Journal, 10.1016/j.ahj.2019.06.004, (2019).
  • Superior Prognostic Value of Right Ventricular Free Wall Compared to Global Longitudinal Strain in Patients With Heart Failure, Journal of the American Society of Echocardiography, 10.1016/j.echo.2019.02.011, (2019).
  • Soluble PD-L1 is a potential biomarker of cutaneous melanoma aggressiveness and metastasis in obstructive sleep apnoea patients, European Respiratory Journal, 10.1183/13993003.01298-2018, 53, 2, (1801298), (2019).
  • Mid-term prognostic impact of residual pulmonary congestion assessed by radiographic scoring in patients admitted for worsening heart failure, International Journal of Cardiology, 10.1016/j.ijcard.2019.01.091, (2019).
  • A genetic risk score predicts recurrent events after myocardial infarction in young adults, Revista Española de Cardiología (English Edition), 10.1016/j.rec.2019.08.006, (2019).
  • The power‐integrated discriminant improvement: An accurate measure of the incremental predictive value of additional biomarkers, Statistics in Medicine, 10.1002/sim.8135, 38, 14, (2589-2604), (2019).
  • Validation of a Model for Identification of Patients With Compensated Cirrhosis at High Risk of Decompensation, Clinical Gastroenterology and Hepatology, 10.1016/j.cgh.2019.01.042, (2019).
  • I-Boost: an integrative boosting approach for predicting survival time with multiple genomics platforms, Genome Biology, 10.1186/s13059-019-1640-4, 20, 1, (2019).
  • Age- and Body Size-Adjusted Left Ventricular End-Diastolic Dimension in a Japanese Hospital-Based Population, Circulation Journal, 10.1253/circj.CJ-18-1095, (2019).
  • Hypertensive pregnancy disorders increase the risk of maternal cardiovascular disease after adjustment for cardiovascular risk factors, International Journal of Cardiology, 10.1016/j.ijcard.2019.01.097, (2019).
  • Prognostic value of serum lactate kinetics in critically ill patients with cirrhosis and acute-on-chronic liver failure: a multicenter study, Aging, 10.18632/aging.102062, 11, 13, (4446-4462), (2019).
  • Age-dependent relationship between preoperative serum aminotransferase and mortality after cardiovascular surgery, Aging, 10.18632/aging.102374, 11, 20, (9060-9074), (2019).
  • Plasma Albumin and Incident Cardiovascular Disease, Arteriosclerosis, Thrombosis, and Vascular Biology, 10.1161/ATVBAHA.119.313681, (2019).
  • An MDS-specific frailty index based on cumulative deficits adds independent prognostic information to clinical prognostic scoring, Leukemia, 10.1038/s41375-019-0666-7, (2019).
  • Body fat estimates from bioelectrical impedance equations in cardiovascular risk assessment: The PREVEND cohort study, European Journal of Preventive Cardiology, 10.1177/2047487319833283, (204748731983328), (2019).
  • High-sensitivity cardiac troponin T as an independent predictor of stroke in patients admitted to an emergency department with atrial fibrillation, PLOS ONE, 10.1371/journal.pone.0212278, 14, 2, (e0212278), (2019).
  • Prediction system for risk of allograft loss in patients receiving kidney transplants: international derivation and validation study, BMJ, 10.1136/bmj.l4923, (l4923), (2019).
  • High A20 expression negatively impacts survival in patients with breast cancer, PLOS ONE, 10.1371/journal.pone.0221721, 14, 8, (e0221721), (2019).
  • Predicting all-cause and lung cancer mortality using emphysema score progression rate between baseline and follow-up chest CT images: A comparison of risk model performances, PLOS ONE, 10.1371/journal.pone.0212756, 14, 2, (e0212756), (2019).
  • External validation of the systemic immune-inflammation index as a prognostic factor in metastatic renal cell carcinoma and its implementation within the international metastatic renal cell carcinoma database consortium model, International Journal of Clinical Oncology, 10.1007/s10147-018-01390-x, (2019).
  • Soluble Urokinase Plasminogen Activator Receptor and Decline in Kidney Function in Autosomal Dominant Polycystic Kidney Disease, Journal of the American Society of Nephrology, 10.1681/ASN.2018121227, (ASN.2018121227), (2019).
  • Sudomotor dysfunction independently predicts incident cardiovascular–renal events and all-cause death in type 2 diabetes: the Joint Asia Diabetes Evaluation register, Nephrology Dialysis Transplantation, 10.1093/ndt/gfy154, 34, 8, (1320-1328), (2018).
  • Assessing the added predictive ability of a metabolic syndrome severity score in predicting incident cardiovascular disease and type 2 diabetes: the Atherosclerosis Risk in Communities Study and Jackson Heart Study, Diabetology & Metabolic Syndrome, 10.1186/s13098-018-0344-3, 10, 1, (2018).
  • Predicting Mortality in African Americans With Type 2 Diabetes Mellitus: Soluble Urokinase Plasminogen Activator Receptor, Coronary Artery Calcium, and High‐Sensitivity C‐Reactive Protein, Journal of the American Heart Association, 10.1161/JAHA.117.008194, 7, 9, (2018).
  • Dynamic thresholds and a summary ROC curve: Assessing prognostic accuracy of longitudinal markers, Statistics in Medicine, 10.1002/sim.7675, 37, 18, (2700-2714), (2018).
  • The incremental value of a geriatric assessment-derived three-item scale on estimating overall survival in older adults with cancer, Journal of Geriatric Oncology, 10.1016/j.jgo.2018.01.007, 9, 4, (329-336), (2018).
  • Relation of High Serum Bilirubin to Short-Term Mortality Following a Myocardial Infarction Complicated by Left Ventricular Systolic Dysfunction (from the High-Risk Myocardial Infarction Database Initiative), The American Journal of Cardiology, 10.1016/j.amjcard.2018.01.012, 121, 9, (1015-1020), (2018).
  • Usefulness of Left Ventricular Vortex Flow Analysis for Predicting Clinical Outcomes in Patients with Chronic Heart Failure: A Quantitative Vorticity Imaging Study Using Contrast Echocardiography, Ultrasound in Medicine & Biology, 10.1016/j.ultrasmedbio.2018.05.015, 44, 9, (1951-1959), (2018).
  • Incorporating Neutrophil-to-lymphocyte Ratio and Platelet-to-lymphocyte Ratio in Place of Neutrophil Count and Platelet Count Improves Prognostic Accuracy of the International Metastatic Renal Cell Carcinoma Database Consortium Model, Cancer Research and Treatment, 10.4143/crt.2017.033, 50, 1, (103-110), (2018).
  • High-sensitivity cardiac troponin I and risk of incident atrial fibrillation hospitalisation in an Australian community-based cohort: The Busselton health study, Clinical Biochemistry, 10.1016/j.clinbiochem.2018.05.003, 58, (20-25), (2018).
  • Sample size determination for high dimensional parameter estimation with application to biomarker identification, Computational Statistics & Data Analysis, 10.1016/j.csda.2017.08.010, 118, (54-65), (2018).
  • Single-cell developmental classification of B cell precursor acute lymphoblastic leukemia at diagnosis reveals predictors of relapse, Nature Medicine, 10.1038/nm.4505, 24, 4, (474-483), (2018).
  • Comprehensive Analysis of the Neutrophil-to-Lymphocyte Ratio for Preoperative Prognostic Prediction Nomogram in Gastric Cancer, World Journal of Surgery, 10.1007/s00268-018-4510-4, 42, 8, (2530-2541), (2018).
  • Elevated serum uric acid concentration at discharge confers additive prognostic value in elderly patients with acute heart failure, Nutrition, Metabolism and Cardiovascular Diseases, 10.1016/j.numecd.2017.12.009, 28, 4, (361-368), (2018).
  • Impact of Nutritional and Inflammation Status on Long-Term Bleeding in Patients Undergoing Percutaneous Coronary Intervention with an Oral Anticoagulant, Journal of Atherosclerosis and Thrombosis, 10.5551/jat.47654, (2018).
  • Development of a Risk Score Based on Aortic Calcification to Predict 1-year Mortality After Transcatheter Aortic Valve Replacement, JACC: Cardiovascular Imaging, 10.1016/j.jcmg.2018.03.018, (2018).
  • Resting Heart Rate and Chronotropic Response to Exercise: Prognostic Implications in Heart Failure Across Left Ventricular Ejection Fraction Spectrum., Journal of Cardiac Failure, 10.1016/j.cardfail.2018.09.015, (2018).
  • Left Ventricular Entropy Is a Novel Predictor of Arrhythmic Events in Patients With Dilated Cardiomyopathy Receiving Defibrillators for Primary Prevention, JACC: Cardiovascular Imaging, 10.1016/j.jcmg.2018.07.003, (2018).
  • Sleep Quality and Sleep Duration with CKD are Associated with Progression to ESKD, Clinical Journal of the American Society of Nephrology, 10.2215/CJN.01340118, 13, 12, (1825-1832), (2018).
  • The ROC curve for regularly measured longitudinal biomarkers, Biostatistics, 10.1093/biostatistics/kxy010, (2018).
  • Admission macrophage migration inhibitory factor predicts long-term prognosis in patients with ST-elevation myocardial infarction, European Heart Journal - Quality of Care and Clinical Outcomes, 10.1093/ehjqcco/qcy020, (2018).
  • Risk Scoring System to Assess Outcomes in Patients Treated with Contemporary Guideline-Adherent Optimal Therapies after Acute Myocardial Infarction, Korean Circulation Journal, 10.4070/kcj.2017.0128, 48, 6, (492), (2018).
  • Immune-unreactive urinary albumin as a predictor of cardiovascular events: the Hortega Study, Nephrology Dialysis Transplantation, 10.1093/ndt/gfy087, (2018).
  • High-sensitivity cardiac troponin I and risk of cardiovascular disease in an Australianpopulation-based cohort, Heart, 10.1136/heartjnl-2017-312093, 104, 11, (895-903), (2017).
  • Hematological Parameters Outperform Plasma Markers in Predicting Long-Term Mortality After Coronary Angiography, Angiology, 10.1177/0003319717743679, 69, 7, (600-608), (2017).
  • Prognostic value of elevated high-sensitivity cardiac troponin T in patients admitted to an emergency department with atrial fibrillation, EP Europace, 10.1093/europace/eux063, 20, 4, (582-588), (2017).
  • Prognostic assessment of repeatedly measured time-dependent biomarkers, with application to dilated cardiomyopathy, Statistical Methods & Applications, 10.1007/s10260-017-0410-2, 27, 3, (545-557), (2017).
  • Heart Failure Stages Among Older Adults in the Community, Circulation, 10.1161/CIRCULATIONAHA.116.023361, 135, 3, (224-240), (2017).
  • Contemporary Assessment of Left Ventricular Diastolic Function in Older Adults, Circulation, 10.1161/CIRCULATIONAHA.116.024825, 135, 5, (426-439), (2017).
  • Prognostic Value of Cardiopulmonary Exercise Testing in Heart Failure With Reduced, Midrange, and Preserved Ejection Fraction, Journal of the American Heart Association, 10.1161/JAHA.117.006000, 6, 11, (2017).
  • Left Atrial Volume and Mortality in Patients With Aortic Stenosis, Journal of the American Heart Association, 10.1161/JAHA.117.006615, 6, 11, (2017).
  • Contemporary Risk Stratification After Myocardial Infarction in the Community: Performance of Scores and Incremental Value of Soluble Suppression of Tumorigenicity‐2, Journal of the American Heart Association, 10.1161/JAHA.117.005958, 6, 10, (2017).
  • GDF-15 (Growth Differentiation Factor 15) Is Associated With Lack of Ventricular Recovery and Mortality After Transcatheter Aortic Valve Replacement, Circulation: Cardiovascular Interventions, 10.1161/CIRCINTERVENTIONS.117.005594, 10, 12, (2017).
  • Abdominal lean muscle is associated with lower mortality among kidney waitlist candidates, Clinical Transplantation, 10.1111/ctr.12911, 31, 3, (2017).
  • Usefulness of the delta neutrophil index to predict 30-day mortality in patients with ST segment elevation myocardial infarction, Scientific Reports, 10.1038/s41598-017-15878-5, 7, 1, (2017).
  • Application of the Third International Consensus Definitions for Sepsis (Sepsis-3) Classification: a retrospective population-based cohort study, The Lancet Infectious Diseases, 10.1016/S1473-3099(17)30117-2, 17, 6, (661-670), (2017).
  • Poor long-term outcomes in stroke patients with asymptomatic coronary artery disease in heart CT, Atherosclerosis, 10.1016/j.atherosclerosis.2017.07.029, 265, (7-13), (2017).
  • Serum uromodulin is a predictive biomarker for cardiovascular events and overall mortality in coronary patients, International Journal of Cardiology, 10.1016/j.ijcard.2016.12.183, 231, (6-12), (2017).
  • Financial and Health Literacy Predict Incident Alzheimer’s Disease Dementia and Pathology, Journal of Alzheimer's Disease, 10.3233/JAD-161132, 56, 4, (1485-1493), (2017).
  • Data on the power of the creatinine to uromodulin ratio in serum to predict cardiovascular events in coronary patients, Data in Brief, 10.1016/j.dib.2017.03.003, 11, (576-580), (2017).
  • Graft and Patient Survival, Kidney Transplantation, Bioengineering and Regeneration, 10.1016/B978-0-12-801734-0.00039-4, (557-571), (2017).
  • Prognostic value of atherosclerotic burden and coronary vascular function in patients with suspected coronary artery disease, European Journal of Nuclear Medicine and Molecular Imaging, 10.1007/s00259-017-3800-7, 44, 13, (2290-2298), (2017).
  • Improvement Screening for Ultra-High Dimensional Data with Censored Survival Outcomes and Varying Coefficients, The International Journal of Biostatistics, 10.1515/ijb-2017-0024, 13, 1, (2017).
  • Urinary Sodium Concentration Is an Independent Predictor of All-Cause and Cardiovascular Mortality in a Type 2 Diabetes Cohort Population, Journal of Diabetes Research, 10.1155/2017/5327352, 2017, (1-10), (2017).
  • Serum Chloride and Sodium Interplay in Patients With Acute Myocardial Infarction and Heart Failure With Reduced Ejection Fraction, Circulation: Heart Failure, 10.1161/CIRCHEARTFAILURE.116.003500, 10, 2, (2017).
  • Pathway-Specific Aggregate Biomarker Risk Score Is Associated With Burden of Coronary Artery Disease and Predicts Near-Term Risk of Myocardial Infarction and Death, Circulation: Cardiovascular Quality and Outcomes, 10.1161/CIRCOUTCOMES.115.001493, 10, 3, (2017).
  • “Model for Early Allograft Function” Outperforms “Early Allograft Dysfunction” as a Predictor of Transplant Survival, Transplantation, 10.1097/TP.0000000000001833, 101, 8, (e258-e264), (2017).
  • See more

The full text of this article hosted at iucr.org is unavailable due to technical difficulties.