Volume 56, Issue 2

Time‐Dependent ROC Curves for Censored Survival Data and a Diagnostic Marker

Patrick J. Heagerty

Corresponding Author

Department of Biostatistics, University of Washington, Seattle, Washington 98195, U.S.A.

Division of Public Health Sciences, Fred Hutchinson Cancer Research Center, Seattle, Washington 98109, U.S.A.

*email:heagertyabiostat.washington.eduSearch for more papers by this author
Thomas Lumley

Department of Biostatistics, University of Washington, Seattle, Washington 98195, U.S.A.

Search for more papers by this author
Margaret S. Pepe

Division of Public Health Sciences, Fred Hutchinson Cancer Research Center, Seattle, Washington 98109, U.S.A.

Search for more papers by this author
First published: 24 May 2004
Citations: 1,167

Abstract

Summary. ROC curves are a popular method for displaying sensitivity and specificity of a continuous marker, X, for a binary disease variable, D. However, many disease outcomes are time dependent, D(t, and ROC curves that vary as a function of time may be mire appropriate. A common examples of a time‐dependent variable is vital status, where D(t) = 1 if a patient has died prior to time t and zero otherwise. We propose summarizing the discrimination potential of a marker X, measured at baseline (t= 0), by calculating ROC Curves for cumulative disease or death incidence by time t, which we denote as ROC(t). A typical complexity with survival data is that observations may be censored. Two ROC curve estimators are proposed that can accommodate censored data. A simple estimator is based on using the Kaplan‐Meier estimated for each possible subset X > c. However, this estimator does not guarantee the necessary condition that sensitivity and specificity are monotone in X. An alternative estimator that does guarantee monotonicity is based on a nearest neighbor estimator for the bivariate distribution function of (X, T), where T represents survival time (Akritas, M. J., 1994, Annals of Statistics22, 1299–1327). We present an example where ROC(t) is used to compare a standard and a modified flow cytometry measurement for predicting survival after detection of breast cancer and an example where the ROC(t) curve displays the impact of modifying eligibility criteria for sample size and power in HIV prevention trials.

Number of times cited according to CrossRef: 1167

  • Circulating tumor DNA methylation profiles enable early diagnosis, prognosis prediction, and screening for colorectal cancer, Science Translational Medicine, 10.1126/scitranslmed.aax7533, 12, 524, (eaax7533), (2020).
  • Complete blood count-based inflammatory score (CBCS) is a novel prognostic marker for gastric cancer patients after curative resection, BMC Cancer, 10.1186/s12885-019-6466-7, 20, 1, (2020).
  • A 5-Gene Signature Is Closely Related to Tumor Immune Microenvironment and Predicts the Prognosis of Patients with Non-Small Cell Lung Cancer, BioMed Research International, 10.1155/2020/2147397, 2020, (1-9), (2020).
  • Prognostic Model for Survival and Recurrence in Patients with Early-Stage Cervical Cancer: A Korean Gynecologic Oncology Group Study (KGOG 1028), Cancer Research and Treatment, 10.4143/crt.2019.124, 52, 1, (320-333), (2020).
  • Unique metastasis‐associated lncRNA signature optimizes prediction of tumor relapse in lung adenocarcinoma, Thoracic Cancer, 10.1111/1759-7714.13325, 11, 3, (728-737), (2020).
  • Construction of a novel gene-based model for prognosis prediction of clear cell renal cell carcinoma, Cancer Cell International, 10.1186/s12935-020-1113-6, 20, 1, (2020).
  • Individual-patient prediction of meningioma malignancy and survival using the Surveillance, Epidemiology, and End Results database, npj Digital Medicine, 10.1038/s41746-020-0219-5, 3, 1, (2020).
  • Long-term outcomes and significance of preoperative lymphocyte-to-monocyte ratio as a prognostic indicator in patients with invasive pancreatic neoplasms after repeat pancreatectomy, BMC Cancer, 10.1186/s12885-020-6602-4, 20, 1, (2020).
  • Aggressive lymphoma subtype is a risk factor for venous thrombosis. Development of lymphoma ‐ specific venous thrombosis prediction models, American Journal of Hematology, 10.1002/ajh.25837, 95, 8, (918-926), (2020).
  • A numerical strategy to evaluate performance of predictive scores via a copula‐based approach, Statistics in Medicine, 10.1002/sim.8566, 39, 20, (2671-2684), (2020).
  • Diagnostic and prognostic nomograms for bone metastasis in hepatocellular carcinoma, BMC Cancer, 10.1186/s12885-020-06995-y, 20, 1, (2020).
  • A bioinformatics investigation into the pharmacological mechanisms of javanica oil emulsion injection in non-small cell lung cancer based on network pharmacology methodologies, BMC Complementary Medicine and Therapies, 10.1186/s12906-020-02939-y, 20, 1, (2020).
  • Comprehensive network analysis of the molecular mechanisms associated with sorafenib resistance in hepatocellular carcinoma, Cancer Genetics, 10.1016/j.cancergen.2020.04.076, 245, (27-34), (2020).
  • Integrating transcriptomes and somatic mutations to identify RNA methylation regulators as a prognostic marker in hepatocellular carcinoma, Hepatobiliary & Pancreatic Diseases International, 10.1016/j.hbpd.2020.05.002, (2020).
  • A Novel Immune-Related Prognostic Signature for Thyroid Carcinoma, Technology in Cancer Research & Treatment, 10.1177/1533033820935860, 19, (153303382093586), (2020).
  • Development of an immune gene prognostic classifier for survival prediction and respond to immunocheckpoint inhibitor therapy/chemotherapy in endometrial cancer, International Immunopharmacology, 10.1016/j.intimp.2020.106735, 86, (106735), (2020).
  • The prognostic value of six survival-related genes in bladder cancer, Cell Death Discovery, 10.1038/s41420-020-00295-x, 6, 1, (2020).
  • Cell cycle progression score as a predictive biomarker for overall survival in patients with adrenocortical carcinoma, Clinical and Translational Medicine, 10.1002/ctm2.138, 10, 3, (2020).
  • Multiomic Analysis of Methylation and Transcriptome Reveals a Novel Signature in Esophageal Cancer, Dose-Response, 10.1177/1559325820942075, 18, 3, (155932582094207), (2020).
  • Establishment of the prognostic index of lung squamous cell carcinoma based on immunogenomic landscape analysis, Cancer Cell International, 10.1186/s12935-020-01429-y, 20, 1, (2020).
  • Dynamic risk assessment based on positron emission tomography scanning in diffuse large B-cell lymphoma: Post-hoc analysis from the PETAL trial, European Journal of Cancer, 10.1016/j.ejca.2019.09.027, 124, (25-36), (2020).
  • Development and validation of a RNA binding protein-associated prognostic model for lung adenocarcinoma, Aging, 10.18632/aging.102828, 12, 4, (3558-3573), (2020).
  • Development of a four-gene prognostic model for pancreatic cancer based on transcriptome dysregulation, Aging, 10.18632/aging.102844, 12, 4, (3747-3770), (2020).
  • Radiomics-based prediction of survival in patients with head and neck squamous cell carcinoma based on pre- and post-treatment 18F-PET/CT, Aging, 10.18632/aging.103508, 12, 14, (14593-14619), (2020).
  • Development of an Immune-Related Risk Signature in Patients with Bladder Urothelial Carcinoma, BioMed Research International, 10.1155/2020/5848493, 2020, (1-13), (2020).
  • An immune landscape based prognostic signature predicts the immune status and immunotherapeutic responses of patients with colorectal cancer, Life Sciences, 10.1016/j.lfs.2020.118368, (118368), (2020).
  • Impact of smoking amount on clinicopathological features and survival in non-small cell lung cancer, BMC Cancer, 10.1186/s12885-020-07358-3, 20, 1, (2020).
  • A novel RNA sequencing-based prognostic nomogram to predict survival for patients with cutaneous melanoma, Medicine, 10.1097/MD.0000000000018868, 99, 3, (e18868), (2020).
  • Hepatocellular carcinoma prediction beyond year 5 of oral therapy in a large cohort of Caucasian patients with chronic hepatitis B, Journal of Hepatology, 10.1016/j.jhep.2020.01.007, (2020).
  • Profiles of prognostic alternative splicing signature in hepatocellular carcinoma, Cancer Medicine, 10.1002/cam4.2875, 9, 6, (2171-2180), (2020).
  • Environmental pollutants damage airway epithelial cell cilia: Implications for the prevention of obstructive lung diseases, Thoracic Cancer, 10.1111/1759-7714.13323, 11, 3, (505-510), (2020).
  • Risk factors for posttreatment recurrence in patients with intermediate-risk papillary thyroid carcinoma, The American Journal of Surgery, 10.1016/j.amjsurg.2020.01.049, (2020).
  • Five gene signatures were identified in the prediction of overall survival in resectable pancreatic cancer, BMC Surgery, 10.1186/s12893-020-00856-y, 20, 1, (2020).
  • Validation and Recalibration of Charlson and Elixhauser Comorbidity Indices to Predict In-hospital Mortality in Hospitalized Patients in a Japanese Hospital-Based Administrative Database, Japanese Journal of Pharmacoepidemiology/Yakuzai ekigaku, 10.3820/jjpe.25.e1, (2020).
  • Plasma Ceramides and Sphingomyelins in Relation to Atrial Fibrillation Risk: The Cardiovascular Health Study, Journal of the American Heart Association, 10.1161/JAHA.119.012853, 9, 4, (2020).
  • Prognostic value of tumor size and minimal extrathyroidal extension in papillary thyroid carcinoma, The American Journal of Surgery, 10.1016/j.amjsurg.2020.02.020, (2020).
  • Baseline and interim PET‐based outcome prediction in peripheral T‐cell lymphoma: A subgroup analysis of the PETAL trial, Hematological Oncology, 10.1002/hon.2697, 38, 3, (244-256), (2020).
  • Analysis of genomic and transcriptomic variations as prognostic signature for lung adenocarcinoma, BMC Bioinformatics, 10.1186/s12859-020-03691-3, 21, S14, (2020).
  • What's the score? A comparison of deceased donor kidney scoring systems and correlation with graft outcome, Clinical Transplantation, 10.1111/ctr.13802, 34, 3, (2020).
  • Transcriptome-wide analysis and modelling of prognostic alternative splicing signatures in invasive breast cancer: a prospective clinical study, Scientific Reports, 10.1038/s41598-020-73700-1, 10, 1, (2020).
  • Positive lymph node number and extranodal extension for predicting recurrence and survival in patients with salivary gland cancer, Head & Neck, 10.1002/hed.26135, 42, 8, (1994-2001), (2020).
  • Identification and prognostic value of metabolism-related genes in gastric cancer, Aging, 10.18632/aging.103838, 12, 17, (17647-17661), (2020).
  • Comprehensive analysis of a 14 immune-related gene pair signature to predict the prognosis and immune features of gastric cancer, International Immunopharmacology, 10.1016/j.intimp.2020.107074, 89, (107074), (2020).
  • Serum Albumin at Partial Remission Predicts Outcomes in Membranous Nephropathy, Kidney International Reports, 10.1016/j.ekir.2020.02.1030, (2020).
  • Visually assessed coronary and cardiac calcium outperforms perfusion data during scintigraphy in the prediction of adverse outcomes, International Journal of Cardiology, 10.1016/j.ijcard.2020.03.032, (2020).
  • Dynamic prediction of time to a clinical event with sparse and irregularly measured longitudinal biomarkers, Biometrical Journal, 10.1002/bimj.201900112, 62, 6, (1371-1393), (2020).
  • Environmental Controls on Tropical Sea Breeze Convection and Resulting Aerosol Redistribution, Journal of Geophysical Research: Atmospheres, 10.1029/2019JD031699, 125, 6, (2020).
  • Analyzing Wildland Fire Smoke Emissions Data Using Compositional Data Techniques, Journal of Geophysical Research: Atmospheres, 10.1029/2019JD032128, 125, 6, (2020).
  • The Association of Preoperative Characteristics with Reintervention Risk in Patients Undergoing Revascularization for Chronic Limb-Threatening Ischemia (Iida. Reintervention Risk in CLTI), Journal of Atherosclerosis and Thrombosis, 10.5551/jat.54866, (2020).
  • Panel of novel urine biomarkers for incident microalbuminuria in people with type 2 diabetes mellitus, Diabetic Medicine, 10.1111/dme.14280, 37, 11, (1910-1918), (2020).
  • Association of Long Noncoding RNA Biomarkers With Clinical Immune Subtype and Prediction of Immunotherapy Response in Patients With Cancer, JAMA Network Open, 10.1001/jamanetworkopen.2020.2149, 3, 4, (e202149), (2020).
  • Predictive accuracy of markers or risk scores for interval censored survival data, Statistics in Medicine, 10.1002/sim.8547, 39, 18, (2437-2446), (2020).
  • Comprehensive insights on pivotal prognostic signature involved in clear cell renal cell carcinoma microenvironment using the ESTIMATE algorithm, Cancer Medicine, 10.1002/cam4.2983, 9, 12, (4310-4323), (2020).
  • Independent Validation of Early-Stage Non-Small Cell Lung Cancer Prognostic Scores Incorporating Epigenetic and Transcriptional Biomarkers With Gene-Gene Interactions and Main Effects, Chest, 10.1016/j.chest.2020.01.048, (2020).
  • Prognostic and predictive value of an immune infiltration signature in diffuse lower-grade gliomas, JCI Insight, 10.1172/jci.insight.133811, 5, 8, (2020).
  • The area between ROC curves, a non‐parametric method to evaluate a biomarker for patient treatment selection, Biometrical Journal, 10.1002/bimj.201900171, 62, 6, (1476-1493), (2020).
  • Prognostic and Predictive Value of Blood Tumor Mutational Burden in Patients With Lung Cancer Treated With Docetaxel, Journal of the National Comprehensive Cancer Network, 10.6004/jnccn.2019.7383, 18, 5, (582-589), (2020).
  • Multi-Objective Optimization for Personalized Prediction of Venous Thromboembolism in Ovarian Cancer Patients, IEEE Journal of Biomedical and Health Informatics, 10.1109/JBHI.2019.2943499, 24, 5, (1500-1508), (2020).
  • A five‐microRNA signature for individualized prognosis evaluation and radiotherapy guidance in patients with diffuse lower‐grade glioma, Journal of Cellular and Molecular Medicine, 10.1111/jcmm.15377, 24, 13, (7504-7514), (2020).
  • Elevated serum interleukin-8 is associated with enhanced intratumor neutrophils and reduced clinical benefit of immune-checkpoint inhibitors, Nature Medicine, 10.1038/s41591-020-0856-x, 26, 5, (688-692), (2020).
  • Prognostic role of alternative splicing events in head and neck squamous cell carcinoma, Cancer Cell International, 10.1186/s12935-020-01249-0, 20, 1, (2020).
  • Incidence, Treatment, and Survival in Primary Central Nervous System Neuroblastoma, World Neurosurgery, 10.1016/j.wneu.2020.04.145, (2020).
  • A four‐long noncoding RNA signature predicts survival of hepatocellular carcinoma patients, Journal of Clinical Laboratory Analysis, 10.1002/jcla.23377, 34, 9, (2020).
  • Incidence and Factors Associated With HBV Relapse After Cessation of Entecavir or Tenofovir in Patients With HBsAg Below 100 IU/mL, Clinical Gastroenterology and Hepatology, 10.1016/j.cgh.2020.04.037, (2020).
  • A robust twelve-gene signature for prognosis prediction of hepatocellular carcinoma, Cancer Cell International, 10.1186/s12935-020-01294-9, 20, 1, (2020).
  • Construction of a metastasis-associated ceRNA network reveals a prognostic signature in lung cancer, Cancer Cell International, 10.1186/s12935-020-01295-8, 20, 1, (2020).
  • Combining texture features of whole slide images improves prognostic prediction of recurrence-free survival for cutaneous melanoma patients, World Journal of Surgical Oncology, 10.1186/s12957-020-01909-5, 18, 1, (2020).
  • Development and validation of a novel 15‐CpG‐based signature for predicting prognosis in triple‐negative breast cancer, Journal of Cellular and Molecular Medicine, 10.1111/jcmm.15588, 24, 16, (9378-9387), (2020).
  • Development and validation of a hypoxia-related gene signature to predict overall survival in early-stage lung adenocarcinoma patients, Therapeutic Advances in Medical Oncology, 10.1177/1758835920937904, 12, (175883592093790), (2020).
  • Characteristics and 2‐year outcomes of dabigatran treatment in patients with heart failure and atrial fibrillation: GLORIA‐AF, ESC Heart Failure, 10.1002/ehf2.12857, 7, 5, (2679-2689), (2020).
  • Group testing in mediation analysis, Statistics in Medicine, 10.1002/sim.8546, 39, 18, (2423-2436), (2020).
  • Caleta el Cobre 022 Martian meteorite: Increasing nakhlite diversity, Meteoritics & Planetary Science, 10.1111/maps.13534, 55, 7, (1539-1563), (2020).
  • Learning genetic and environmental graphical models from family data, Statistics in Medicine, 10.1002/sim.8545, 39, 18, (2403-2422), (2020).
  • Sample‐weighted semiparametric estimation of cause‐specific cumulative risk and incidence using left‐ or interval‐censored data from electronic health records, Statistics in Medicine, 10.1002/sim.8544, 39, 18, (2387-2402), (2020).
  • Seamless phase 2/3 oncology trial design with flexible sample size determination, Statistics in Medicine, 10.1002/sim.8543, 39, 18, (2373-2386), (2020).
  • A deep learning MR-based radiomic nomogram may predict survival for nasopharyngeal carcinoma patients with stage T3N1M0, Radiotherapy and Oncology, 10.1016/j.radonc.2020.06.050, 151, (1-9), (2020).
  • Quasi-linear Cox proportional hazards model with cross- L1 penalty, BMC Medical Research Methodology, 10.1186/s12874-020-01063-2, 20, 1, (2020).
  • Prognostic alternative mRNA splicing signatures and associated splicing factors in acute myeloid leukemia, Neoplasia, 10.1016/j.neo.2020.06.004, 22, 9, (447-457), (2020).
  • Smoothed time‐dependent receiver operating characteristic curve for right censored survival data, Statistics in Medicine, 10.1002/sim.8671, 39, 24, (3373-3396), (2020).
  • Spatial domain analysis predicts risk of colorectal cancer recurrence and infers associated tumor microenvironment networks, Nature Communications, 10.1038/s41467-020-17083-x, 11, 1, (2020).
  • Pancreatic Ductal Adenocarcinoma: Interface Enhancement Gradient Measured on Dual-Energy CT Images Improves Prognostic Evaluation, Radiology: Imaging Cancer, 10.1148/rycan.2020190074, 2, 4, (e190074), (2020).
  • Risk Stratification for 2-Year Mortality in Patients with Chronic Limb-Threatening Ischemia Undergoing Endovascular Therapy, Journal of Atherosclerosis and Thrombosis, 10.5551/jat.57711, (2020).
  • Risk scores for occult cancer in patients with unprovoked venous thromboembolism: Results from an individual patient data meta‐analysis, Journal of Thrombosis and Haemostasis, 10.1111/jth.15001, 18, 10, (2622-2628), (2020).
  • Response to The challenges of diagnosing heparin‐induced thrombocytopenia in patients with COVID‐19, Research and Practice in Thrombosis and Haemostasis, 10.1002/rth2.12417, 4, 6, (1068-1069), (2020).
  • Old recipes, new strategies: Paleoenvironment, georesources, building materials, and trade networks in Roman Tuscany (Italy), Geoarchaeology, 10.1002/gea.21792, 35, 5, (678-700), (2020).
  • Genome-wide Modeling of Polygenic Risk Score in Colorectal Cancer Risk, The American Journal of Human Genetics, 10.1016/j.ajhg.2020.07.006, (2020).
  • Identification and validation of an immune prognostic signature in colorectal cancer, International Immunopharmacology, 10.1016/j.intimp.2020.106868, 88, (106868), (2020).
  • Identification of an immune-related gene-based signature to predict prognosis of patients with gastric cancer, World Journal of Gastrointestinal Oncology, 10.4251/wjgo.v12.i8.857, 12, 8, (857-876), (2020).
  • Identification of an immune-related risk signature for predicting prognosis in clear cell renal cell carcinoma, Aging, 10.18632/aging.102746, 12, 3, (2302-2332), (2020).
  • Development of an immune-related prognostic index associated with hepatocellular carcinoma, Aging, 10.18632/aging.102926, 12, 6, (5010-5030), (2020).
  • Development of a prognostic index and screening of potential biomarkers based on immunogenomic landscape analysis of colorectal cancer, Aging, 10.18632/aging.102979, 12, 7, (5832-5857), (2020).
  • Genome-wide analyses of the prognosis-related mRNA alternative splicing landscape and novel splicing factors based on large-scale low grade glioma cohort, Aging, 10.18632/aging.103491, 12, 13, (13684-13700), (2020).
  • Predictive Medicine in Multiple Sclerosis: a Systematic Review, Multiple Sclerosis and Related Disorders, 10.1016/j.msard.2020.101928, (101928), (2020).
  • Elevated baseline serum PD-1 or PD-L1 predicts poor outcome of PD-1 inhibition therapy in metastatic melanoma, Annals of Oncology, 10.1016/j.annonc.2019.09.005, 31, 1, (144-152), (2020).
  • Risk for venous thromboembolic events in patients with advanced urinary tract cancer treated with 1st-line chemotherapy, Clinical Genitourinary Cancer, 10.1016/j.clgc.2019.12.021, (2020).
  • An effective seven-CpG-based signature to predict survival in renal clear cell carcinoma by integrating DNA methylation and gene expression, Life Sciences, 10.1016/j.lfs.2020.117289, 243, (117289), (2020).
  • Definition of high-risk prostate cancer impacts oncological outcomes after radical prostatectomy, Urologic Oncology: Seminars and Original Investigations, 10.1016/j.urolonc.2019.12.014, (2020).
  • ELMO: An Efficient Logistic Regression-Based Multi-Omic Integrated Analysis Method for Breast Cancer Intrinsic Subtypes, IEEE Access, 10.1109/ACCESS.2019.2960373, 8, (5121-5130), (2020).
  • Identification of an Autophagy-Related Signature Predicting Overall Survival for Papillary Thyroid Carcinoma, Dose-Response, 10.1177/1559325819899265, 18, 1, (155932581989926), (2020).
  • Spatio‐temporal dynamics of technical efficiency in China’s specialized markets: A stochastic frontier analysis approach, Growth and Change, 10.1111/grow.12399, 51, 3, (1182-1202), (2020).
  • See more

The full text of this article hosted at iucr.org is unavailable due to technical difficulties.