Time-dependent ROC curve analysis in medical research: current methods and applications

Adina Najwa Kamarudin, Trevor Cox, Ruwanthi Kolamunnage-Dona, Adina Najwa Kamarudin, Trevor Cox, Ruwanthi Kolamunnage-Dona

Abstract

Background: ROC (receiver operating characteristic) curve analysis is well established for assessing how well a marker is capable of discriminating between individuals who experience disease onset and individuals who do not. The classical (standard) approach of ROC curve analysis considers event (disease) status and marker value for an individual as fixed over time, however in practice, both the disease status and marker value change over time. Individuals who are disease-free earlier may develop the disease later due to longer study follow-up, and also their marker value may change from baseline during follow-up. Thus, an ROC curve as a function of time is more appropriate. However, many researchers still use the standard ROC curve approach to determine the marker capability ignoring the time dependency of the disease status or the marker.

Methods: We comprehensively review currently proposed methodologies of time-dependent ROC curves which use single or longitudinal marker measurements, aiming to provide clarity in each methodology, identify software tools to carry out such analysis in practice and illustrate several applications of the methodology. We have also extended some methods to incorporate a longitudinal marker and illustrated the methodologies using a sequential dataset from the Mayo Clinic trial in primary biliary cirrhosis (PBC) of the liver.

Results: From our methodological review, we have identified 18 estimation methods of time-dependent ROC curve analyses for censored event times and three other methods can only deal with non-censored event times. Despite the considerable numbers of estimation methods, applications of the methodology in clinical studies are still lacking.

Conclusions: The value of time-dependent ROC curve methods has been re-established. We have illustrated the methods in practice using currently available software and made some recommendations for future research.

Keywords: Biomarker evaluation; Event-time; Longitudinal data; ROC curve; Software; Time-dependent AUC.

Figures

Fig. 1
Fig. 1
a Illustration for cases and controls of C/D, I/D and I/S (baseline) definitions. C/D: A, B and E are cases and C, D and F are controls; I/D: Only A is the case and C, D and F are controls; I/S: Only A is the case and D and F are controls. b Illustration for cases and controls of I/S (longitudinal) definitions. Only A is the case and D and F are the controls
Fig. 2
Fig. 2
Time-dependent ROC curves for 0, 1, 3, 5 years prior to death for the marker measured at visit time at ten years

References

    1. Heagerty PJ, Lumley T, Pepe MS. Time-dependent ROC curves for censored survival data and a diagnostic marker. Biometrics. 2000;56(2):337–344. doi: 10.1111/j.0006-341X.2000.00337.x.
    1. Hung H, Chiang CT. Estimation methods for time-dependent AUC models with survival data. Can J Stat Revue Can Stat. 2010;38(1):8–26.
    1. Song X, Zhou XH, Ma S. Nonparametric receiver operating characteristic-based evaluation for survival outcomes. Stat Med. 2012;31(23):2660–2675. doi: 10.1002/sim.5386.
    1. Chambless LE, Diao G. Estimation of time-dependent area under the ROC curve for long-term risk prediction. Stat Med. 2006;25(20):3474–3486. doi: 10.1002/sim.2299.
    1. Lambert J, Chevret S. Summary measure of discrimination in survival models based on cumulative/dynamic time-dependent ROC curves. Stat Methods In Med Res. 2014;25(5):2088–102.
    1. Cai T, Pepe MS, Lumley T, Zheng Y, Jenny NJ. The sensitivity and specificity of markers for event times. Biostatistics. 2006;7(2):182–197. doi: 10.1093/biostatistics/kxi047.
    1. Kalbfleisch JD, Prentice RL. The statistical analysis of failure time data, vol. 360. New Jersey: Wiley; 2011. .
    1. Pepe MS. The statistical evaluation of medical tests for classification and prediction. USA: Oxford University Press; 2003.
    1. Zheng Y, Heagerty PJ. Semiparametric estimation of time-dependent ROC curves for longitudinal marker data. Biostatistics. 2004;5(4):615–632. doi: 10.1093/biostatistics/kxh013.
    1. Bamber D. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psychol. 1975;12(4):387–415. doi: 10.1016/0022-2496(75)90001-2.
    1. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36. doi: 10.1148/radiology.143.1.7063747.
    1. Heagerty PJ, Zheng Y. Survival model predictive accuracy and ROC curves. Biometrics. 2005;61(1):92–105. doi: 10.1111/j.0006-341X.2005.030814.x.
    1. Blanche P, Dartigues JF, Jacqmin-Gadda H. Review and comparison of ROC curve estimators for a time-dependent outcome with marker-dependent censoring. Biom J. 2013;55(5):687–704. doi: 10.1002/bimj.201200045.
    1. Zheng Y, Heagerty PJ. Prospective accuracy for longitudinal markers. Biometrics. 2007;63(2):332–341. doi: 10.1111/j.1541-0420.2006.00726.x.
    1. Kaplan EL, Meier P. Nonparametric estimation from incomplete observations. J Am Stat Assoc. 1958;53(282):457–481. doi: 10.1080/01621459.1958.10501452.
    1. Akritas MG. Nearest neighbor estimation of a bivariate distribution under random censoring. Ann Stat. 1994;1299–1327.
    1. Cai T, Gerds TA, Zheng Y, Chen J. Robust Prediction of t‐Year Survival with Data from Multiple Studies. Biometrics. 2011;67(2):436–444. doi: 10.1111/j.1541-0420.2010.01462.x.
    1. Hung H, Chiang CT. Optimal Composite Markers for Time-Dependent Receiver Operating Characteristic Curves with Censored Survival Data. Scand J Stat. 2010;37(4):664–679. doi: 10.1111/j.1467-9469.2009.00683.x.
    1. Song X, Zhou XH. A semiparametric approach for the covariate specific ROC curve with survival outcome. Statistica Sinica. 2008;18(3):947-65.
    1. Viallon V, Latouche A. Discrimination measures for survival outcomes: connection between the AUC and the predictiveness curve. Biom J. 2011;53(2):217–236. doi: 10.1002/bimj.201000153.
    1. Uno H, Cai TX, Tian L, Wei LJ. Evaluating prediction rules for t-year survivors with censored regression models. J Am Stat Assoc. 2007;102(478):527–37.
    1. Royston P, Parmar MK. The use of restricted mean survival time to estimate the treatment effect in randomized clinical trials when the proportional hazards assumption is in doubt. Stat Med. 2011;30(19):2409–2421. doi: 10.1002/sim.4274.
    1. Cox DR. lRegression Models and Life Tables. mJ R Stat Soc Ser B. 1972;34(2):187–220.
    1. Aalen OO. A linear regression model for the analysis of life times. Stat Med. 1989;8(8):907–925. doi: 10.1002/sim.4780080803.
    1. Cai Z, Sun Y. Local Linear Estimation for Time‐Dependent Coefficients in Cox's Regression Models. Scand J Stat. 2003;30(1):93–111. doi: 10.1111/1467-9469.00320.
    1. Grambsch PM, Therneau TM. Proportional hazards tests and diagnostics based on weighted residuals. Biometrika. 1994;81(3):515–526. doi: 10.1093/biomet/81.3.515.
    1. Xu R, O'Quigley J. Proportional hazards estimate of the conditional survival function. J R Stat Soc Ser B (Stat Methodol) 2000;62(4):667–680. doi: 10.1111/1467-9868.00256.
    1. Saha-Chaudhuri P, Heagerty PJ. Non-parametric estimation of a time-dependent predictive accuracy curve. Biostatistics. 2013;14(1):42–59. doi: 10.1093/biostatistics/kxs021.
    1. Shen W, Ning J, Yuan Y. A direct method to evaluate the time‐dependent predictive accuracy for biomarkers. Biometrics. 2015;71(2):439–449. doi: 10.1111/biom.12293.
    1. Royston P, Altman DG. Regression Using Fractional Polynomials of Continuous Covariates - Parsimonious Parametric Modeling. Appl Stat-J Roy St C. 1994;43(3):429-67.
    1. Leisenring W, Pepe MS, Longton G. A marginal regression modelling framework for evaluating medical diagnostic tests. Stat Med. 1997;16(11):1263–1281. doi: 10.1002/(SICI)1097-0258(19970615)16:11<1263::AID-SIM550>;2-M.
    1. Etzioni R, Pepe M, Longton G, Hu C, Goodman G. Incorporating the time dimension in receiver operating characteristic curves: a case study of prostate cancer. Med Decis Mak. 1999;19(3):242–251. doi: 10.1177/0272989X9901900303.
    1. Tosteson ANA, Begg CB. A general regression methodology for ROC curve estimation. Med Decis Mak. 1988;8(3):204–215. doi: 10.1177/0272989X8800800309.
    1. Pepe MS. Three approaches to regression analysis of receiver operating characteristic curves for continuous test results. Biometrics. 1998; 54(1):124-35.
    1. Heagerty PJ, Pepe MS. Semiparametric estimation of regression quantiles with application to standardizing weight for height and age in US children. J R Stat Soc: Ser C: Appl Stat. 1999;48(4):533–551. doi: 10.1111/1467-9876.00170.
    1. Yang S-S. Linear combination of concomitants of order statistics with application to testing and estimation. Ann Inst Stat Math. 1981;33(1):463–470. doi: 10.1007/BF02480956.
    1. Zheng Y, Heagerty PJ. Partly conditional survival models for longitudinal data. Biometrics. 2005;61(2):379–391. doi: 10.1111/j.1541-0420.2005.00323.x.
    1. Heagerty PJ, Saha-Chaudhuri P, Saha-Chaudhuri MP. Package ‘survivalROC’. 2013.
    1. Potapov S, Adler W, Schmid M: survAUC: Estimators of Prediction Accuracy for Time-to-Event Data. R package version 1.0-5. In.; 2012.
    1. Blanche P. TimeROC: Time-dependent ROC curve and AUC for censored survival data. R package version 02, URL .
    1. Therneau TM, Lumley T. Package ‘survival’. In.: Verze; 2015
    1. Scheike T. Timereg Package. In.: R Package Version; 2009.
    1. Gerds TA, Rcpp I, Rcpp L, Gerds MTA. Package ‘prodlim’. 2015.
    1. Heagerty PJ, Saha-Chaudhuri P, Saha-Chaudhuri MP. Package ‘risksetROC’. 2012.
    1. Lu Y, Wang L, Liu P, Yang P, You M. Gene-expression signature predicts postoperative recurrence in stage I non-small cell lung cancer patients. PLoS One. 2012;7(1):e30880. doi: 10.1371/journal.pone.0030880.
    1. Tse LA, Dai JC, Chen MH, Liu YW, Zhang H, Wong TW, Leung CC, Kromhout H, Meijer E, Liu S et al. Prediction models and risk assessment for silicosis using a retrospective cohort study among workers exposed to silica in China. Scientific Reports. 2015;5.
    1. Yue Y, Cui X, Bose S, Audeh W, Zhang X, Fraass B. Stratifying triple-negative breast cancer prognosis using 18 F-FDG-PET/CT imaging. Breast Cancer Res Treat. 2015;153(3):607–616. doi: 10.1007/s10549-015-3558-1.
    1. Yue Y, Astvatsaturyan K, Cui X, Zhang X, Fraass B, Bose S. Stratification of Prognosis of Triple-Negative Breast Cancer Patients Using Combinatorial Biomarkers. PLoS One. 2016;11(3):e0149661. doi: 10.1371/journal.pone.0149661.
    1. Desmedt C, Giobbie-Hurder A, Neven P, Paridaens R, Christiaens M-R, Smeets A, Lallemand F, Haibe-Kains B, Viale G, Gelber RD. The Gene expression Grade Index: a potential predictor of relapse for endocrine-treated breast cancer patients in the BIG 1–98 trial. BMC Med Genet. 2009;2(1):1.
    1. Fleming TR, Harrington DP. Counting processes and survival analysis, vol. 169. New Jersey: Wiley; 2011. doi:10.1002/9781118150672.

Source: PubMed

스폰서 및 공동 작업자

건강 상태

약물 개입

3
구독하다