Predictive modeling of treatment resistant depression using data from STAR*D and an independent clinical study

Abstract

Identification of risk factors of treatment resistance may be useful to guide treatment selection, avoid inefficient trial-and-error, and improve major depressive disorder (MDD) care. We extended the work in predictive modeling of treatment resistant depression (TRD) via partition of the data from the Sequenced Treatment Alternatives to Relieve Depression (STAR*D) cohort into a training and a testing dataset. We also included data from a small yet completely independent cohort RIS-INT-93 as an external test dataset. We used features from enrollment and level 1 treatment (up to week 2 response only) of STAR*D to explore the feature space comprehensively and applied machine learning methods to model TRD outcome at level 2. For TRD defined using QIDS-C16 remission criteria, multiple machine learning models were internally cross-validated in the STAR*D training dataset and externally validated in both the STAR*D testing dataset and RIS-INT-93 independent dataset with an area under the receiver operating characteristic curve (AUC) of 0.70-0.78 and 0.72-0.77, respectively. The upper bound for the AUC achievable with the full set of features could be as high as 0.78 in the STAR*D testing dataset. Model developed using top 30 features identified using feature selection technique (k-means clustering followed by χ2 test) achieved an AUC of 0.77 in the STAR*D testing dataset. In addition, the model developed using overlapping features between STAR*D and RIS-INT-93, achieved an AUC of > 0.70 in both the STAR*D testing and RIS-INT-93 datasets. Among all the features explored in STAR*D and RIS-INT-93 datasets, the most important feature was early or initial treatment response or symptom severity at week 2. These results indicate that prediction of TRD prior to undergoing a second round of antidepressant treatment could be feasible even in the absence of biomarker data.

Trial registration: ClinicalTrials.gov NCT00021528.

Conflict of interest statement

Drs. Vairavan, Narayan and Li are employees of Janssen Research & Development, LLC. Drs. Vairavan, Narayan and Li may be shareholders in Johnson & Johnson, which is the parent company of the Janssen companies. Drs. Vairavan, Narayan and Li declare that, except for income received from their primary employer, no financial support or compensation has been received from any individual or corporate entity over the past three years for research or professional service, and there are no personal financial holdings that could be perceived as constituting a potential conflict of interest. Dr. Ye reports past consultancy relationship with Janssen Research & Development, LLC. Mr. Nie reports no biomedical financial interests or potential conflicts of interest and can now be contacted at Samsung Electronics Co., Ltd. This study was funded by Janssen Research & Development, LLC, Titusville, NJ, USA, and by a research funding from Janssen Research & Development, LLC to University of Michigan for the collaboration with Dr. Ye’s laboratory. This manuscript reflects the views of the authors and may not reflect the opinions or views of the STAR*D Study Investigators or the funders. This does not alter our adherence to PLOS ONE policies on sharing data and materials.

Figures

Fig 1. Schematic diagram showing the predictive modeling process.

The STAR*D training dataset was used to create 30 subsamples with equal ratios of cases and controls and 30 models were constructed using the entire training dataset. The 30 models were used to predict the outcome for the independent STAR*D test dataset and RIS-INT-93 dataset and the predicted outcome were average across 30 models.

Fig 2. AUC for models containing top 2 to 75 representative predictors from k-means cluster (k = 75) was plotted against the number of predictors for each of the machine learning methods in the STAR*D training and test datasets, respectively.

Remission status was used to define TRD using QIDS-C16 data.

Fig 3. Receiver operating characteristic curves in training and test dataset (STAR*D) using the full set of features, top n features (n ~ 30), and the overlapping features where remission status was used to define TRD (STAR*D remission status was defined using QIDS-C16 data, and RIS-INT-93 remission status was defined using HAM-D17).

Fig 4. Permutation process to access the model robustness.

The outcome label of the STAR*D training dataset was randomly shuffled 1,000 times and the AUC distribution of the 1,000 null models were plotted for each machine learning machine method (A) XGBoost, (B) Random Forest, (C) l2 penalized logistic regression, and (D) GBDT. In all cases, the observed AUC out-perform the random noise from the 1,000 null models.

Fig 5

variable of importance in statistical learning approaches for outcomes defined by (A) remission status (B) responder status. In both cases, the outcomes were defined using QIDS-C16. SFHS: Short Form Health Survey (SF-12); WSAS: The Work and Social Adjustment Scale; *from PRISE: The Patient Rated Inventory of Side Effects, which collected symptoms one had experienced in the past week. Those symptoms may or may not have been caused by the treatment.