Validation of Computerized Adaptive Testing in an Outpatient Nonacademic Setting: The VOCATIONS Trial

Abstract

Objective: Computerized adaptive testing (CAT) provides an alternative to fixed-length assessments. The study validated a suite of computerized adaptive tests for mental health (CAT-MH) in a community psychiatric sample.

Methods: A total of 145 adults from a community outpatient clinic, including 19 with no history of a mental disorder (control group), were prospectively evaluated with CAT for depression (CAD-MDD and CAT-DI), mania (CAT-MANIA), and anxiety symptoms (CAT-ANX). Ratings were compared with gold-standard psychiatric assessments, including the Structured Clinical Interview for DSM-IV-TR (SCID), Hamilton Rating Scale for Depression (HAM-D-25), Patient Health Questionnaire (PHQ-9), Center for Epidemiologic Studies Depression Scale (CES-D), and Global Assessment of Functioning (GAF).

Results: Sensitivity and specificity for CAD-MDD were .96 and .64, respectively (.96 and 1.00 for major depression versus the control group). CAT for depression severity (CAT-DI) correlated well with the HAM-D-25 (r=.79), PHQ-9 (r=.90), and CES-D (r=.90) and had an odds ratio (OR) of 27.88 across its range for current SCID major depressive disorder. CAT-ANX correlated with the HAM-D-25 (r=.73), PHQ-9 (r=.78), and CES-D (r=.81) and had an OR of 11.52 across its range for current SCID generalized anxiety disorder. CAT-MANIA did not correlate well with the HAM-D-25 (r=.31), PHQ-9 (r=.37), and CES-D (r=.39), but it had an OR of 11.56 across its range for a current SCID bipolar diagnosis. Participants found the CAT-MH acceptable and easy to use, averaging 51.7 items and 9.4 minutes to complete the full battery.

Conclusions: Compared with gold-standard diagnostic and assessment measures, CAT-MH provided an effective, rapidly administered assessment of psychiatric symptoms.

Conflict of interest statement

DISCLOSURES

Dr. A has received research grant support from Pfizer, Janssen, Otsuka, AssurEx, Pine Rest Foundation, NIMH, NIDA, ARRA, NIAAA, CMMS, Dartmouth College, the General Hospital Corporation, North Shore Long Island Jewish Health System and served as a consultant for Publicis Healthcare Communications Group. He also serves on an advisory board for Roche.

Dr. B has received research grant support from the Pine Rest Foundation.

Ms. C has no potential conflicts of interest to disclose related to this study.

Ms. D has no potential conflicts of interest to disclose related to this study.

Dr. E has the following disclosures: she received royalties from the American Psychological Association and Guilford Press; member of the Advisory Board of Servier International; Editorial Consultant for the American Psychiatric Press; and has received honoraria from Lundbeck; and she and her spouse, Dr. F have financial interests in Adaptive Testing Technologies (www.adaptivetestingtechnologies.com), through which the CAT-MH tests will be made available. Dr. F is also a consultant to the American Psychiatric Association (as Chair of the DSM-5 Task Force); holds joint ownership of copyright for the Pittsburgh Sleep Quality Index (PSQI); and he is a stockholder in AliphCom.

Dr. F has the following disclosures: Consultant to the American Psychiatric Association (as Chair of the DSM-5 Task Force); joint ownership of copyright for the Pittsburgh Sleep Quality Index (PSQI); received honorarium for manuscript submission to Medicographia (Servier); member of the Valdoxan Advisory Board of Servier International; a stockholder in AliphCom; and he and his spouse, Dr. E are stockholders in Psychiatric Assessments, Inc. Dr. E also has the following disclosures: received royalties from the American Psychological Association and Guilford Press; member of the Valdoxan Advisory Board of Servier International; Editorial Consultant for the American Psychiatric Press; and has received honoraria from Lundbeck.

Dr. G has been an expert witness for the US Department of Justice, Wyeth, Merck, and Pfizer. He has financial interests in Adaptive Testing Technologies (www.adaptivetestingtechnologies.com), through which the CAT-MH tests will be made available.