Machine learning determination of applied behavioral analysis treatment plan type

Abstract

Background: Applied behavioral analysis (ABA) is regarded as the gold standard treatment for autism spectrum disorder (ASD) and has the potential to improve outcomes for patients with ASD. It can be delivered at different intensities, which are classified as comprehensive or focused treatment approaches. Comprehensive ABA targets multiple developmental domains and involves 20-40 h/week of treatment. Focused ABA targets individual behaviors and typically involves 10-20 h/week of treatment. Determining the appropriate treatment intensity involves patient assessment by trained therapists, however, the final determination is highly subjective and lacks a standardized approach. In our study, we examined the ability of a machine learning (ML) prediction model to classify which treatment intensity would be most suited individually for patients with ASD who are undergoing ABA treatment.

Methods: Retrospective data from 359 patients diagnosed with ASD were analyzed and included in the training and testing of an ML model for predicting comprehensive or focused treatment for individuals undergoing ABA treatment. Data inputs included demographics, schooling, behavior, skills, and patient goals. A gradient-boosted tree ensemble method, XGBoost, was used to develop the prediction model, which was then compared against a standard of care comparator encompassing features specified by the Behavior Analyst Certification Board treatment guidelines. Prediction model performance was assessed via area under the receiver-operating characteristic curve (AUROC), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV).

Results: The prediction model achieved excellent performance for classifying patients in the comprehensive versus focused treatment groups (AUROC: 0.895; 95% CI 0.811-0.962) and outperformed the standard of care comparator (AUROC 0.767; 95% CI 0.629-0.891). The prediction model also achieved sensitivity of 0.789, specificity of 0.808, PPV of 0.6, and NPV of 0.913. Out of 71 patients whose data were employed to test the prediction model, only 14 misclassifications occurred. A majority of misclassifications (n = 10) indicated comprehensive ABA treatment for patients that had focused ABA treatment as the ground truth, therefore still providing a therapeutic benefit. The three most important features contributing to the model's predictions were bathing ability, age, and hours per week of past ABA treatment.

Conclusion: This research demonstrates that the ML prediction model performs well to classify appropriate ABA treatment plan intensity using readily available patient data. This may aid with standardizing the process for determining appropriate ABA treatments, which can facilitate initiation of the most appropriate treatment intensity for patients with ASD and improve resource allocation.

Keywords: Applied behavioral analysis; Artificial intelligence; Autism spectrum disorder; Machine learning.

Conflict of interest statement

All authors are employees or contractors of Montera Inc. dba Forta.

© 2023. The Author(s).

Figures

Fig. 1

Data processing and feature selection flowchart. The raw data from the ABA intake forms were processed and subjected to rigorous feature selection to generate the feature matrix used to train and test the machine learning (ML) prediction model. ABA applied behavioral analysis; SHAP SHapley Additive exPlanations; AUROC area under the receiver-operator characteristic curve

Fig. 2

Cross-validation area under the receiver-operator characteristic curve (AUROC) vs. number of features. AUROC area under the receiver operator characteristic curve

Fig. 3

Model training and evaluation workflow. The training dataset was used for feature selection and optimization of hyperparameters using a fivefold cross-validation method. The tuned model hyperparameters were then used to train the machine learning prediction model with the full training dataset. The trained model was then evaluated on the hold-out test dataset

Fig. 4

AUROCs demonstrating the superior performance of the machine learning prediction model by comparison with the standard of care comparator, and a random forest model. AUROC area under the receiver operator characteristic curve

Fig. 5

Confusion matrix providing a visual representation of the machine learning prediction model’s output for the hold-out test dataset. Out of the 71 patients in the hold-out test dataset, the ML prediction model successfully classified 57 patients as requiring comprehensive ABA treatment or focused ABA treatment. ABA applied behavioral analysis; TP true positive; FP false positive; TN true negative; FN false negative

Fig. 6

SHAP feature importance plot showing the 15 most important input features that contributed to the discriminative ability of the machine learning prediction model. SHAP SHapley Additive explanations; ABA applied behavioral analysis; RRB restrictive and repetitive behavior