Community-Based Care for Minority Adolescents With ADHD: Improving Fidelity With Machine Learning-Assisted Supervision and Fidelity Feedback.

This project proposes to reduce disparities in care among disadvantaged racial/ethnic minority adolescents with ADHD by improving community therapist fidelity to evidence-based behavior therapy through a technology-assisted supervision intervention. In Y01, the research team will work with stakeholders to develop the proposed supervision intervention utilizing two novel technologies: Lyssn + Care4 (LC4S). In Y02, a preliminary clinical trial (N=72) will be conducted in three community mental health agencies in Miami, FL. Adolescent participants will be randomly assigned to receive supervision from a therapist who is trained in LCS4 or provides enhanced supervision as usual(ESAU)using a permuted block randomization strategy that randomizes within site. There will also be double randomization of agency therapists to supervisors. Supervisors will deliver both conditions and investigators will test for contamination to determine the integrity of this design prior to a future R01 that measures patient outcomes. Data from therapists, adolescents and their parents, and supervisors will be collected pre-training, post-training, weekly during service delivery, at EBT completion, and at the end of the trial. The proximal intervention target is therapist fidelity to EBT and the distal targets are service delivery outcomes that include quality, quantity, and speed of delivery. Investigators will also measure indices of consumer fit: cost, acceptability, feasibility, and fidelity to supervision procedures. Sources of data will be audio recorded therapy and supervision sessions, therapist and supervisor report, and project and electronic health records. In longitudinal analyses, time will be modeled as a person-specific variable representing months since baseline. Investigators will nest adolescents within therapists for all analyses.

Study Overview

• Status: Completed
• Conditions: Attention Deficit Hyperactivity Disorder
• Intervention / Treatment: Other: Artificial Intelligence-Assisted Supervision Protocol

Detailed Description

Y02 of this study is a small (N=72) phase 1/phase 2 clinical trial of a supervision intervention designed to improve therapist treatment fidelity and subsequent service delivery outcomes. The parallel design includes random assignment of eligible and consenting patients at three community agencies to two active supervision intervention arms (LC4S or ESAU) using a permuted block randomization strategy that accounts for agency. Participants will receive behavioral interventions from community agency staff and their service utilization will be tracked using project and agency electronic health records. Agency therapists and supervisors will also be participants in this trial. Therapists will be randomized to receive either LC4S or ESAU from their supervisor (double randomization) using a block randomization strategy that accounts for site. Supervisors will administer both supervision conditions in this trial; however, investigators will systematically assess for contamination to assess whether this design is appropriate for a future R01.

To minimize bias, adolescent and parent participants will not be informed of the group to which they have been assigned. However, full masking of therapists and supervisors is not feasible in this trial because both supervisors and therapists will know whether they are participating in technology-assisted supervision activities or standard supervision activities due to the nature of these conditions. However, the primary investigator will blind therapists and supervisors to our study hypotheses and the nature of outcome measures to minimize bias in the trial. Many of these measures will be observational and objective (i.e., therapy records, therapy audio recordings), which should reduce bias stemming from self-reports. Investigators will also assess whether there are group effects on therapist accuracy of self-report. All interventions will be delivered by agency staff, who will not be required to follow intervention delivery protocols because an outcome of this study is the extent to which agency therapists follow intervention procedures with guidance provided from their supervisors. Study assessments will be administered electronically via Care4 and data collection will be oversee by study staff.

Each measure of fidelity will be analyzed using a separate mixed /growth model (Duncan et al., 1999). In this design, treatment sessions (level 1) are nested within adolescents (level 2), which are nested within therapists (level 3). Each adolescent attends up to 10 sessions; each therapist treats 3 adolescents. Supervisors serve as an additional higher level, but with so few supervisors (approximately 6), this will be addressed by including dummy predictor variables representing the supervisors. The direct effect of LC4S vs. ESAU on fidelity intercept and slope will be tested for each fidelity outcome (see Table 3). With time centered at the first session, the intercept reflects initial fidelity for the ESAU condition, the group effect reflects the initial fidelity difference between ESAU and LC4S, the time effect reflects the linear change in fidelity over time for the ESAU condition, and the interaction of time and group reflects the difference between ESAU and LC4S in linear change in fidelity over time. As part of the R34, in will estimate the intraclass correlation (ICC) and design effect for the clustering effect of therapist and supervisor on outcome to determine the extent to which additional clustering will be needed in a future R01. ICC ranges from 0 to 1, with larger values reflecting a larger proportion of variance at the higher levels (here, therapist and supervisor rather than adolescent). Investigators will also test both linear and non-linear slopes to ascertain the expected shape of the LGCA in a future R01. Time to therapist MI competence, proportion of EBT delivered by 10th session, and number of sessions and days to completion of the EBT will be modeled using regression (linear, logistic, or Poisson depending on the distribution of the resulting variables) with group as a predictor. Accuracy of therapist self-report will be analyzed using polynomial regression (Laird & LaFleur, 2013).

Using the R package powerlmm, investigators have an estimate of .8 power to detect large effects (d = 0.8) representing group differences at the adolescent (level 2) level; investigators have .4 power to detect medium effects (d = .5) representing group differences at the adolescent level. Our hypotheses, however, are primarily at the level of the therapist (level 3), which has fewer units (i.e., 12 therapists per condition versus 36 adolescents per condition). As such, measures of both adolescent-level and therapist-level effects will be estimated. For power for the ICC calculation, investigators used the ICC.Sample.Size R package based on Zou (2012). With a sample size of 72 participants, 10 observations per participant, alpha equal to .05, and a two tailed test, investigators have greater than .9 power detect an ICC of 0.2. ICC values for cross-sectional data such as children within classrooms are approximately 0.2 (Hedges & Hedberg, 2007). Investigators expect excellent precision for estimating ICC values in this study.

• Study Type: Interventional
• Enrollment (Actual): 51
• Phase: Not Applicable

Contacts and Locations

Study Locations

• United States
  • Washington
    • Seattle, Washington, United States, 98145
      • Seattle Children's Research Institute

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 11 years to 17 years (Child)
• Accepts Healthy Volunteers: No

Description

Inclusion Criteria:

• DSM-5 ADHD diagnosis, Enrollment in 6th-12th grade, IQ greater than 70, no history of autism spectrum disorder or thought disorder, client of one of two community mental health agencies

Exclusion Criteria:

• Autism Spectrum/Thought Disorders, IQ<70

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Health Services Research
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: Single

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Experimental: Artificial Intelligence-Assisted Supervision Protocol; Measurement-based supervision protocol that incorporates fidelity measurement from a machine learning tool and feedback reports from this tool into a standardized supervision protocol for behavior therapy to task-shift burdensome supervision tasks to a machine, reducing costs and improving precision of fidelity measurement for agencies.	Other: Artificial Intelligence-Assisted Supervision Protocol; Measurement-based supervision protocol that incorporates fidelity measurement from a machine learning tool and feedback reports from this tool into a standardized supervision protocol for behavior therapy to task-shift burdensome supervision tasks to a machine, reducing costs and improving precision of fidelity measurement for agencies.
No Intervention: Enhanced Supervision as Usual (ESAU) Condition; ESAU therapists will be given standard, paper-based facilitation resources for STAND and will receive 4 hours of training on how to navigate these materials and self-assess fidelity. ESAU therapists will also be trained how to upload recordings into Care4 and complete self-assessments for each session. Supervisors will be given access to these data and recordings once uploaded (but not Lyssn scores or electronic facilitation resources).

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Therapist STAND Fidelity	Behavior therapy content: STAND treatment fidelity checklists (Sibley et al., 2013, 2016, 2019). This data will be collected via therapist self-report, Lyssn (the machine learning tool), and coded by trained research assistants from audio recordings. If there is a discrepancy in sources, a trained RA will code the tape to resolve the discrepancy.	Weekly for an average of 9 months
Therapist MITI Fidelity	Behavior therapy content:Motivational Interviewing Treatment Integrity (MITI) measure (Moyers et al., 2014) will measure MI integrity.This data will be collected via therapist self-report, Lyssn (the machine learning tool), and coded by trained research assistants from audio recordings. If there is a discrepancy in sources, a trained RA will code the tape to resolve the discrepancy.	Weekly for an average of 9 months

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Service Delivery Quality	Quality: (1) Time to Therapist MI Competence; MITI Benchmarks,(Moyers et al., 2014) Accuracy of therapist self-report on MITI indices and fidelity checklists (tested using Polynomial Regression)--will assess concordance between reporters over time. This data will be collected from electronic health records and by coding audio recordings of therapy sessions.	Weekly for duration that case is active in the agency, an average of 9 months
Service Delivery Quantity	Quantity: (2) Proportion of EBT delivered by 10th session Speed of Delivery: Number of sessions and days to completion of EBT. This data will be collected from electronic health records and by coding audio recordings of therapy sessions.	Weekly for duration that case is active in the agency, an average of 9 months

Other Outcome Measures

Outcome Measure	Measure Description	Time Frame
Cost	Cost of implementing/subscribing to Care4/Lyssn; Supervisor/ therapist time , technical supports, stakeholder-identified costs	cost will be computed at the end of the year from study records and electronic heath records, an average of 9 months
Technology Acceptability	Therapists and supervisors will complete an end of study questionnaire on the acceptability of the Lyssn Care4 Study protocol.	At end of the study, after an average of 9 months
Satisfaction with supervision	The Short Supervisory Relationship Questionnaire (Cliffe et al., 2006), which will be completed by therapists and supervisors.	At end of the study, after an average of 9 months
Supervision Fidelity	We will adapt a project specific version of the Supervisor Adherence and Competence Scale (SACS; Martino et al., 2016), which will be completed by trained research assistants who listen to audio recordings. In addition, we will train Lyssn to evaluate supervision sessions on this checklist by the end of the trial for potential use as supervisor feedback tool in a future R01.	Weekly, through study completion, an average of 9 months
Therapist engagement	Therapist Supervision Attendance: Supervisor logs of supervision dates/times; cancelations/reasons for cancelation; to be developed Weekly Facilitation Resources Use-measure; and completion rate for weekly Care4 support tasks (uploading session audio file and completing fidelity self-report form prior to supervision) forms are very brief questionnaires completed by therapists and supervisors prior to uploading each session in Care4 to measure therapist engagement with Care4 and Lyssn integrations	Weekly, through study completion, an average of 9 months
Supervisor Burden	Minutes discussing case on audio recording; supervisor log of prep work for case (not including research tasks). Tapes will be coded by RAs for length and supervision logs will be kept by the supervisors.	Weekly, through study completion, an average of 9 months
Exit Interviews	Semi-Structured exit interviews designed to collect qualitative perspectives on the LC4S intervention.	At end of the study, after an average of 9 months

Collaborators and Investigators

• Sponsor: Seattle Children's Hospital
• Collaborators: Florida International University
• Investigators: Principal Investigator: Margaret H Sibley, Ph.D, Seattle Children's Hospital

Publications and helpful links

General Publications

• Sibley MH, Bickman L, Atkins D, Coxe S, King J, Tanana M, Martin P, Page TF, Ortiz M, Tapia J, Sparber A, Zhao X. Improving Community-Based Care for Adolescents with ADHD: a Randomized Controlled Trial of Artificial Intelligence-Assisted Fidelity Supports. Prev Sci. 2025 Dec 24. doi: 10.1007/s11121-025-01868-x. Online ahead of print.

Study record dates

Study Major Dates

• Study Start (Actual): November 18, 2021
• Primary Completion (Actual): August 30, 2024
• Study Completion (Actual): December 1, 2024

Study Registration Dates

• First Submitted: September 24, 2021
• First Submitted That Met QC Criteria: November 13, 2021
• First Posted (Actual): November 26, 2021

Study Record Updates

• Last Update Posted (Actual): April 24, 2026
• Last Update Submitted That Met QC Criteria: April 21, 2026
• Last Verified: April 1, 2026

More Information

Terms related to this study

• Keywords: Adolescents; ADHD; Community Mental Health
• Additional Relevant MeSH Terms: Mental Disorders; Neurodevelopmental Disorders; Attention Deficit and Disruptive Behavior Disorders; Attention Deficit Disorder with Hyperactivity
• Other Study ID Numbers: R34MH125037 (U.S. NIH Grant/Contract)

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No