ClientBot: a Conversational Agent That Supports Skills Practice and Feedback for Motivational Interviewing

ClientBot: a Conversational Agent That Supports Skills Practice and Feedback for Motivational Interviewing for Alcohol Use Disorder (AUD)

This multi-stage study includes qualitative interviews, usability research, and a randomized training study of technology implementation in a real-world clinical setting. The proposed study will enhance and evaluate an artificial intelligence (AI) based, conversational agent (ClientBot) that simulates a realistic client with alcohol concerns and provides performance-based feedback to support counselor training. The research is in collaboration with PRI, which focuses on training alcohol and substance use counselors.

Study Overview

• Status: Completed
• Conditions: Alcohol Use Disorder; Motivational Interviewing
• Intervention / Treatment: Other: ClientBot

Detailed Description

Stage 1. Interviews with 5 PRI trainers will assess current PRI training goals, content, and format, but also how training practices are conducted, including what types of technologies are currently used for what purposes. Trainer participants will then be introduced to the prototype ClientBot platform, including an opportunity to use it, and will be asked to describe how they might integrate the prototype into their existing training practices. Trainers will roleplay with researchers to illustrate work processes, and data collection will include interview audio recordings and field notes. Trainer participants will also be interviewed regarding their use of MI skills and which are most relevant and/or in need of training for their typical trainees.

Stage 1 will also include trainees who have participated in a PRI training within the previous 3 months. Recent trainees will provide an informed perspective on the ClientBot platform relative to current PRI skills training practices, which include an MI skills focused workbook and role plays. Participant sessions will be conducted online, using an interview-based usability format. Research staff will provide a brief introduction to ClientBot as a computer-based skills training platform and then ask participants to complete specific tasks, using a 'think aloud' protocol (i.e., provide a self narration as they pursue each task). The goal of the lab-based usability analysis is to verify that the software functions as intended in a controlled setting. As part of the hour-long session, participants will have an opportunity to use 2-3 MI skills training modules.

Stage 2. PRI has a robust schedule of in-person and online trainings throughout each year. For stage 2 usability study, we will recruit 20 counselors who have participated in a PRI training within the previous 3 months. Following software engineering work to implement the Phase I ClientBot tool into Lyssn's HIPAA-compliant cloud-based platform, we will conduct a single-arm, 2-week field trial among 20 recent PRI trainees to better assess user workflow, system interaction, and general usability in a community AUD setting. Participants will be encouraged to use ClientBot for 10 minutes per work day. Participants will be contacted daily via email by research staff to inquire about technical reliability (e.g., difficulties operating the tool), functional reliability (e.g., experience using ClientBot functions), and experiences integrating the system into the day's workflow. ClientBot use will be captured automatically by the system, including: which modules were used, the number of times resources were accessed and repeated, time spent viewing introductory materials, fidelity feedback provided during skills practice, and time spent in review of NLP-generated feedback.

Following each practice session, participants will answer a brief set of likert questions on 0-10 scale, including: How coherent was the client's narrative and responses? How realistic was this simulated client relative to client's you have seen? How much of the client session focused on alcohol use ? How accurate was the feedback on MI skill usage? After completing all practice sessions, participants will complete the system usability scale (SUS). To gauge implementation readiness, we will also use the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM). Each are brief, 5-item measures targeting core implementation outcomes with strong psychometric properties. Trainees will evaluate the perceived accuracy of the ClientBot feedback, which will also be evaluated via objective human coding.

Stage 3. During Stage 3 PRI research members will identify 10-15 trainings to recruit from (assuming we will be able to recruit 50% of participants in each training, based on previous participation rates in PRI-based research). Individuals that have signed up for a PRI training will be recruited into the study approximately 6 weeks prior to the training via email invitation. Interested participants will complete informed consent online, and following consent, therapist participants will be randomized into ClientBot (CB) vs. training as usual (TAU) conditions. Participants will then be asked to complete the computer assessment of simulated patient interviews (CASPI), a computer-based, standardized assessment of MI skills that is highly correlated with objective measures of MI fidelity. We will recruit and randomize a total of 200 PRI trainees, using their ongoing online training. PRI trains approximately 1,200 individuals each year. Based on our pilot study, we anticipate a 30% recruitment rate, which would yield the required sample size of 200 in approximately 6-8 months. Research participants will be randomized to use ClientBot immediately following their PRI training, or training-as-usual (TAU), which involves no intervention immediately following training. Prior to the ClientBot intervention, participants in both arms will complete the Computer Assessment of Simulated Patient Interviews (CASPI). The CASPI presents brief client videos with prompts to use various MI skills after each video, which are recorded and then scored for MI fidelity. Participants in both arms will complete follow-up CASPI assessments after the ClientBot training period. After a 2 month washout period, the ClientBot arm will receive an additional follow-up CASPI. Both arms will then be asked to engage in standardized patient sessions to fulfill their PRI training requirements. After completion of SP sessions, participants in the TAU arm will be offered the ClientBot training intervention, followed by a final CASPI. The final months of Phase 2 / Year 2 will focus on data analyses and dissemination of findings via conference presentations and publications.

• Study Type: Interventional
• Enrollment (Actual): 164
• Phase: Not Applicable

Contacts and Locations

Study Locations

• United States
  • Kentucky
    • Lexington, Kentucky, United States, 40503
      • Prevention Research Institute

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 18 years and older (Adult, Older Adult)
• Accepts Healthy Volunteers: Yes

Description

Inclusion Criteria:

• Clinician trainee participating in training with Prevention Research Institute (PRI)

Exclusion Criteria:

• None

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Other
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: None (Open Label)

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
No Intervention: Treatment as Usual (TAU); TAU will receive no training during the intervention period.
Experimental: ClientBot (CB); CB trainees will engage in 5 weekly practice sessions with ClientBot	Other: ClientBot; ClientBot is an AI-based simulated client that supports skills improvement for providers learning and using Motivational Interviewing

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
CASPI: Computer Assessment of Simulated Patient Interviews	The CASPI measures clinician effective use of Motivational Interviewing	Baseline
CASPI: Computer Assessment of Simulated Patient Interviews	The CASPI measures clinician effective use of Motivational Interviewing	Post-training with ClientBot, an average of 1 month following Baseline
CASPI: Computer Assessment of Simulated Patient Interviews	The CASPI measures clinician effective use of Motivational Interviewing	2 months post-training with ClientBot, an average of 3 months following Baseline

Collaborators and Investigators

• Sponsor: Lyssn.io, Inc.
• Collaborators: National Institute on Alcohol Abuse and Alcoholism (NIAAA); Prevention Research Institute

Study record dates

Study Major Dates

• Study Start (Actual): January 31, 2023
• Primary Completion (Actual): November 10, 2024
• Study Completion (Actual): November 10, 2024

Study Registration Dates

• First Submitted: July 25, 2022
• First Submitted That Met QC Criteria: July 26, 2022
• First Posted (Actual): July 29, 2022

Study Record Updates

• Last Update Posted (Estimated): December 13, 2024
• Last Update Submitted That Met QC Criteria: December 10, 2024
• Last Verified: December 1, 2024

More Information

Terms related to this study

• Keywords: Motivational Interviewing; Alcohol Use Disorder; Chatbot
• Additional Relevant MeSH Terms: Mental Disorders; Substance-Related Disorders; Chemically-Induced Disorders; Drinking Behavior; Alcohol-Related Disorders; Alcoholism; Alcohol Drinking
• Other Study ID Numbers: Pro00042065; 1R44AA028463-01 (U.S. NIH Grant/Contract)

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No
• product manufactured in and exported from the U.S.: No