Validation of an Automated Online Language Interpreting Tool - Phase Two.

A Clinical Trial to Validate an Automated Online Language Interpreting Tool With Hispanic Patients Who Have Limited English Proficiency - Phase Two.

There is a pressing national need to provide higher-quality, more effectively accessible language interpretation services to improve the health outcomes of Americans who have limited English proficiency (LEP). This project addresses a critical component of this problem: The need to improve access to high quality, mental health services for diverse populations by improving the flow of clinical work across care settings (primary care and specialty care) through the use of innovative online asynchronous methods of language interpretation and clinical communication. The investigators are conducting a two phase study. The first phase is completed and involved developing and testing the interpreting tool. The second phase of the research is a clinical trial to compare two methods of cross-language psychiatric assessment.

Study Overview

• Status: Completed
• Conditions: Substance Use Disorder; Anxiety Disorder; Mood Disorder; Chronic Medical Condition
• Intervention / Treatment: Behavioral: Active comparator: Human interpreter; Behavioral: Intervention: Asynchronous Telepsychiatry

Detailed Description

The investigators propose to develop and test a novel automated asynchronous interpretation tool. The proposed project builds on previous research, piloting the automated asynchronous interpretation tool. This 5-year project will be conducted in two phases. In Phase 1 the investigators iteratively evaluate and refine the automated asynchronous interpretation tool already developed in prior studies. In Phase 2, the investigators evaluate this tool using a two-group randomized cross-over trial. Investigators compare:

• Method A (current gold standard of in-person real-time interpreting practice). A Spanish-speaking patient is diagnostically assessed in-person by an English-speaking psychiatrist through a Spanish-speaking interpreter.
• Method B (comparative practice - ATP). A Spanish-speaking patient is interviewed in Spanish by a trained mental health interviewer. The interview is recorded in real time, translated into English using the automatic interpretation tool and adding sub-titles to the video file, and sent to an English-speaking psychiatrist to asynchronously - that is, at a later time - review the video and make a diagnosis.

All patients will undergo evaluation by both methods. Half of the patients will be randomized to be assessed by Method A first, followed by Method B and half to be assessed by Method B first, followed by Method A. The specific aims of the study are :

• Aim 1: To iteratively evaluate and refine the automated asynchronous interpretation tool already developed in prior studies.
• Aim 2: To compare patient satisfaction of Method A vs. Method B.
• Aim 3: To compare the diagnostic accuracy and psychiatrist inter-rater reliability of Method A vs. Method B and demonstrate psychiatrist inter-rater reliability for Method B.
• Aim 4: To compare the interview and language interpretation quality and accuracy of Method A vs. Method B.

• Study Type: Interventional
• Enrollment (Actual): 114
• Phase: Not Applicable

Contacts and Locations

Study Locations

• United States
  • California
    • Sacramento, California, United States, 95817
      • UC Davis Medical Center

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 18 years and older (ADULT, OLDER_ADULT)
• Accepts Healthy Volunteers: Yes
• Genders Eligible for Study: All

Description

Inclusion Criteria: Inclusion Criteria: Hispanic patients with significant Limited English proficiency (LEP)

• aged 18 or older
• diagnosis of a mood disorder, anxiety disorder, or substance/alcohol abuse disorder(s)
• Diagnosis of a chronic medical condition
• referred by PCP or self-referral with PCP informed

Exclusion Criteria: Criteria:

• less than 18 years
• imminent suicidal ideation and/or plans
• immediate violent intentions or plans
• significant cognitive deficits
• patient who's PCP recommends not participating.

Study Plan

How is the study designed?

Design Details

• Primary Purpose: HEALTH_SERVICES_RESEARCH
• Allocation: RANDOMIZED
• Interventional Model: CROSSOVER
• Masking: NONE

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Other: Method A first then Method B; Conventional in-person interview with a psychiatrist with a live human interpreter with crossover to asynchronous telepsychiatry	Behavioral: Active comparator: Human interpreter; A Spanish-speaking patient is diagnostically assessed in-person by an English-speaking psychiatrist through a Spanish-speaking interpreter.; Behavioral: Intervention: Asynchronous Telepsychiatry; A Spanish-speaking patient is interviewed in Spanish by a trained mental health interviewer. The interview is video-recorded in real-time, automatically translated into English with sub-titles added to the video file, and sent to a psychiatrist and to 3 English-speaking psychiatrists to asynchronously review the video and make a diagnosis.
Other: Method B first then Method A; Asynchronous telepsychiatry - that is, video-recorded interviews that are subsequently processed with automated speech recognition and machine translation technologies, with crossover to conventional in-person interview with a psychiatrist with a live human interpreter	Behavioral: Active comparator: Human interpreter; A Spanish-speaking patient is diagnostically assessed in-person by an English-speaking psychiatrist through a Spanish-speaking interpreter.; Behavioral: Intervention: Asynchronous Telepsychiatry; A Spanish-speaking patient is interviewed in Spanish by a trained mental health interviewer. The interview is video-recorded in real-time, automatically translated into English with sub-titles added to the video file, and sent to a psychiatrist and to 3 English-speaking psychiatrists to asynchronously review the video and make a diagnosis.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Diagnostic accuracy	Diagnosis accuracy with respect to gold standard SCID for Method B will be evaluated against Method A.	Measured at one time point by each psychiatrist, once both methods of interviews are completed in one day
Diagnostic Inter-rater reliability	Using independent assessments from four psychiatrists for each of the 100 method B interviews the investigators will also examine inter-rater reliability of Method B diagnosis.	Measured at one time point by each psychiatrist, once both methods of interviews are completed in one day

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Satisfaction ratings	Satisfaction ratings from patients' questionnaires	Measured at one time point by each psychiatrist, once both methods of interviews are completed in one day
Interview and language interpretation quality	All 200 interviews in both methods will be video recorded and the audio transcribed. The investigators will measure and compare the two interview methods interpretation quality through perception of interpretation quality	Measured at one time point by each psychiatrist, once both methods of interviews are completed in one day
Interview and language interpretation accuracy	All 200 interviews in both methods will be video recorded and the audio transcribed. The investigators will measure and compare the two interview methods accuracy through the presence of medical errors, and the presence of language errors	Measured at one time point by each psychiatrist, once both methods of interviews are completed in one day

Collaborators and Investigators

• Sponsor: University of California, Davis
• Collaborators: Agency for Healthcare Research and Quality (AHRQ)

Publications and helpful links

General Publications

• Tougas H, Chan S, Shahrvini T, Gonzalez A, Chun Reyes R, Burke Parish M, Yellowlees P. The Use of Automated Machine Translation to Translate Figurative Language in a Clinical Setting: Analysis of a Convenience Sample of Patients Drawn From a Randomized Controlled Trial. JMIR Ment Health. 2022 Sep 6;9(9):e39556. doi: 10.2196/39556.
• Parish MB, Gonzalez A, Hilty D, Chan S, Xiong G, Scher L, Liu D, Sciolla A, Shore J, McCarron R, Kahn D, Iosif AM, Yellowlees P. Asynchronous Telepsychiatry Interviewer Training Recommendations: A Model for Interdisciplinary, Integrated Behavioral Health Care. Telemed J E Health. 2021 Sep;27(9):982-988. doi: 10.1089/tmj.2020.0076. Epub 2021 Jan 12.

Study record dates

Study Major Dates

• Study Start (Actual): November 14, 2017
• Primary Completion (Actual): July 31, 2021
• Study Completion (Actual): July 31, 2021

Study Registration Dates

• First Submitted: March 1, 2018
• First Submitted That Met QC Criteria: May 15, 2018
• First Posted (Actual): May 29, 2018

Study Record Updates

• Last Update Posted (Actual): October 14, 2021
• Last Update Submitted That Met QC Criteria: October 12, 2021
• Last Verified: October 1, 2021

More Information

Terms related to this study

• Keywords: Telepsychiatry; AsynchronousTelepsychiatry; Automated Translation
• Additional Relevant MeSH Terms: Mental Disorders; Chemically-Induced Disorders; Pathologic Processes; Substance-Related Disorders; Disease; Anxiety Disorders; Mood Disorders
• Other Study ID Numbers: 1131922; R01HS024949 (U.S. AHRQ Grant/Contract)

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No