Strategic Treatment and Assessment for Youth (STAY) (STAY)

Strategic Treatment Assessment With Youth (STAY): A Measurement-based Care Approach to Promote Treatment Retention Among Racial and Ethnic Minoritized Youth With Depression or Suicide Risk

This multi-method, multi-phase pilot trial is designed to explore initial implementation and effectiveness of a theoretically-driven, culturally-tailored, measurement-based care (MBC) approach, Strategic Treatment Assessment for Youth (STAY), for racial and ethnic minoritized (REM) youth with depressive symptoms or suicidal thoughts and behaviors (STB). STAY offers an innovative, culturally-tailored approach to retain REM youth with depressive symptoms with or without STB in mental health treatment. In this R34 study, the investigators will first refine the STAY clinical protocol and implementation plan, develop STAY instrumentation (STAY, Aim 1), and then pilot test the effectiveness of STAY in three community-based mental health clinics (Aim 2). Aim 2 focuses on testing the protocol through a pilot Hybrid Type 2 trial by comparing clinicians randomly assigned to STAY (intervention condition) as compared to those assigned to the MBC As Usual (active comparison condition). Results from this pilot trial will inform optimal study procedures, measures, and site selection for a subsequent, fully-powered Hybrid Type 2 trial to examine STAY implementation, effectiveness, and scalability.

Study Overview

• Status: Enrolling by invitation
• Conditions: Suicide Prevention; Depression in Adolescence
• Intervention / Treatment: Behavioral: Strategic Treatment Assessment for Youth (STAY); Behavioral: Measurement Based Care As Usual

Detailed Description

This multi-method, multi-phase pilot trial is designed to explore initial implementation and effectiveness of a theoretically-driven, culturally-tailored, measurement-based care (MBC) approach, Strategic Treatment Assessment for Youth (STAY), for racial and ethnic minoritized (REM) youth with depressive symptoms or suicidal thoughts and behaviors (STB). STAY offers an innovative, culturally-tailored approach to retain REM youth with depressive symptoms with or without STB in mental health treatment. In this R34 study, the investigators will first refine the STAY clinical protocol and implementation plan, develop STAY instrumentation (STAY, Aim 1), and then pilot test the effectiveness of STAY in three community-based mental health clinics (Aim 2). Aim 2 focuses on testing the protocol through a pilot Hybrid Type 2 trial by comparing clinicians randomly assigned to STAY (intervention condition) as compared to those assigned to the MBC As Usual (active comparison condition). Results from this pilot trial will inform optimal study procedures, measures, and site selection for a subsequent, fully-powered Hybrid Type 2 trial to examine STAY implementation, effectiveness, and scalability.

In Aim 1 of the research plan, the investigators will refine the STAY protocol and intervention plan based on the study team's preliminary development of the STAY model, protocol, and implementation plan. The STAY protocol includes a clinician manual (including clinician guide for each STAY component, step-by-step instructions, sample scripts, clinical vignettes) and assessment measures. The STAY implementation plan includes the STAY training, goals and objectives, responsibilities, and timelines. The investigators will employ user-centered design (UCD) methods guided by a Discover, Design, Build, and Test (DDBT) Development process to revise the STAY protocol and implementation plan to ensure STAY is aligned with the needs of deployment contexts and end users. A sample size of N=12 users (i.e., clinicians and clinical administrators) divided into three cohorts of four participants each will be used as per recommendations for user-centered design (UCD) testing for complex interventions. Aligned with best practices, sampling will balance both novice and more experienced users. Participants will interact with the STAY protocol and implementation plan prototypes and engage in usability tests, including cognitive walk-throughs and lab-based user testing. Participants will also compete the 10-item Intervention Usability Scale. Iterative data analysis and prototyping between each cohort will be used to rapidly refine the STAY protocol and implementation plan to optimize usability. Then, the STAY Observational Coding System (SOCS) will be developed to assess its feasibility and scalability for future trials, as well as a 10-item STAY Knowledge, Skills and Attitudes (KSAT) Measure to tap core knowledge domains and implementation mechanisms to be assessed in the Aim 2 trial. At the end of Aim 1, the study team will have refined a STAY protocol and implementation plan. The investigators will also have an observational coding system to assess STAY fidelity and a brief, pragmatic measure to assess knowledge, attitudes and practices of clinicians trained in STAY.

In Aim 2 of the research plan, the investigators will conduct a pilot effectiveness-implementation Hybrid Type 2 trial of STAY vs MBC As Usual. Using a randomized design, 20 clinicians at three public mental health sites will be stratified by site and randomly assigned to STAY (intervention condition) or to MBC As Usual (active control condition). Clinicians randomized to STAY will receive the STAY protocol, and MBC As Usual clinicians will receive standard MBC educational materials used in the study team's other MBC studies. Clinicians in both conditions will attend a one-day training and monthly, 1-hour, post-training consultation calls for 6 months. Clinicians in both conditions will be asked to implement MBC with REM youth and their families who meet eligibility criteria and facilitate study referrals. All clinicians will complete measures at baseline (pre-training) and post-training and post-implementation (6-month post-training follow-up). Half of the clinicians in each condition will be randomly selected to participate in the SOCS. Youth/caregiver dyads will be recruited on a rolling basis due to enrollment patterns at the clinical sites, which is why the investigators are planning 20 months of data collection to achieve 4-month follow-up measures from all youth. Youth and caregivers will complete measures at baseline (within 30 days of their first contact with the site), 2-months (8 weeks) and 4-months (16 weeks) following the baseline measure. Youth measures capture implementation outcomes (e.g., clinicians' cultural humility and cultural comfort), engagement mechanisms (e.g., treatment relevance and acceptability, therapeutic alliance), service outcomes (e.g., attitudinal engagement), and youth symptom outcomes (e.g., depression symptoms, suicidal ideation and behavior). Caregiver measures capture implementation outcomes (e.g., clinicians' cultural comfort), engagement mechanisms (e.g., treatment relevance and acceptability, therapeutic alliance), and service outcomes (e.g., attitudinal engagement). Youth and caregivers will also be asked to consent to secondary data collection by the study team to obtain medical records information for their attendance, discharge reason and fidelity of MBC. Clinician measures capture implementation outcomes (e.g., feasibility, acceptability, appropriateness).

The investigators will assess STAY implementation outcomes of fidelity, feasibility, acceptability, and appropriateness (primary outcomes). Further, the investigators will explore whether STAY, as compared to MBC As Usual, will result in (secondary outcomes): (1) Greater therapeutic alliance and treatment relevance and acceptability (engagement mechanisms); (2) Greater session attendance, attitudinal engagement and treatment completion (service outcomes); and (3) Greater reduction in youth depressive symptoms, suicidal ideation and behavior (treatment outcomes).

Appropriateness of Methods Follow up intervals are informed by typical treatment length and retention measures in prior research. Sample size was determined based on the resources and time frame constraints of 3 years; participating clinicians and families will be recruited in year 1, quarter 4 to finish all data collection in year 2. Also, individual clinician patterns in MBC administration and feedback can feasibly be examined with this small sample size, as has been useful in other implementation pilot studies.

Knowledge gained from this pilot will be used to refine the research strategy for a fully powered, subsequent R01 with optimal procedures. Piloting STAY with procedures that would be exactly mirrored in a future, multi-site R01 provides numerous learning opportunities about the feasibility and parameters of measures, study recruitment and data collection procedures (e.g., collecting youth- and caregiver-reported data over the phone), clinician engagement in consultation calls, record review procedures to track fidelity, missing data patterns, and range and pattern of session attendance for both the STAY and MBC As Usual groups. Also, site variation in this trial is expected to provide information about how different sites handle STAY implementation, which the investigators can further investigate as site-level variation in a fully-powered R01 trial. Outcomes of this trial could inform potential additions to the R01 design, such as 1) enhancements to STAY to more comprehensively address logistical barriers, 2) testing STAY among REM adolescents with conditions other than depression, and/or 3) a fully-powered Type 2 Hybrid trial to detect clinician effects, moderators, and effectiveness outcomes.

• Study Type: Interventional
• Enrollment (Estimated): 140
• Phase: Not Applicable

Contacts and Locations

Study Locations

• United States
  • Connecticut
    • New Haven, Connecticut, United States, 06511
      • Yale University

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Child; Adult; Older Adult
• Accepts Healthy Volunteers: No

Description

*Note: This study includes data collection from mental health clinicians, youth, and parents/caregivers. The age limits for each participant group vary and are specified below.

Inclusion Criteria for Aim 2 Adolescent client participants:

• Age 12 to 18
• Identify as a racial or ethnic minoritized (i.e., Black or African American; American Indian or Alaska Native; Asian American, Native Hawaiian or Other Pacific Islander; Hispanic or Latinx) youth
• Have at least moderate depression symptoms based on moderate range of a validated depression assessment collected at intake by the participating agency such as the Center for Epidemiological Studies Depression Scale for Children (CES-DC) or Hamilton Depression Index with or without Parent- or adolescent- reported or healthcare provider- documented thoughts and/or behaviors related to self-harm or suicide in the three (3) months prior to first contact with participating agency
• Assigned or triaged to work with one of the clinician participants included in the study (below)
• Made first contact (e.g., initial phone call, attended intake appointment) with a participating agency less than or equal to one (1) month prior to the recruitment date (but ideally within (1) week of intake appointment)
• English- or Spanish-speaking
• Legal parent or guardian available to provide informed consent
• Able to provide informed assent

Inclusion Criteria for Aim 2 Parent/Guardian participants:

• Legal guardian of one of the adolescent client participants included in the study (above)
• English-speaking
• Able to provide informed consent

Inclusion Criteria for Aim 2 Clinician participants:

• Provides community-based mental health treatment to youth
• Holds a professional license or certification in their state to provide mental health treatment OR are supervised by a clinician with professional license or certification in their state to provide mental health treatment
• Are a clinician identified for recruitment because they are part of a participating mental health clinic/site
• Age 18 or older
• English- or Spanish-speaking
• Able to provide informed consent

Individuals who do not meet all inclusion criteria are, by definition, excluded. There are no other exclusion criteria.

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Treatment
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: Single

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Experimental: Strategic Treatment Assessment for Youth (STAY); Clinicians assigned to the STAY condition will receive training and ongoing consultation in the STAY protocol and implementation plan. STAY is a culturally-modified version of measurement-based care (MBC) for racial and ethnic minoritized adolescents. Clinicians in this condition will receive training materials and consultation on standard MBC practices using the PHQ-9-A for depression plus cultural competence training, the "Introduce" component to assess and address treatment expectations at the start of care, and individualized goal and alliance progress measures.	Behavioral: Strategic Treatment Assessment for Youth (STAY); Culturally tailored measurement based care
Active Comparator: Measurement-Based Care (MBC) As Usual; Clinicians assigned to the active comparison condition will receive standard MBC training and ongoing consultation that is routinely used and evaluated by the research team with child and adolescent mental health clinicians. MBC As Usual training materials will be customized to focus on collection and use of the PHQ-A with adolescents and their caregivers using a feedback system. MBC As Usual clinicians will be offered the opportunity to be trained in STAY at the conclusion of data collection	Behavioral: Measurement Based Care As Usual; Measurement based care as usual

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Patient Health Questionnaire-9 Item Version	The Patient Health Questionnaire-9 (PHQ-9) is a 9-item measure for use with individuals over 12 years of age. Items are scored on a 3-point Likert scale (0="Not at all"; 3= "Nearly every day") with greater scores indicative of higher depression. The PHQ-9 has been well established as a brief depression severity measure and used in numerous studies examining the effectiveness of MBC for depression. Internal consistency of the PHQ-9 is high (α>0.80) and criterion validity was established by conducting 580 structured interviews which showed that individuals who scored high (≥ 10) on the PHQ-9 were 7 to 13.6 times more likely to be diagnosed with depression. Moreover, among a sample of 442 adolescents enrolled in mental health treatment, a PHQ-9 score of ≥11 had a sensitivity of 89.5% and specificity of 77.5% for detecting youth with major depression.	Baseline, 2-Month Follow-Up, 4-Month Follow-Up
Columbia-Suicide Severity Rating Scale (C-SSRS)	The Columbia-Suicide Severity Rating Scale (C-SSRS) is a semi-structured interview that measures suicide ideation and behavior. It is predictive of lifetime history of non-suicidal self-injury among adolescents and suicide attempts at return visit to an emergency department. The C-SSRS is the leading national suicide assessment measure and has been used in numerous National Institute of Mental Health (NIMH) clinical trials. In three multisite studies with 673 adolescents and adults, internal consistency on the intensity subscale ranged from moderate (α=.73) to high (α=.95) and the C-SSRS demonstrated good convergent and divergent validity with other prominent suicidal ideation/behavior scales (100% specificity and 100% sensitivity in correctly identifying lifetime suicide attempts).	Baseline, 2-Month Follow-Up, 4-Month Follow-Up
Session Attendance (Behavioral Treatment Engagement)	Behavioral engagement will be measured by participation in sessions, which is defined as the number of sessions attended divided by number of sessions scheduled over a 4-month period.	4-Month Follow-Up
Patient Activation Measure (Attitudinal Treatment Engagement)	Attitudinal engagement will be measured using the reduced, 13-item version of the Patient Activation Measure (PAM). This measure has strong psychometric properties particularly in relation to the other limited measures of this construct; the theoretical range of items and mean scores is 0-100 but means among a sample of 1,515 individuals with a high school education or less range from 38-51, where higher scores reflect higher levels of activation.	Baseline, 2-Month Follow-Up, 4-Month Follow-Up
Treatment Retention (Treatment Completion/Continuation)	Treatment retention will be assessed via records review for successful completion or continuation vs. whether they terminated inappropriately (e.g., early) at four months after treatment started. If the patient has terminated services by the 4-month mark, the investigators will code the discharge reason as "successful completion" (i.e., retained) or "premature termination" (i.e., not retained). If the patient is still in services at the 4-month mark, they will be considered "retained" unless they have missed all three of the most recently scheduled consecutive sessions. This is an established treatment retention measure.	4-Month Follow-Up
Treatment Evaluation Inventory - Short Form	Treatment relevance and acceptability will be measured by the caregiver- and youth-reported Treatment Evaluation Inventory (TEI). The 15 items are rated on a 7-point Likert scale (1="strongly disagree"; 7="strongly agree") that load on a unitary factor. Item scores are summed to provide a general treatment acceptability index, with higher values indicating higher rated acceptability. The Treatment Evaluation Inventory-Short Form (TEI-SF) has shown variable internal consistency (α>0.35 to 0.80) and is considered to be the first and most frequently used instrument of treatment acceptability. The study team will adapt wording in items 1 and 2 to refer to depression instead of problem behavior.	Baseline, 2-Month Follow-Up, 4-Month Follow-Up
Working Alliance Inventory - Short Revised	Therapeutic alliance will be measured by youth and caregiver report on the Working Alliance Inventory - Short Revised (WAI-SR). This 12-item measure was developed using confirmatory factor analysis and item response theory on the original WAI and includes Goal, Task and Bond constructs. Items are rated on a 5-point Likert scale (1="Seldom"; 5="Always"), with higher scores indicating stronger alliance. Reliability is strong (α>0.80), convergent validity is adequate r >0.64), as is model fit.	Baseline, 2-Month Follow-Up, 4-Month Follow-Up
Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition, Text Revision (DSM-5-TR) Self-Rated Level 1 Cross-Cutting Symptom Measure Child Age 11-17	This measure consists of 25 questions that assess 12 mental health domains that are important across psychiatric diagnoses. These domains include depression, anger, irritability, mania, anxiety, somatic symptoms, inattention, suicidal ideation/attempt, psychosis, sleep disturbance, repetitive thoughts and behaviors, and substance use. Each item asks the child, age 11-17, to rate how much or how often they have been bothered by each symptom during the past 2 weeks. 19 items are rated on a 5-point scale (0=none or not at all; 1=slight or rare, less than a day or two; 2=mild or several days; 3=moderate or more than half the days; and 4=severe or nearly every day). The suicidal ideation, suicide attempt, and substance abuse items are rated on a "Yes" or "No" scale. This measure was found to have good test-retest reliability in the DSM-5 Field Trials conducted in pediatric clinical samples across the United States.	Baseline, 2-Month Follow-Up, 4-Month Follow-Up
Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition, Text Revision (DSM-5-TR) Parent/Guardian-Rated Level 1 Cross-Cutting Symptom Measure-Child Age 6-17	This measure consists of 25 questions that assess 12 mental health domains that are important across psychiatric diagnoses. These domains include depression, anger, irritability, mania, anxiety, somatic symptoms, inattention, suicidal ideation/attempt, psychosis, sleep disturbance, repetitive thoughts and behaviors, and substance use. Each item asks the parent/guardian to rate how much or how often their child has been bothered by each symptom during the past 2 weeks. 19 items are rated on a 5-point scale (0=none or not at all; 1=slight or rare, less than a day or two; 2=mild or several days; 3=moderate or more than half the days; and 4=severe or nearly every day). The suicidal ideation, suicide attempt, and substance abuse items are rated on a "Yes", "No" or "Don't know" scale. This measure was found to have good test-retest reliability in the DSM-5 Field Trials conducted in pediatric clinical samples across the United States.	Baseline, 2-Month Follow-Up, 4-Month Follow-Up
Revised Child Anxiety and Depression Scale (RCADS-25)	This 25-item measure is a revised shortened version of the original Revised Child Anxiety and Depression Scale (RCADS). Each item is rated on a 4-point scale (0=never; 3=Always). There are two subscales: an Anxiety Total scale and a Depression Total scale, with higher sum scores representing greater severity of general anxiety and depressive symptoms. Both subscales have shown acceptable reliability within clinic-referred and school-based samples.	Baseline, 2-Month Follow-Up, 4-Month Follow-Up

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Strategic Treatment Assessment for Youth (STAY) Observational Coding System	The STAY Observational Coding System (SOCS) will be developed by the investigators to include observational items with anchored ratings. A coder's manual will be developed for each STAY component and reviewed by collaborators and consultants. To optimize likelihood of the SOCS being used in whole or in part by others, the investigators will include recommendations for modified versions (e.g., cultural competence delivering other interventions, standard measurement-based care competence, and use by clinical leaders for adherence monitoring). Clinicians will be asked to audiotape three sessions with two clients and securely transfer them to the study team for coding using the observational coding system.	6 Months Post-Training
Strategic Treatment Assessment for Youth (STAY) Feasibility	Feasibility will be measured using clinician self-report on the Feasibility of Intervention Measure (FIM), collected using a web-based, clinician-report survey. The FIM is a 4-item measure (1= 'Completely Disagree"; 5 = "Completely Agree"), with higher values reflecting higher fidelity, developed based on input from 36 implementation scientists and 27 mental health professionals. Structural validity, test-retest reliability, and sensitivity to change have been established.	Baseline (pre-training), Immediately Post-Training, 6 Months Post-Training
Strategic Treatment Assessment for Youth (STAY) Appropriateness	Appropriateness will be measured by clinician self-report on the Intervention Appropriateness Measure (IAM). This 4-item measure (1= 'Completely Disagree"; 5 = "Completely Agree"), with higher values reflecting higher fidelity, developed based on input from 36 implementation scientists and 27 mental health professionals. Structural validity, test-retest reliability, and sensitivity to change have been established.	Baseline (pre-training), Immediately Post-Training, 6 Months Post-Training
Strategic Treatment Assessment for Youth (STAY) Acceptability	Acceptability will be measured by clinician self-report on the Acceptability of Intervention Measure (AIM) via web-based, clinician-report survey. This 4-item measure (1= 'Completely Disagree"; 5 = "Completely Agree"), with higher values reflecting higher fidelity, developed based on input from 36 implementation scientists and 27 mental health professionals. Structural validity, test-retest reliability, and sensitivity to change have been established.	Baseline (pre-training), Immediately Post-Training, 6 Months Post-Training
Strategic Treatment Assessment for Youth (STAY) Knowledge, Skills and Attitudes	Knowledge, skills and attitudes regarding Strategic Treatment Assessment for Youth (STAY) will be evaluated using a 10-item measure developed by the investigators to tap the core knowledge domains and implementation mechanisms potentially influencing STAY implementation. Items will be modeled after the 40-item and more recent 17-item Knowledge of Evidence-Based Services Questionnaire, the 18-item Attitudes toward Standardized Assessment Scale-Monitoring and Feedback and the 4-item Current Assessment Practice Evaluation. The study team will select and/or adapt items most specific to central aspects of the STAY training content and refine and reduce a final set of items based on rapid pilot testing and feedback among the study team, collaborators and contributors. The response scale for the measure will also be developed based on feedback from collaborators and contributors.	Baseline (pre-training), Immediately Post-Training, 6 Months Post-Training
Current Assessment Practice Evaluation (CAPE)	Measurement-Based Care practices in general will be assessed by the Current Assessment Practice Evaluation (CAPE). This 4-item measure collects clinician self-reported use of MBC practices across different phases of intervention (e.g., at intake, ongoing during treatment, at termination). Items are reported on a 4-point scale (None, Some [1-39%], Half [40-60%], Most [61-100%]) with higher ratings reflecting more frequent use of MBC practices. Internal consistency is acceptable (alpha = 0.72) and has been replicated in the study team's prior work with a sample of 95 clinicians.	Baseline (pre-training), Immediately Post-Training, 6 Months Post-Training
Attitudes toward Standardized Assessment Scale - Monitoring and Feedback	Attitudes toward MBC in general will be assessed via the Attitudes toward Standardized Assessment Scale-Monitoring and Feedback (ASA-MF). This 18-item measure includes three subscales, Clinical Utility, Treatment Planning and Practicality, which have shown acceptable internal consistency (alphas between 0.81 and 0.85) which has been replicated in the study team's prior work. Items are scored on a scale from 1 (Strongly Disagree) to 5 (Strongly Agree), with higher values indicating more positive attitudes towards standardized assessment.	Baseline (pre-training), Immediately Post-Training, 6 Months Post-Training
Multicultural Counseling Knowledge and Awareness Scale	Clinician practices in culturally-responsive treatment delivery will be assessed via the clinician self-reported Multicultural Counseling Knowledge and Awareness Scale (MCKAS) and the youth-reported Cultural Humility Scale (CHS). The Multicultural Counseling Knowledge and Awareness Scale (MCKAS) is a 32-item measure (1 = "Not At All True"; 7 = "Totally True") consisting of 20 knowledge items and 12 awareness items, with higher values indicating higher knowledge and awareness. The MCKAS possesses a moderate degree of validity and a moderate to high degree of both internal consistency and test-retest reliability and was found to be the least influenced by social desirability attitudes among respondents on similar measures. The MCKAS was developed by distinguished national experts in multicultural counseling and appears to be the leading measure in this domain.	Baseline (pre-training), Immediately Post-Training, 6 Months Post-Training
Cultural Humility Scale (CHS)	Cultural humility, or the degree to which a therapist can create a therapeutic environment that respects cultural differences and an openness to different beliefs, values, and worldviews, will be measured by youth report on the Cultural Humility Scale (CHS). The 12 items are rated on a 5-point Likert scale (1 = "strongly disagree"; 5 = "strongly agree") by the youth about their clinician, with higher values reflecting higher perceptions of cultural humility. Reliability is strong.	Baseline, 2-Month Follow-Up, 4-Month Follow-Up
Implementation Climate Scale (ICS)	Implementation climate will also be measured for each site in order to understand organizational-level climate to support STAY implementation. The Implementation Climate Scale (ICS) will be used to understand clinicians' perceptions of the policies, practices, procedures and behaviors that are rewarded, supported and expected to facilitate effective STAY implementation. This 38-item measure (0 = "Not at all"; 4 = Very great extent") assesses strategic climate for evidence-based practice implementation, with higher values indicating a more positive and supportive implementation climate. The investigators will adjust measures to assess specifically for STAY. Internal consistency is strong (α = 0.81-0.91) with high factor loadings on 6 factors among a sample of 630 individuals working in 128 teams.	Baseline, 6 Months Post-Training
Treatment Outcome Expectations Scale (TOES)	Treatment outcome expectations will be measured by the caregiver and youth-reported Treatment Outcome Expectations Scale (TOES). TOES contains eight items that measure youth and caregiver expectations of possible therapy outcomes. Items are scored on a three-point Likert scale (0="I do not expect this"; 3= "I do expect this"). The mean of each informant's scores is calculated to yield a TOES Total Score, with a higher total score indicating higher expectations. Sound psychometric properties have been established, with internal consistency ranging from (α = 0.83-0.91) for youth reported TOES, and adequate internal consistency (α = 0.86) for caregiver reported TOES. Confirmatory factor analysis revealed that a one-factor model fit data for youth responses on TOES best, although fit indices for a one-factor model were less ideal for the caregiver reported TOES.	Baseline, 2-Month Follow-Up, 4-Month Follow-Up
Treatment Process Expectations Index (TPEI)	Expectations about youth and caregivers' role in treatment and the treatment process will be measured by the Treatment Process Expectations Index (TPEI). The TPEI contains nine items, which are scored on a three-point Likert scale (0= "I do not expect this"; 3= "I do expect this"). Psychometric findings are not available, as responses on the TPEI are meant to be used on an item level, offering clinical utility and relevance that can supplement data from TOES.	Baseline, 2-Month Follow-Up, 4-Month Follow-Up
Therapist Cultural Comfort Scale (TCCS)	The perceived cultural comfort of therapists will be measured by the youth reported Therapist Cultural Comfort Scale (TCCS). The TCCS contains 13 items, which are scored on a five-point Likert scale (1= "Strongly Disagree"; 5= "Strongly Agree"). An exploratory and confirmatory factor analysis supported two subscales, a Negative subscale, which had items that were negative markers of cultural comfort, and Positive subscale, which had items indicative of greater cultural comfort. Sound psychometric properties have been found including reliability, convergent validity, and incremental validity. Internal consistencies have been found adequate (α = 0.77-0.94). Adequate test-retest reliability has also been demonstrated (interclass correlation coefficient= .82).	Baseline, 2-Month Follow-Up, 4-Month Follow-Up
Intervention Usability Scale (IUS)	Usability will be evaluated by a clinician self-report on the Intervention Usability Scale (IUS). This 10-item measure (0= "Strongly Disagree"; 4= "Strongly Agree") was adapted from the System Usability Scale (SUS) to assess usability of evidence-based psychosocial interventions; a two-factor structure has been found, with adequate internal consistency (α = 0.83). Higher values indicate greater perceived usability.	6 Months Post-Training

Collaborators and Investigators

• Sponsor: Yale University
• Collaborators: National Institute of Mental Health (NIMH); Teachers College, Columbia University
• Investigators: Principal Investigator: Elizabeth Connors, PhD, Yale University

Publications and helpful links

General Publications

• Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. 2001 Sep;16(9):606-13. doi: 10.1046/j.1525-1497.2001.016009606.x.
• Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, Currier GW, Melvin GA, Greenhill L, Shen S, Mann JJ. The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry. 2011 Dec;168(12):1266-77. doi: 10.1176/appi.ajp.2011.10111704.
• Hibbard JH, Mahoney ER, Stockard J, Tusler M. Development and testing of a short form of the patient activation measure. Health Serv Res. 2005 Dec;40(6 Pt 1):1918-30. doi: 10.1111/j.1475-6773.2005.00438.x.
• Weiner BJ, Lewis CC, Stanick C, Powell BJ, Dorsey CN, Clary AS, Boynton MH, Halko H. Psychometric assessment of three newly developed implementation outcome measures. Implement Sci. 2017 Aug 29;12(1):108. doi: 10.1186/s13012-017-0635-3.
• Hibbard JH, Stockard J, Mahoney ER, Tusler M. Development of the Patient Activation Measure (PAM): conceptualizing and measuring activation in patients and consumers. Health Serv Res. 2004 Aug;39(4 Pt 1):1005-26. doi: 10.1111/j.1475-6773.2004.00269.x.
• Turner AM, Reeder B, Ramey J. Scenarios, personas and user stories: user-centered evidence-based design representations of communicable disease investigations. J Biomed Inform. 2013 Aug;46(4):575-84. doi: 10.1016/j.jbi.2013.04.006. Epub 2013 Apr 22.
• Ebesutani C, Chorpita BF, Higa-McMillan CK, Nakamura BJ, Regan J, Lynch RE. A psychometric analysis of the Revised Child Anxiety and Depression Scales--parent version in a school sample. J Abnorm Child Psychol. 2011 Feb;39(2):173-85. doi: 10.1007/s10802-010-9460-8.
• Gipson PY, Agarwala P, Opperman KJ, Horwitz A, King CA. Columbia-suicide severity rating scale: predictive validity with adolescent psychiatric emergency patients. Pediatr Emerg Care. 2015 Feb;31(2):88-94. doi: 10.1097/PEC.0000000000000225.
• Guo T, Xiang YT, Xiao L, Hu CQ, Chiu HF, Ungvari GS, Correll CU, Lai KY, Feng L, Geng Y, Feng Y, Wang G. Measurement-Based Care Versus Standard Care for Major Depression: A Randomized Controlled Trial With Blind Raters. Am J Psychiatry. 2015 Oct;172(10):1004-13. doi: 10.1176/appi.ajp.2015.14050652. Epub 2015 Aug 28.
• Lawson GM, Moore TM, Okamura KH, Becker-Haimes EM, Beidas RS. Knowledge of Evidence-Based Services Questionnaire: Development and Validation of a Short Form. Adm Policy Ment Health. 2020 Jul;47(4):581-596. doi: 10.1007/s10488-020-01020-7.
• Munder T, Wilmers F, Leonhart R, Linster HW, Barth J. Working Alliance Inventory-Short Revised (WAI-SR): psychometric properties in outpatients and inpatients. Clin Psychol Psychother. 2010 May-Jun;17(3):231-9. doi: 10.1002/cpp.658.
• Lyon AR, Koerner K. User-Centered Design for Psychosocial Intervention Development and Implementation. Clin Psychol (New York). 2016 Jun;23(2):180-200. doi: 10.1111/cpsp.12154. Epub 2016 Jun 17.
• Ehrhart MG, Aarons GA, Farahnak LR. Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation Climate Scale (ICS). Implement Sci. 2014 Oct 23;9:157. doi: 10.1186/s13012-014-0157-1.
• Ingoldsby EM. Review of Interventions to Improve Family Engagement and Retention in Parent and Child Mental Health Programs. J Child Fam Stud. 2010 Oct 1;19(5):629-645. doi: 10.1007/s10826-009-9350-2.
• Lyon AR, Pullmann MD, Jacobson J, Osterhage K, Achkar MA, Renn BN, Munson SA, Arean PA. Assessing the Usability of Complex Psychosocial Interventions: The Intervention Usability Scale. Implement Res Pract. 2021 Jan-Dec;2:2633489520987828. doi: 10.1177/2633489520987828. Epub 2021 Feb 8.
• Lyon AR, Dopp AR, Brewer SK, Kientz JA, Munson SA. Designing the Future of Children's Mental Health Services. Adm Policy Ment Health. 2020 Sep;47(5):735-751. doi: 10.1007/s10488-020-01038-x.
• Yeager DS, Romero C, Paunesku D, Hulleman CS, Schneider B, Hinojosa C, Lee HY, O'Brien J, Flint K, Roberts A, Trott J, Greene D, Walton GM, Dweck CS. Using Design Thinking to Improve Psychological Interventions: The Case of the Growth Mindset During the Transition to High School. J Educ Psychol. 2016 Apr;108(3):374-391. doi: 10.1037/edu0000098.
• Bauer AM, Baldwin SA, Anguera JA, Arean PA, Atkins DC. Comparing Approaches to Mobile Depression Assessment for Measurement-Based Care: Prospective Study. J Med Internet Res. 2018 Jun 19;20(6):e10001. doi: 10.2196/10001.
• Fortney JC, Unutzer J, Wrenn G, Pyne JM, Smith GR, Schoenbaum M, Harbin HT. A Tipping Point for Measurement-Based Care. Focus (Am Psychiatr Publ). 2018 Jul;16(3):341-350. doi: 10.1176/appi.focus.16303. Epub 2018 Jul 16.
• Glazer K, Rootes-Murdy K, Van Wert M, Mondimore F, Zandi P. The utility of PHQ-9 and CGI-S in measurement-based care for predicting suicidal ideation and behaviors. J Affect Disord. 2020 Apr 1;266:766-771. doi: 10.1016/j.jad.2018.05.054. Epub 2018 Jun 15.
• Richardson TM, He H, Podgorski C, Tu X, Conwell Y. Screening depression aging services clients. Am J Geriatr Psychiatry. 2010 Dec;18(12):1116-23. doi: 10.1097/JGP.0b013e3181dd1c26.
• Miller GE, Prinz RJ. Engagement of families in treatment for childhood conduct problems. Behavior Therapy. 2003; 34(4): 517-534.
• Kazdin AE. Acceptability of alternative treatments for deviant child behavior. J Appl Behav Anal. 1980 Summer;13(2):259-73. doi: 10.1901/jaba.1980.13-259.
• Spirrison, CL, Noland, KA, Savoie, LB. Factor structure of the Treatment Evaluation Inventory: Implications for measurement of treatment acceptability. J Psychopathol Behav Assess. 1992; 14: 65-79.
• Carter, SL. Review of Recent Treatment Acceptability Research. Education and Training in Developmental Disabilities. 2007; 42(3): 301-316.
• Hatcher, RL, Gillaspy, JA. Development and validation of a revised short version of the working alliance inventory. Psychotherapy Research. 2006; 16(1): 12-25.
• Stumpf RE, Higa-McMillan CK, Chorpita BF. Implementation of evidence-based services for youth: assessing provider knowledge. Behav Modif. 2009 Jan;33(1):48-65. doi: 10.1177/0145445508322625. Epub 2008 Aug 22.
• Jensen-Doss A, Haimes EMB, Smith AM, Lyon AR, Lewis CC, Stanick CF, Hawley KM. Monitoring Treatment Progress and Providing Feedback is Viewed Favorably but Rarely Used in Practice. Adm Policy Ment Health. 2018 Jan;45(1):48-61. doi: 10.1007/s10488-016-0763-0.
• Lyon AR, Pullmann MD, Whitaker K, Ludwig K, Wasse JK, McCauley E. A Digital Feedback System to Support Implementation of Measurement-Based Care by School-Based Mental Health Clinicians. J Clin Child Adolesc Psychol. 2019;48(sup1):S168-S179. doi: 10.1080/15374416.2017.1280808. Epub 2017 Mar 2.
• Connors EH, Douglas S, Jensen-Doss A, Landes SJ, Lewis CC, McLeod BD, Stanick C, Lyon AR. What Gets Measured Gets Done: How Mental Health Agencies can Leverage Measurement-Based Care for Better Patient Care, Clinician Supports, and Organizational Goals. Adm Policy Ment Health. 2021 Mar;48(2):250-265. doi: 10.1007/s10488-020-01063-w.
• Ponterotto, JG, Potere, JC. The Multicultural Counseling Knowledge and Awareness Scale (MCKAS): Validity, Reliability, and User Guidelines. In D. B. Pope-Davis, H. L. K. Coleman, W. M. Liu, & R. L. Toporek (Eds.), Handbook of multicultural competencies: In counseling & psychology. Sage Publications, Inc. 2003: 137-153.
• Constantine, MG, Ladany, N. Self-report multicultural counseling competence scales: Their relation to social desirability attitudes and multicultural case conceptualization ability. Journal of Counseling Psychology. 2000; 47(2): 155-164.
• Dew-Reeves SE, Athay MM. Validation and use of the youth and caregiver Treatment Outcome Expectations Scale (TOES) to assess the relationships between expectations, pretreatment characteristics, and outcomes. Adm Policy Ment Health. 2012 Mar;39(1-2):90-103. doi: 10.1007/s10488-012-0406-z.
• Riemer M, Athay MM, Bickman L, Breda C, Kelley SD, Vides de Andrade AR. The Peabody Treatment Progress Battery: history and methods for developing a comprehensive measurement battery for youth mental health. Adm Policy Ment Health. 2012 Mar;39(1-2):3-12. doi: 10.1007/s10488-012-0404-1.
• Perez-Rojas AE, Bartholomew TT, Lockard AJ, Gonzalez JM. Development and initial validation of the Therapist Cultural Comfort Scale. J Couns Psychol. 2019 Oct;66(5):534-549. doi: 10.1037/cou0000344. Epub 2019 Apr 18.
• Bartholomew TT, Perez-Rojas AE, Lockard AJ, Joy EE, Robbins KA, Kang E, Maldonado-Aguiniga S. Therapists' cultural comfort and clients' distress: An initial exploration. Psychotherapy (Chic). 2021 Jun;58(2):275-281. doi: 10.1037/pst0000331. Epub 2020 Nov 19.
• Lyon AR, Munson SA, Renn BN, Atkins DC, Pullmann MD, Friedman E, Arean PA. Use of Human-Centered Design to Improve Implementation of Evidence-Based Psychotherapies in Low-Resource Communities: Protocol for Studies Applying a Framework to Assess Usability . JMIR Res Protoc. 2019 Oct 9;8(10):e14990. doi: 10.2196/14990.
• Hook JN, Davis DE, Owen J, Worthington EL, Utsey SO. Cultural humility: measuring openness to culturally diverse clients. J Couns Psychol. 2013 Jul;60(3):353-366. doi: 10.1037/a0032595. Epub 2013 May 6.

Study record dates

Study Major Dates

• Study Start (Actual): March 25, 2025
• Primary Completion (Estimated): October 1, 2026
• Study Completion (Estimated): November 1, 2026

Study Registration Dates

• First Submitted: June 26, 2024
• First Submitted That Met QC Criteria: July 16, 2024
• First Posted (Actual): July 23, 2024

Study Record Updates

• Last Update Posted (Actual): May 21, 2025
• Last Update Submitted That Met QC Criteria: May 16, 2025
• Last Verified: May 1, 2025

More Information

Terms related to this study

• Additional Relevant MeSH Terms: Behavioral Symptoms; Self-Injurious Behavior; Suicide Prevention; Suicide; Depression
• Other Study ID Numbers: 2000036704; 24-063 (Teachers College Columbia University IRB); R34MH134915 (U.S. NIH Grant/Contract)

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO
• IPD Plan Description: However, the study team's data and other materials will be structured and described according to best practices and the National Data Archive (NDA) five required data elements (e.g., the Global Unique Identifier, GUID) which will ensure that data are consistent with the NDA data harmonization approach. Data will also be stored in common and open formats, such as clinicaltrials.gov as a national dissemination and resource sharing platform. Information needed to make use of these data [e.g. the meaning of variable names, codes, information about missing data, other metadata etc] will be recorded in data dictionaries/codebooks that will be made accessible by the research team and will subsequently be shared alongside final datasets. Information about the study team's research process, including the details of their analysis pipeline will be maintained. This information will be accessible to all members of the research team and will be shared alongside study data.

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No