Evaluation of a Remote Training Strategy

Evaluation of a Remote Training Strategy for School Personnel

An increasing number of schools in rural settings are employing the multi-tier positive behavioral interventions and supports (PBIS) framework to improve school-climate. PBIS can also be used as a framework for the adoption and integration of evidence-based practices (EBPs) for children's mental health concerns. A key challenge is that school personnel need technical assistance (training plus ongoing consultation) in order to implement EBPs with fidelity. In urban and suburban schools, this support can be provided to school staff on site. However, providing ongoing on-site support is not feasible or sustainable in the majority of rural schools, due to their remote physical location. For this reason, video technology has been recommended for the training of behavioral health staff (BHS) in under-served rural communities.

Study Overview

• Status: Completed
• Conditions: Behavior Disorders
• Intervention / Treatment: Behavioral: Coping Power Program (CPP); Behavioral: CBT for Anxiety Treatment in Schools (CATS); Behavioral: Check-in/Check-out (CI/CO); Behavioral: Initial Training (CC); Behavioral: Initial Training plus Video (RV); Behavioral: Initial Training plus Video plus Coaching (RV+)

Detailed Description

Objectives: The purpose of this study is to test the effectiveness of a remote training platform with BHS in schools serving rural communities.

Study Design: This study is a pilot randomized controlled trial to compare implementation and student outcomes of three training strategies: Control Condition (CC), Remote Video (RV), and Remote Video plus Coaching (RV+).

Setting/Participants: Participants will be school personnel with and without prior mental health training and students in grades 4-8 in schools serving rural communities who are deemed at risk for externalizing and internalizing mental health disorders. The pilot trial will be conducted in 24 schools (8 in CC, 8 in RV, and 8 in RV+). It is estimated that a total of 72 behavioral health staff (3 staff per school- 24 per arm) and 312 students (13 students per school - 104 per arm) will participate in this study. Additionally, we estimate about 24 school administrators such as principals or assistant principals will approve school's participation.

Study Interventions and Measures: Each school will be randomly assigned to receive one of three training supports: (a) Initial training (CC) (b) Initial training, plus video (RV) or (c) Initial training plus video, plus remote coaching (RV+). BHS in the RV+ condition will receive consultation from a Children's Hospital of Philadelphia (CHOP) consultant regarding implementation of EBPs, whereas BHS in the RV condition will be given access to the asynchronous training video modules. Participants in all three conditions will receive treatment manuals for the interventions they have chosen. The EBPs to be implemented by the BHS include: Coping Power Program (CPP), CBT for Anxiety Treatment in Schools (CATS), and Check-in/Check-out (CI/CO). All BHS will conduct one CPP or CATS group with 5 students of similar developmental level (e.g., a group of 4th & 5th graders), and that each school staff in CI/CO will implement the intervention with 4 separate individual students of any eligible school grade.

Analyses: This study is designed to test the effectiveness of initial training compared to remote video training and remote video training plus coaching to train BHS to implement EBPs.

• Study Type: Interventional
• Enrollment (Actual): 140
• Phase: Not Applicable

Contacts and Locations

Study Locations

• United States
  • Pennsylvania
    • Philadelphia, Pennsylvania, United States, 19104
      • Children's Hospital of Philadelphia

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 4 years to 95 years (Child, Adult, Older Adult)
• Accepts Healthy Volunteers: Yes

Description

Inclusion Criteria:

Administrator: Any school principal or assistant principal from participating schools implementing PBIS.

Behavioral Health Staff: Any counselor, social worker, or teacher from participating schools implementing PBIS who work with students in grades 4-8.

Students: Inclusion of students to receive a Tier 2 intervention is as follows:

• attending one of the participating schools
• being in grades 4-8
• identified by the Tier 2 team as not responding to Tier 1 intervention, thus needing Tier 2 support
• scoring ≥ 1 SD above the mean on the Emotional Symptoms or Conduct Problems scales of the Strength andDifficulties Questionnaire (SDQ) completed by a parent or a teacher

The cut-off score level for the SDQ is appropriate for identifying students at risk for a behavioral/mental health disorder. We limit participation to students in grades 4-8 because the group EBPs are appropriate for this age group.

Exclusion Criteria:

Administrator: School staff who are not principals or assistant principals.

Behavioral Health Staff: School staff who are not part of the PBIS team and who do not work with students in grades 4-8.

Students: Students who do not meet screening or group participation criteria will not be included in the study. Students with a history of intellectual disability or serious developmental delays according to school records will not be included because they would be unlikely to benefit from the interventions used in the study. Students with a history of psychotic or autistic spectrum disorders as reported by parents will not be included.

Subjects that do not meet all the enrollment criteria may not be enrolled. Any violations of these criteria must be reported in accordance with the Institutional Review Board (IRB) Policies and Procedures.

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Health Services Research
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: None (Open Label)

Number of Arms

3

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Active Comparator: Initial training; School personnel will implement CATS or CPP and CICO with students in the school setting. They will participate in an initial live remote training to learn about implementing the three EBPs (CC).	Behavioral: Coping Power Program (CPP); CPP is an evidence-based intervention designed for students with externalizing behavior disorder. CPP consists of twelve 45-minute sessions. This EBP has been found to be effective at reducing aggressive behavior, covert delinquent behavior and substance abuse among aggressive boys, with gains maintained at one-year follow-up. Growth curve analyses showed that CPP had linear effects for three years after intervention on reductions in aggressive behavior and academic behavior problems.; Behavioral: CBT for Anxiety Treatment in Schools (CATS); CATS is an adaptation of Friends for Life (FRIENDS). The adapted protocol retains the core elements of evidence-based CBT for anxiety and the FRIENDS group format. Investigators implemented planned adaptations to the protocol based on collective experience. Changes were made to the language, cultural methods, number of sessions, and activities while maintaining the 5 essential components of the treatment. This resulted in a briefer (8-session) and more feasible, engaging and culturally appropriate protocol for urban under-resourced schools than the original FRIENDS.; Behavioral: Check-in/Check-out (CI/CO); CI/CO is a targeted, Tier 2 intervention for students at risk of developing externalizing and internalizing mental health disorders. CI/CO is designed to provide immediate feedback (i.e., at the end of each class period) to students, based on the use of a daily report card. This feedback is developmentally sensitive. CI/CO implementers meet individually with students for a brief 'check-in' in the morning and a brief 'check-out' in the afternoon. Research on the use of CI/CO has shown it to be effective in reducing externalizing and internalizing problems with elementary school students. CI/CO will be offered to individual students for a three-month period of time, which is the same time-frame needed for the implementation of CPP.; Behavioral: Initial Training (CC); School personnel will participate in an initial live remote training.
Experimental: Initial training plus video; School personnel will implement CATS or CPP and CICO with students in the school setting. They will participate in an initial live remote training and receive access to asynchronous video training modules about the EBPs (RV).	Behavioral: Coping Power Program (CPP); CPP is an evidence-based intervention designed for students with externalizing behavior disorder. CPP consists of twelve 45-minute sessions. This EBP has been found to be effective at reducing aggressive behavior, covert delinquent behavior and substance abuse among aggressive boys, with gains maintained at one-year follow-up. Growth curve analyses showed that CPP had linear effects for three years after intervention on reductions in aggressive behavior and academic behavior problems.; Behavioral: CBT for Anxiety Treatment in Schools (CATS); CATS is an adaptation of Friends for Life (FRIENDS). The adapted protocol retains the core elements of evidence-based CBT for anxiety and the FRIENDS group format. Investigators implemented planned adaptations to the protocol based on collective experience. Changes were made to the language, cultural methods, number of sessions, and activities while maintaining the 5 essential components of the treatment. This resulted in a briefer (8-session) and more feasible, engaging and culturally appropriate protocol for urban under-resourced schools than the original FRIENDS.; Behavioral: Check-in/Check-out (CI/CO); CI/CO is a targeted, Tier 2 intervention for students at risk of developing externalizing and internalizing mental health disorders. CI/CO is designed to provide immediate feedback (i.e., at the end of each class period) to students, based on the use of a daily report card. This feedback is developmentally sensitive. CI/CO implementers meet individually with students for a brief 'check-in' in the morning and a brief 'check-out' in the afternoon. Research on the use of CI/CO has shown it to be effective in reducing externalizing and internalizing problems with elementary school students. CI/CO will be offered to individual students for a three-month period of time, which is the same time-frame needed for the implementation of CPP.; Behavioral: Initial Training plus Video (RV); School personnel will participate in an initial live remote training and receive access to asynchronous video training modules.
Experimental: Initial training plus video, plus coaching; School personnel will implement CATS or CPP and CICO with students in the school setting. They will participate in an initial live remote training, receive access to asynchronous video training modules about the EBPs, and receive coaching support by study staff (RV+). The coaching will be from a study consultant regarding the implementation of EBPs.	Behavioral: Coping Power Program (CPP); CPP is an evidence-based intervention designed for students with externalizing behavior disorder. CPP consists of twelve 45-minute sessions. This EBP has been found to be effective at reducing aggressive behavior, covert delinquent behavior and substance abuse among aggressive boys, with gains maintained at one-year follow-up. Growth curve analyses showed that CPP had linear effects for three years after intervention on reductions in aggressive behavior and academic behavior problems.; Behavioral: CBT for Anxiety Treatment in Schools (CATS); CATS is an adaptation of Friends for Life (FRIENDS). The adapted protocol retains the core elements of evidence-based CBT for anxiety and the FRIENDS group format. Investigators implemented planned adaptations to the protocol based on collective experience. Changes were made to the language, cultural methods, number of sessions, and activities while maintaining the 5 essential components of the treatment. This resulted in a briefer (8-session) and more feasible, engaging and culturally appropriate protocol for urban under-resourced schools than the original FRIENDS.; Behavioral: Check-in/Check-out (CI/CO); CI/CO is a targeted, Tier 2 intervention for students at risk of developing externalizing and internalizing mental health disorders. CI/CO is designed to provide immediate feedback (i.e., at the end of each class period) to students, based on the use of a daily report card. This feedback is developmentally sensitive. CI/CO implementers meet individually with students for a brief 'check-in' in the morning and a brief 'check-out' in the afternoon. Research on the use of CI/CO has shown it to be effective in reducing externalizing and internalizing problems with elementary school students. CI/CO will be offered to individual students for a three-month period of time, which is the same time-frame needed for the implementation of CPP.; Behavioral: Initial Training plus Video plus Coaching (RV+); School personnel will participate in an initial live remote training, receive access to asynchronous video training modules, and receive coaching support from study staff.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Measures of content fidelity for group cognitive-behavioral treatment (GCBT)	The primary endpoints related to school staff implementing the interventions are measures of content fidelity. Content fidelity for the group interventions CPP and CATS will be measured using the Content Fidelity Checklists that reflects each activity component of the session agenda of the treatment protocols. Raters use a yes/no response scale to indicate whether or not the implementer covered a particular component.	Up to 3 years
Measures of content fidelity for Check-in, Check-Out	Content fidelity for CI/CO will be assessed via the Check-In/Check-Out Fidelity Checklist. The CI/CO Fidelity Checklist is completed by the consultant (research staff) on a monthly basis after collecting the completed Daily Progress Report (DPR) student forms for each student involved in CICO.	Up to 3 years
Measures of process fidelity	The primary endpoints related to school staff implementing the interventions are measures of content and process fidelity. Process fidelity for the group interventions CPP and CATS will be assessed via an adapted version of the rating system developed by Lochman and colleagues. 70 Ten items are rated on a scale of 0 to 4, with 0 being "Not At All" and 4 being "Very Often". Ratings are given on the extent to which school staff members delivered the intervention in an orderly fashion, using active learning strategies. etc. Exploratory factor analysis (EFA) yielded two factors on the 10-item scale.	Up to 3 years
Perceived adoption and penetration	Perceived adoption and penetration will be assessed via a survey completed by school staff. The adoption inventory will track the number of times each intervention is used per school, per condition. The penetration inventory will list EBP penetration at the student level (students receiving EBPs at Tier 2).	Up to 3 years
Changes in student mental health symptoms from Baseline to Post Intervention	Student Mental Health Symptoms will be measured by comparing parent scores pre-intervention and post-intervention. The Behavior Assessment System for Children-3rd edition (BASC-3) is a 138-item, 4-point, Likert-type rating scale (N=Never, S=Sometimes, O=Often, A=Almost Always) for assessing parental report of child mental health functioning, standardized for ages 2.5 to 18 years.	Baseline and 14 weeks (post intervention)
Changes in academic engagement from Baseline to Post Intervention	Student Academic Engagement will be measured by comparing teacher scores pre-intervention and post-intervention. This is a 20-item, four-point instrument (1 = Not At All True, 2 = Not Very True, 3 = Sort Of True, 4 = Very True) with four sub-scales. Investigators will use the average score for each of the four scales at pre- and post-participation in group cognitive behavioral therapy (CBT) or Check-in/Check-out (CICO).	Baseline and 14 weeks (post intervention)
Changes in externalizing and internalizing problems	Student externalizing and internalizing problems will be measure by comparing student scores pre-intervention and post-intervention. The Behavior and Feeling Survey-Youth Report (YFS-SR) is a brief (10 items) measure of internalizing and externalizing problems rated on a 5-point scale. The YFS-SR has excellent psychometric properties, and it is sensitive to change. The YFS-SR provides three scores (Internalizing, Externalizing, Total).	Baseline and 14 weeks (post intervention)

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Perceived mediators and moderators of consultation support	Secondary endpoints are the mediators and moderators effects of consultation support on behavioral health staff fidelity and the qualitative examination of school staff's perceived feasibility, appropriateness and acceptability of the support they received.	Up to 3 years

Collaborators and Investigators

• Sponsor: Children's Hospital of Philadelphia
• Collaborators: Agency for Healthcare Research and Quality (AHRQ); Devereux Center for Effective Schools
• Investigators: Principal Investigator: Ricardo B Eiraldi, PhD, Children's Hospital of Philadelphia

Publications and helpful links

General Publications

• Eiraldi R, McCurdy BL, Khanna MS, Goldstein J, Comly R, Francisco J, Rutherford LE, Wilson T, Henson K, Farmer T, Jawad AF. Development and evaluation of a remote training strategy for the implementation of mental health evidence-based practices in rural schools: pilot study protocol. Pilot Feasibility Stud. 2022 Jun 17;8(1):128. doi: 10.1186/s40814-022-01082-4.

Study record dates

Study Major Dates

• Study Start (Actual): September 20, 2022
• Primary Completion (Actual): June 13, 2025
• Study Completion (Actual): June 13, 2025

Study Registration Dates

• First Submitted: August 26, 2021
• First Submitted That Met QC Criteria: September 1, 2021
• First Posted (Actual): September 9, 2021

Study Record Updates

• Last Update Posted (Estimated): August 15, 2025
• Last Update Submitted That Met QC Criteria: August 14, 2025
• Last Verified: August 1, 2025

More Information

Terms related to this study

• Keywords: Mental Health; Evidence-based Practices; Positive-behavior Interventions and Supports; Tier 2; Rural Schools
• Additional Relevant MeSH Terms: Mental Disorders
• Other Study ID Numbers: 21-018863

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No