Increasing the Reach of Promising Dropout Prevention Programs: Examining the Trade-offs Between Scale and Effectiveness

The inability to consistently deliver at large scale promising education interventions is an important contributing cause to inequality in the U.S. The research team applies insights from price theory and field-based randomized controlled trials to examine the effect of implementing a promising academic skills development program at large scale before implementing at scale. The project is designed to provide evidence of direct scientific and policy value for attempts to scale up a specific intervention, but also stimulate a much more thorough investigation of social policy scale-up challenges by refining these methods and demonstrating their feasibility and value.

The research team examines the challenge of program scale up for a promising intervention studied in Chicago at medium scale in the past - SAGA tutoring. Past work has demonstrated that SAGA's intensive, individualized, during-the-school-day math tutoring can generate very large gains in academic outcomes in a short period, even among students who are many years behind grade level. This study will explicitly explore the extent to which there is a trade-off between effectiveness and scale for this intervention. By taking advantage of the power of random sampling, this study will also allow for observation of the program's effectiveness as if it were running at three-and-a-half times the proposed scale in a subset of the study population.

Study Overview

• Status: Unknown
• Conditions: Educational Achievement
• Intervention / Treatment: Other: SAGA Innovations; Other: Scale-up tutors; Other: Standard Tutors

Detailed Description

The University of Chicago Education Lab and Crime Lab New York research teams are carrying out a randomized controlled trial during the 2016-17 and 2017-18 academic years to build on previous collaborations with the Chicago Public Schools (CPS), the New York City Department of Education, and SAGA Innovations that have found that SAGA's intensive, individualized, during-the-school-day tutoring can generate very large gains in academic outcomes in a short period of time, even among students who are many years behind grade level. This research suggests the promise of this approach for improving the academic skills and educational attainment of disadvantaged youth, even once they have reached adolescence. However, to truly affect outcomes at the local and national level, SAGA would have to be rolled out on a much greater scale than researchers have been able to study in Chicago. Yet little is known about how to take promising interventions to scale. This study seeks to build the science of scale-up, by examining the extent to which this individualized tutoring program can be implemented at an even greater scale and by explicitly exploring the trade-offs between effectiveness and scale.

The SAGA Innovations program expands on the nationally recognized innovation of high- dosage, in-school-day tutoring developed in Match Education's charter school in Boston. The tutoring program meets as a scheduled course, Math Lab, once a day during the normal school day, and is provided in addition to a student's regular math class. Students work two-on-one (two students with one tutor) with the same full-time, professional tutor for the entirety of the school year. The content of the tutoring sessions is aligned with what students are learning in their regular math courses, but is also targeted to address individual gaps in math knowledge. Also following the original model developed by Match Education, SAGA tutors use frequent internal formative assessments of student progress to individualize instruction.

A previous randomized controlled trial conducted by the University of Chicago research team found that one year of this intervention, delivered in AY2013-14 in the Chicago Public Schools, generated between one and two extra years of academic growth in math, over and above what the normal U.S. high school student learns in one year (Cook et al., 2015; Reardon, 2011). The estimated effects for math achievement are on the order of 0.19 to 0.30 standard deviations, depending on the exact test and norming used. The intervention also improved student grades in math by 0.58 points on a 1-4 grade point scale, compared to a control mean of 1.77. These gains are particularly important because of the growing evidence on the importance of math specifically for short- and medium-term success in school, and for long-term life outcomes such as employment and earnings (Duncan et al., 2007).

This study aims to build upon the investigators' previous evaluations of the program, and will provide insight into the ability of this program to serve youth at a much larger scale. Specifically, this study aims to answer the following research questions:

1. What is the effect of implementing an evidence-based individualized tutoring program at larger scale?
2. What is the relationship between the effect of the program and the scale at which the program is implemented?

Implementation sites are divided into two sets: sites in Chicago at which students are randomized to receive tutoring (hereby referred to as "scale-up" schools), and sites in Chicago and New York City where principals have primary discretion to choose which students receive tutoring (hereby referred to as "returning schools").

In order to study research question #1, investigators will take advantage of the power of random sampling to study scale-up of this program without actually having to implement the program at a much larger scale. The first research question seeks to measure the average quality of the scaled up program in Chicago, in which researchers will utilize data from the scale-up schools. Students in both scale up and returning schools are both randomly assigned to tutors. However, students in the scale-up schools have two additional randomizations - (1) tutor applicant randomization and (2) treatment assignment randomization. For the first feature, the research team is having SAGA over-recruit tutors as though they were implementing at larger than the intended scale in the scale up schools. Investigators then randomly select one in three-and-a-half tutor applicants to continue through SAGA's standard hiring process, and positions at the scale-up schools are only filled by these randomly selected tutors. As students are randomly assigned to treatment in these scale-up schools, investigators will be able to measure program effects at about three-and-a-half times the scale at which the program is being implemented at these schools.

To measure treatment effects for research question #1, the research team will estimate both intent to treat (ITT) and treatment on the treated (TOT) impacts. Researchers will estimate the ITT effect as follows:

Y=B0 + B1T + B2X + E

where Y is the outcome of interest, T indicates students that are randomly assigned to be offered the chance to participate in the tutoring program, X is a set of baseline controls (which specifically includes randomization block, gender, age, learning disability, free/reduced lunch status, race, baseline grade level, GPA, number of As/Bs/Cs/Ds/Fs in the prior year, math and reading standardized test scores from the prior year, days absent from school, disciplinary incidents including suspensions and arrests, and a binary flag for students with missing GPA and attendance data), E is a random error term, and B0, B1, B2 are parameters to be estimated. The random assignment of T assures that under standard assumptions, Ordinary Least Squares (OLS) estimation yields an unbiased estimate of the ITT as the estimate of B1, or the effect of being offered participation in the SAGA tutoring program. As all students who were randomized into the program were paired with tutors that were randomized via the process described above, our ITT (and subsequently, TOT) effect will specifically measure the effect of the program when administered at three-and-a-half times the scale as it is currently being administered.

The ITT measures the effect of being offered the chance to participate. As students assigned to treatment are not required to participate in the program, the ITT may not measure the effect of participation. The research team will measure the effect of participation using random assignment of T as an instrument for participation. If all participants were randomly selected (i.e. if there are no control students who are allowed to participate in the tutoring program) this method calculates the effect of the treatment on the treated (TOT), or the effect of participating for the group of students who choose to participate.

To gain insight into research question #2 above, which seeks to determine the relationship between program scale and effects, tutors at all sites are ranked by SAGA leadership based on relative expected quality. The research team will then randomize student pairs to tutors in an effort to identify a tutor's effect on student outcomes. Using this methodology, researchers can study whether tutor ranking predicts the size of the program effects. As the research team assumes that the program would hire tutors in the order of their ranking depending on the number of tutor slots they needed to fill, this analysis will shed light on the relationship between scale and effectiveness.

To analyze the impact of being assigned to a tutor of a particular ranking (i.e. the ITT estimate of research question #2), investigators will run regression models that regress academic outcomes on tutor rank. Our main outcome of interest is math standardized test scores. Our primary analysis will model outcomes as a linear function of tutor rank. As a secondary exploratory analysis, we will estimate the shape of the relationship between outcomes and tutor rank using the following leave-one-out cross-validation exercise:

1. Estimate the relationship non-parametrically by running a regression at the student-level of the outcome (primary: math standardized test scores) on a full set of separate fixed effects for every tutor rank. Call the coefficient on the fixed effect of the rth ranked tutor, γ ̂_r.
2. For rank r=1,…,R: Estimate using the student-level data the relationship between the outcome and tutor rank as a polynomial of order p=0, 1, 2, …, 10 holding the students assigned to tutors ranked r out of the sample. Use the coefficients from the polynomial terms to predict γ ̂_r, and call that f ̂_p (r). Save the squared error for each polynomial: (f ̂_p (r)-γ ̂_r )^2.
3. Select the polynomial p that minimizes ∑_(r=1)^R(f ̂_p (r)-γ ̂_r )^2 .
4. Report the estimated relationship between the outcome and rank using the selected order of polynomial.

In addition to the functions of tutor rank, each regression will include block fixed effects that capture how student pairs were randomly assigned to tutors. The blocks include student groups within a classroom with shared special restrictions (e.g. having no restrictions, needing a Spanish-speaking tutor, or needing a tutor who is qualified for advanced mathematics courses). Other covariates in the model will be the same as those included in our ITT and TOT analysis for research question #1, noted above, to measure the effect of the program when administered at three-and-a-half times the scale as it is currently being administered.

As students switch tutors for a variety of reasons, researchers will also need to calculate the TOT estimate to look at the impact of being assigned to and actually working with a tutor of a particular ranking. To do so, investigators will use the rank of the randomly assigned tutor as an instrument for the weighted average tutor rank, where the weight placed on each tutor's rank is equal to the proportion of time (measured in days) the student spent with that tutor. Investigators will use daily attendance data to create this weighted average. This method will help investigators understand whether working with a higher-ranked tutor means that a student actually receives better quality instruction, or whether an additional relationship between tutor rank and actual quality exists.

We will use only observed outcomes for the analysis. If an outcome is missing for more than 5 percent of the sample, we will also report the treatment effect on whether or not the outcome is observed. When baseline covariates are missing, we will impute missing values with zero and include an indicator for missingness as an additional baseline control.

• Study Type: Interventional
• Enrollment (Anticipated): 6600
• Phase: Not Applicable

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Child; Adult; Older Adult
• Accepts Healthy Volunteers: No
• Genders Eligible for Study: All

Description

Inclusion Criteria:

• Chicago Public School and New York City Department of Education high schools students attending schools in low-income communities. Schools in the study are chosen in collaboration with the Chicago Public Schools and New York City Department of Education based on criteria such as dropout rate, test scores, scores on academic rating scale, etc.
• School administrators are enthusiastic about the program and agree to terms and conditions of the experimental design
• Male and female youth within these schools who are rising 9th and 10th graders in academic year (AY) 2016-17 and 2017-18
• Applicants who apply to be a tutor for SAGA Innovations

Exclusion Criteria:

• In Chicago (where the randomized controlled trial is being run), youth who have missed >60% of days during AY2015-16 or AY2016-17 (through March), and so would not be expected to show up in school enough during intervention years (AY2016-17 and AY2017-2018) to benefit from school-based programming
• In Chicago, youth who have failed >75% of classes during AY2015-16 and AY2016-17 (through March)
• In Chicago, youth who have designations for autism, "educable mentally handicapped," and/or traumatic brain injury

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Treatment
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: Single

Number of Arms

3

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
No Intervention: Control group; These youth will receive standard mathematics instruction and support (including possibly other tutoring interventions), but not the daily, intensive, during-the-school-day math tutoring provided by SAGA.
Experimental: Scale-up SAGA math tutoring; These youth will receive intensive, daily mathematics tutoring, and students will be paired with scale-up tutors. Tutors will be hired using the randomization process, and will be randomly assigned to youth.	Other: SAGA Innovations; An intensive math tutoring program; Other: Scale-up tutors; Tutors who have been selected for hire via the one in three-and-a-half randomization process
Experimental: Standard SAGA math tutoring; These youth will receive intensive, daily mathematics tutoring, and students will be paired with tutors hired via SAGA's standard process, which does not involve randomization. Tutors will be randomly assigned to youth.	Other: SAGA Innovations; An intensive math tutoring program; Other: Standard Tutors; Tutors who have been selected for hire via SAGA's standard hiring process, which does not entail randomization

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Difference in math achievement	Performance on math standardized achievement tests	1-year, 2-years, 3-years

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Difference in math course grades	Math course grades, obtained from Chicago Public Schools and New York City Department of Education administrative database	1-year, 2-years
Difference in absentee rate	Number of school absences, obtained from Chicago Public Schools and New York City Department of Education administrative database	1-year, 2-years, 3-years
Difference in index of schooling outcomes	Index of standardized (in Z-score form) outcomes for school persistence, absences, and course grades, obtained from Chicago Public Schools and New York City Department of Education administrative data	1-year, 2-years, 3-years
Difference in student misconduct	Number of school misconduct infractions, obtained from Chicago Public Schools and New York City Department of Education administrative database	1-year, 2-years, 3-years
Difference in total courses failed	Number of total school courses failed, obtained from Chicago Public Schools and New York City Department of Education administrative database	1-year, 2-years, 3-years
Difference in math courses failed	Number of math courses failed, obtained from Chicago Public Schools and New York City Department of Education administrative database	1-year, 2-years, 3-years
Difference in non-math courses grades	Non-math course grades, obtained from Chicago Public Schools and New York City Department of Education administrative database	1-year, 2-years, 3-years
Difference in non-math courses failures	Number of non-math courses failed, obtained from Chicago Public Schools and New York City Department of Education administrative database	1-year, 2-years, 3-years
Difference in school persistence	Measure from CPS and NYC DOE student records of school persistence (enrollment or graduation status by end of academic year)	1-year, 2-years, 3-years
Difference in violent crime arrests	Number of violent crime arrests, obtained from Chicago Police Department and Illinois State Police, New York City Police Department and New York State Police administrative databases (if available)	1-year, 2-years, 3-years
Difference in other arrests (property, drug, and other crimes)	Number of non-violent crime arrests, including property crimes, drug crimes, and other crimes, obtained from Chicago Police Department and Illinois State Police, New York City Police Department and New York State Police administrative databases (if available)	1-year, 2-years, 3-years
Difference in standardized test score achievement	Performance on additional sections of standardized tests (i.e. reading)	1-year, 2-years, 3-years
Difference in high school graduation rate	Difference in four-year and five-year high school graduation rates, obtained from Chicago Public Schools and New York City Department of Education administrative data	2-years, 3-years, 4-years
Difference in college enrollment rate	Difference in college enrollment data, obtained from Chicago Public Schools and New York City Department of Education administrative data	3-years, 4-years, 5-years, 6-years, 7-years, 8-years

Collaborators and Investigators

• Sponsor: University of Chicago
• Collaborators: Northwestern University; Chicago Public Schools; Abdul Latif Jameel Poverty Action Lab; William T. Grant Foundation; SAGA Innovations; Chicago Beyond; New York City Department of Education
• Investigators: Principal Investigator: Jonathan Guryan, PhD, Northwestern University; Principal Investigator: Kelly Hallberg, PhD, University of Chicago

Publications and helpful links

General Publications

• Cook P, Dodge K, Farkas G, Fryer RG, Guryan J, Ludwig J, Mayer S, Pollack H, Steinberg L. Not Too Late: Improving Academic Outcomes for Disadvantaged Youth. Northwestern Institute for Policy Research Working Paper, February 2015.
• Cook P, Dodge K, Farkas G, Fryer RG, Guryan J, Ludwig J, Mayer S, Pollack H, Steinberg L. The (Surprising) Efficacy of Academic and Behavioral Intervention with Disadvantaged Youth: Results from a Randomized Experiment in Chicago. Cambridge, MA: National Bureau of Economic Research, Working Paper No. 19862, 2014.
• Fryer RG. Injecting Charter School Best Practices into Traditional Public Schools: Evidence from Field Experiments. The Quarterly Journal of Economics 129(3): 1355-1407, 2014.

Study record dates

Study Major Dates

• Study Start: September 1, 2016
• Primary Completion (Actual): June 1, 2018
• Study Completion (Anticipated): January 1, 2021

Study Registration Dates

• First Submitted: August 31, 2016
• First Submitted That Met QC Criteria: August 31, 2016
• First Posted (Estimate): September 5, 2016

Study Record Updates

• Last Update Posted (Actual): April 24, 2020
• Last Update Submitted That Met QC Criteria: April 22, 2020
• Last Verified: April 1, 2020

More Information

Terms related to this study

• Keywords: Mathematics; Educational Achievement; Individualized Tutoring; Academic Mismatch; Achievement Gap
• Other Study ID Numbers: SBS IRB16-0346