A cluster-randomized controlled trial of the effectiveness of the JUMP Math program of math instruction for improving elementary math achievement

Abstract

Students in many western countries struggle to achieve acceptable standards in numeracy despite its recognition as an important 21st century skill. As commercial math programs remain a staple of classroom instruction, investigations of their effectiveness are essential to inform decision-making regarding how to invest limited resources while maximizing student gains. We conducted a cluster randomized-controlled trial of the effectiveness of JUMP Math, a distinctive math program whose central tenets are empirically supported, for improving elementary math achievement (clinical trial.gov no. NCT02456181). The study involved 554 grade 2 (primary) and 592 grade 5 (junior) students and 193 teachers in 41 schools, in an urban-rural Canadian school board. Schools were randomly assigned to use either JUMP Math or their business-as-usual, problem-based approach to math instruction. We tracked student progress in math achievement on standardized and curriculum-based measures of computation and problem solving, for 2 consecutive school years. Junior students taught with JUMP Math made significantly greater progress in computation than their non-JUMP peers but the groups did not differ significantly in problem solving. Effects took hold relatively quickly, replicating the results from an earlier pilot study. Primary students in the non-JUMP group made significantly greater gains in problem solving and computation in year 1. But those taught with JUMP Math made significantly greater gains in problem solving and the groups did not differ in computation, in year 2. The positive effects of JUMP Math are noteworthy given that the JUMP Math teachers were likely still adjusting to the new program. That these positive findings were obtained in an effectiveness study (i.e. in real-world conditions), suggests that JUMP Math may be a valuable evidence-based addition to the teacher's toolbox. Given the importance of numeracy for 21st century functioning, identifying and implementing effective math instruction programs could have far-reaching, positive implications.

Conflict of interest statement

Bruce Ferguson and Tracy Solomon received funding from Jump Math for other projects and activities.

Figures

Fig 1. Pilot study results.

Results are based on standard scores and therefore indicate progress relative to same aged peers, which is represented by the 0 line. Vertical lines indicate 95% confidence limits around mean change scores. P-values and effect sizes (ES) are for the difference between the group means. Vertical lines that do not intersect the zero line indicate mean change that is significantly different from expected change, based on available norms.

Fig 2. CONSORT flow diagram of student participation in the Scale-Up RCT.

Primary students are shown on the left and junior students on the right side of the Fig. See main text for reasons for being lost to follow-up. Young/old for grade denotes students whose date of birth indicated they had started school either a year earlier or a year later than usual. Students were excluded from the analysis for a given time period if they did not have data for either the beginning or the end of that time period, which was determined separately for each outcome measure. The number of students excluded due to missing data shown in the Fig is based on the broad math outcome measure (see measures) but this number varied slightly across the different outcome measures.

Fig 3. Distribution of time spent on teacher activities and in different student configurations in observed classes.

Fig 4. Results for the Junior Students.

Broad Math (left side of panel a) is based on performance on applied problems, calculation and math fluency (shown separately in panel b). Vertical lines indicate 95% confidence limits around mean change scores. P-values and effect sizes (ES) in the Fig are for the difference between the group means. Results for the Woodcock-Johnson III measures are based on standard scores and therefore indicate progress relative to same aged peers, which is represented by the 0 line. Vertical lines that do not intersect the 0 line indicate mean change that is significantly different from expected change, based on test norms for the standardized measures (panels a and b) and 0 change for the supplementary measures (panel c).

Fig 5. Results for the primary students.

Broad Math (left side of panel a) is based on performance on applied problems, calculation and math fluency (shown separately in panel b). Vertical lines indicate 95% confidence limits around mean change scores. P-values and effect sizes (ES) in the Fig are for the difference between the group means. Results for the Woodcock-Johnson III measures are based on standard scores and therefore indicate progress relative to same aged peers, which is represented by the 0 line. Vertical lines that do not intersect the 0 line indicate mean change that is significantly different from expect change, based on test norms for the standardized measures (panels a and b) and 0 change for the supplementary measures (panel c).

Fig 6. Results for the high fidelity primary students.

Results for students who received high fidelity instruction. Broad Math (left side of panel a) is based on performance on applied problems, calculation and math fluency (shown separately in panel b). Vertical lines indicate 95% confidence limits around mean change scores. P-values and effect sizes in the Fig are for the difference between the group means. Results for the Woodcock-Johnson III measures are based on standard scores and therefore indicate progress relative to same aged peers, which is represented by the 0 line. Vertical lines that do not intersect the 0 line indicate mean change that is significantly different from expected change, based on test norms for the standardized measures (panels a and b) and 0 change for the supplementary measures (panel c).