The Effect of Blend-learning in STEM in Young Children's Affective, Cognitive, and Academic Outcomes

The study is a randomised controlled trial that investigates the effectiveness of an online app-based STEM programme on improving Hong Kong primary school students' affective, cognitive, and academic outcomes.

Study Overview

• Status: Completed
• Conditions: Educational Problems
• Intervention / Treatment: Behavioral: STEM learning

Detailed Description

The Effect of Blend-learning in STEM in Young Children's Affective, Cognitive, and Academic Outcomes.

Background of the Study Traditionally, research in science education has predominantly focused on formal classroom learning. With the advancement of technology, Gilbert (2015) argues that formal science curriculum is limited in its elaboration and use of interesting scientific illustrations and equipment. Moreover, while this learning context facilitates knowledge acquisition, it may not promote life-long learning nor motivation to learn (Vedder-Weiss & Fortus, 2011). Motivation for science learning may be enhanced through informal learning contexts such as extracurricular science-related activities and programs (Fortus & Vedder-Weiss, 2014). In these contexts, experiences that focus on real-world problems-outside of school-may help in turn strengthen reasoning skills and understanding in science. The current study investigates the impact of blended learning approach in an informal learning environment on children's affective, cognitive, and academic outcomes in science. In particular, students' motivation, identity, reasoning, and performance in science are considered holistically.

Research objectives The current study aims to implement an app-based intervention program that focuses on STEM learning outside of a formal classroom environment. The overarching goal is to examine the efficacy of blended learning that utilizes a combination of an app and lab experiments in children's affective, cognitive, and academic outcomes.

Methods

Participants Participants will consist of 400 primary school students aged 8-10 years from 3 school in Hong Kong. The schools will be selected based on several criteria: 1) English Medium-of-Instruction (EMI) schools with 2) mixed-ability classes. The data will be collected during the summer holiday in 2021 and will be anonymized before processing and used in the analyses. The classes will be randomly assigned to the experimental condition or the control condition based on the random sampling method. Recruitment letter and advertisement poster will be published both online and offline through different platforms. Information sheet, consent form, and assent form will be distributed to the principals of the schools.

Design and Procedures Classes will be randomly assigned to the experimental group and to the control group. As treatment,the experimental group will be assigned to an intervention that uses a STEM blended learning app with gamified videos for content knowledge along with hands-on experiments. The videos are fictitious educational narratives related to the content knowledge and students were asked to "step into the shoes" of the cartoon scientist (a chipmunk) to solve various problems during the stories. The science experiments will be completed with step-by-step video guidance on the app. The control group will be assigned to read the same content on an e-book and receive an arts-and-craft kit for each lesson that is unrelated to the content knowledge. The content is designed to be curriculum aligned; both groups will learn about the same topics:

1. Electricity (Physics): 4 lessons on topics such as an introduction to electricity, simple and complex electrical circuits, renewable energy and static electricity.
2. Body Parts (Biology): 4 lessons on topics such as the muscoskeletal system, lungs, heart and brain
3. Environmental Science: 4 lessons on topics such as an introduction to the rainforest, classification, parts and stages of plant growth, photosynthesis and water transport in plants.

Participants in the experimental group will receive a 2.5-month intervention with the blended learning app while the control group receive e-book chapters during the same period. Five assessments, each of which will last for about 20-30 minutes, will be conducted throughout the study: (1) before the intervention (Time 1) and; (2) after the completion of the intervention (Time 2). Regardless of the group assignment, both groups will complete the pre-assessments a week before the intervention using the same set of assessments in the classroom. Moreover, both groups will receive hands-on materials during the intervention albeit the materials differ between groups. As part of the study, the experimental group will receive a science experimental kit for each of the lessons they watch on the app. Each experiment corresponds to the content knowledge they learn for a particular lesson. The control group will receive an arts-and-crafts kit for each chapter they read on the e-book. All students will engage in the same amount of time for each lesson (45 mins) for 12 lessons over the 2.5 months.

Measures

Demographics variables Students 'age, gender, and academic grades will be collected via teachers during Time 1.

Pre/post-tests Measures The study will consist of multi-measure self-reported questionnaires. It will include the following questionnaires that measure science motivation, science identify, science performance, and scientific reasoning.

Science motivation Science motivation will be assessed using the Expectancy-Value-Cost Scale based on the expectancyvalue model (Kosovich et al., 2015). The questionnaire was designed to measure students' motivation in STEM. It includes 10 items to assess 3 sub-domains: how successful students think they can perform (expectancy); how worthwhile science is (value) and; perceived effort in doing well (cost). Students reported their responses using a 6-point Likert scale ranged from 1 (strongly disagree) to 6 (strongly agree). Instructions are explained to students prior to the administration: "This is not a test. It is a short survey about how much you like your science class. You can respond openly and honestly. All of your responses will be kept confidential. No one will see your individual answers (not your teacher, not your classmates, not your parents)."

Science identity Science identity is arguably a multidimensional construct that involves many aspects and context. In this study, we will focus on informal settings outside of the classroom. That is, how students relate to science in general and in their everyday lives. A subset of 8 items were adapted from the science attitude scale in (Pell & Jarvis (2001) to assess science identity. The items in the questionnaire focus on real world science rather than science in the classroom context. The students reported the degree of their agreement with each item on a 6-point Likert scale (1= strongly disagree, 6= strongly agree).

Science performance Science performance will be measured using a combination of a real-time content knowledge quizzes designed as part of the app as well as a subset of items from the Trends in International Mathematics and Science Study (TIMSS). The content knowledge quizzes are designed to be interactive, immersive and real-time quizzes as part of the stories that students engage with in the app. The questions are constructed to be curriculum aligned and were reviewed by science teachers. Each topic has 40 questions; in total, there are 120 questions for the three topics covered in the app. The sum score is used to quantify science performance in the app. The items selected from TIMSS will be used as a second measurement of students' knowledge in science. A total of 12 multiple-choice items that are relevant to the topics will be selected.

Scientific reasoning Scientific reasoning will be assessed using the Science-P Reasoning Inventory (SPR-I) 7-item version designed for primary school students (Koerber & Osterhaus, 2019). The SPR-I is an inventory that is based on the conceptual model of scientific reasoning that relies on the understanding of hypothesisevidence relation in multiple non-exclusive levels (i.e., naïve, intermediate, and advanced). The 7 items include 3 items on nature of science, 3 items on experimentation, and 1 item on data interpretation. All items are scored using a partial credit scale (0,1,2), which represents the three levels of scientific reasoning. A score of 0 indicates a naïve understanding; a score of 1 indicates an intermediate understanding and; a score of 2 indicates an advanced level of understanding.

Analysis Descriptive and inferential statistical methods will be both employed in this study. A reliability analysis will be first conducted using scores obtained from the pre and post-assessments. Comparisons between the mean scores of the experimental and control group on all the outcome measures are then performed by evaluating the pre-intervention equivalency of the groups. A multivariate analysis of variance (MANOVA) will be used as the main analysis with science motivation, identity, achievement, and scientific reasoning scores as dependent variables and group assignment status as the independent variable. Assignment status had two levels, experimental and control.

• Study Type: Interventional
• Enrollment (Actual): 209
• Phase: Not Applicable

Contacts and Locations

Study Locations

• Hong Kong
  • Hong Kong, Hong Kong
    • University of Hong Kong

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 8 years to 10 years (CHILD)
• Accepts Healthy Volunteers: No
• Genders Eligible for Study: All

Description

Inclusion Criteria:

• 1) students from English Medium-of-Instruction (EMI) schools with
• 2) students from mixed-ability classes.

Exclusion Criteria:

• students with physical or motor disabilities that restrain them from completing the tasks

Study Plan

How is the study designed?

Design Details

• Primary Purpose: OTHER
• Allocation: RANDOMIZED
• Interventional Model: PARALLEL
• Masking: SINGLE

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
EXPERIMENTAL: App-based STEM online learning group; Participants in the experimental group will receive a 2.5-month intervention with the blended STEM learning app.	Behavioral: STEM learning; Classes will be randomly assigned to the experimental group and to the control group. As treatment, the experimental group will be assigned to an intervention that uses a STEM blended learning app with gamified videos for content knowledge along with hands-on experiments. The videos are fictitious educational narratives related to the content knowledge and students were asked to "step into the shoes" of the cartoon scientist (a chipmunk) to solve various problems during the stories. The science experiments will be completed with step-by-step video guidance on the app. The control group will be assigned to read the same content on an e-book and receive an arts-and-craft kit for each lesson that is unrelated to the content knowledge.
ACTIVE_COMPARATOR: E-book learning group; The control group receive e-book chapters about STEM for learning for the same period of 2.5-month as the experimental group.	Behavioral: STEM learning; Classes will be randomly assigned to the experimental group and to the control group. As treatment, the experimental group will be assigned to an intervention that uses a STEM blended learning app with gamified videos for content knowledge along with hands-on experiments. The videos are fictitious educational narratives related to the content knowledge and students were asked to "step into the shoes" of the cartoon scientist (a chipmunk) to solve various problems during the stories. The science experiments will be completed with step-by-step video guidance on the app. The control group will be assigned to read the same content on an e-book and receive an arts-and-craft kit for each lesson that is unrelated to the content knowledge.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Science motivation(questionnaire)	Science motivation was assessed using the Expectancy-Value-Cost Scale based on the expectancy-value model (Kosovich et al., 2014). The questionnaire was designed to measure students' motivation in STEM. It includes 10 items to assess 3 sub-domains: how successful students think they can perform (expectancy); how worthwhile science is (value) and; perceived effort in doing well (cost). Students reported their responses using a 6-point Likert scale ranged from 1 (strongly disagree) to 6 (strongly agree). Instructions are explained to students prior to the administration: "This is not a test. It is a short survey about how much you like your science class. You can respond openly and honestly. All of your responses will be kept confidential. No one will see your individual answers (not your teacher, not your classmates, not your parents)." A sum score is used to quantify science motivation; a higher score indicates a better outcome.	Pre-test: it will be assessed 2 weeks before the intervention.
Science motivation(questionnaire)	Science motivation was assessed using the Expectancy-Value-Cost Scale based on the expectancy-value model (Kosovich et al., 2014). The questionnaire was designed to measure students' motivation in STEM. It includes 10 items to assess 3 sub-domains: how successful students think they can perform (expectancy); how worthwhile science is (value) and; perceived effort in doing well (cost). Students reported their responses using a 6-point Likert scale ranged from 1 (strongly disagree) to 6 (strongly agree). Instructions are explained to students prior to the administration: "This is not a test. It is a short survey about how much you like your science class. You can respond openly and honestly. All of your responses will be kept confidential. No one will see your individual answers (not your teacher, not your classmates, not your parents)." A sum score is used to quantify science motivation; a higher score indicates a better outcome.	Post-test: it will be assessed within 2 weeks after the intervention.
Science identity (questionnaire)	Science identity is arguably a multidimensional construct that involves many aspects and context. In this study, we focus on informal settings outside of the classroom. That is, how students relate to science in general and in their everyday lives. A subset of 8 items were adapted from the Science Attitude Scale in Pell & Jarvis (2001) to assess science identity. The items in the questionnaire focus on real world science rather than science in the classroom context. The students reported the degree of their agreement with each item on a 6-point Likert scale (1= strongly disagree, 6= strongly agree). A sum score is used to quantify science identity; a higher score indicates a better outcome.	Pre-test: it will be assessed 2 weeks before the intervention.
Science identity (questionnaire)	Science identity is arguably a multidimensional construct that involves many aspects and context. In this study, we focus on informal settings outside of the classroom. That is, how students relate to science in general and in their everyday lives. A subset of 8 items were adapted from the Science Attitude Scale in Pell & Jarvis (2001) to assess science identity. The items in the questionnaire focus on real world science rather than science in the classroom context. The students reported the degree of their agreement with each item on a 6-point Likert scale (1= strongly disagree, 6= strongly agree). A sum score is used to quantify science identity; a higher score indicates a better outcome.	Post-test: it will be assessed within 2 weeks after the intervention.

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Science performance (standardized test)	Science performance was measured using a combination of a real-time content knowledge quizzes designed as part of the app as well as a subset of items from the Trends in International Mathematics and Science Study (TIMSS). Permission from the International Association for the Evaluation of Education Achievement was granted for the use of a total of 13 questions from the item bank of the year of 2019. A sum score is used to quantify science performance. A higher score indicates a better outcome. The content knowledge quizzes were designed to be interactive, immersive and real-time quizzes as part of the stories that students engage with in the app. The questions were constructed to be curriculum aligned and were reviewed by science teachers. Each topic has 40 questions; in total, there are 120 questions for the three topics covered in the app. The sum score is used to quantify science performance in the app; a higher score indicates a better outcome.	Pre-test: it will be assessed 2 weeks before the intervention.
Science performance (standardized test)	Science performance was measured using a combination of a real-time content knowledge quizzes designed as part of the app as well as a subset of items from the Trends in International Mathematics and Science Study (TIMSS). Permission from the International Association for the Evaluation of Education Achievement was granted for the use of a total of 13 questions from the item bank of the year of 2019. A sum score is used to quantify science performance. A higher score indicates a better outcome. The content knowledge quizzes were designed to be interactive, immersive and real-time quizzes as part of the stories that students engage with in the app. The questions were constructed to be curriculum aligned and were reviewed by science teachers. Each topic has 40 questions; in total, there are 120 questions for the three topics covered in the app. The sum score is used to quantify science performance in the app; a higher score indicates a better outcome.	Post-test: it will be assessed within 2 weeks after the intervention.
Scientific reasoning (performance task)	Scientific reasoning was assessed using the Science-P Reasoning Inventory (SPR-I) 7-item version designed for primary school students (Osterhaus et al., 2019). The SPR-I is an inventory that is based on the conceptual model of scientific reasoning that relies on the understanding of hypothesis-evidence relation in multiple non-exclusive levels (i.e., naïve, intermediate, and advanced). The 7 items include 3 items on nature of science, 3 items on experimentation, and 1 item on data interpretation. All items are scored using a partial credit scale (0,1,2), which represents the three levels of scientific reasoning. A score of 0 indicates a naïve understanding; a score of 1 indicates an intermediate understanding and; a score of 2 indicates an advanced level of understanding. Both the sum score and the partial credit scale score are used. A higher score indicates a better outcome.	Pre-test: it will be assessed 2 weeks before the intervention.
Scientific reasoning (performance task)	Scientific reasoning was assessed using the Science-P Reasoning Inventory (SPR-I) 7-item version designed for primary school students (Osterhaus et al., 2019). The SPR-I is an inventory that is based on the conceptual model of scientific reasoning that relies on the understanding of hypothesis-evidence relation in multiple non-exclusive levels (i.e., naïve, intermediate, and advanced). The 7 items include 3 items on nature of science, 3 items on experimentation, and 1 item on data interpretation. All items are scored using a partial credit scale (0,1,2), which represents the three levels of scientific reasoning. A score of 0 indicates a naïve understanding; a score of 1 indicates an intermediate understanding and; a score of 2 indicates an advanced level of understanding. Both the sum score and the partial credit scale score are used. A higher score indicates a better outcome.	Post-test: it will be assessed within 2 weeks after the intervention.

Collaborators and Investigators

• Sponsor: The University of Hong Kong
• Collaborators: University of Cambridge

Study record dates

Study Major Dates

• Study Start (ACTUAL): July 5, 2021
• Primary Completion (ACTUAL): December 31, 2021
• Study Completion (ACTUAL): December 31, 2021

Study Registration Dates

• First Submitted: July 22, 2021
• First Submitted That Met QC Criteria: September 28, 2021
• First Posted (ACTUAL): October 12, 2021

Study Record Updates

• Last Update Posted (ACTUAL): August 9, 2022
• Last Update Submitted That Met QC Criteria: August 8, 2022
• Last Verified: August 1, 2022

More Information

Terms related to this study

• Keywords: STEM, app-based, online learning
• Other Study ID Numbers: EA210315

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: UNDECIDED

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No