Evaluation of a Chatbot With CALMO (Chatbot-Assisted Learning Model) to Enhance Clinical Learning in Nursing Students (CALMO)

Development and Evaluation of a Chatbot With CALMO to Enhance Clinical Learning in Nursing Students: A Quasi-Experimental Mixed-Methods Study

The present proposed study will design and develop a chatbot to support graduating nursing students' clinical learning. The efficacy of this chatbot technology will be evaluated, using a mixed methods quasi-experimental design. the proposed study hypothesizes that the chatbot-assisted clinical education has the potential to promote clinical performance by improving nursing knowledge, general self-efficacy, and self-reflection and insight.

Study Overview

• Status: Not yet recruiting
• Conditions: Clinical Competence
• Intervention / Treatment: Behavioral: chatbot-assisted clinical learning

Detailed Description

The present proposed study will design and develop a chatbot to support graduating nursing students' clinical learning. The objectives of the proposed study are: (1) Design and develop a chatbot prototype that can simulate conversations, provide clinical information, and offer guidance relevant to various clinical scenarios; (2) Evaluate the impact of the chatbot intervention on nursing students' nursing knowledge, general self-efficacy, and self-reflection and insight; (3) Explore students' perceptions, satisfaction, and acceptance of the chatbot as an educational tool.The efficacy of this chatbot technology will be evaluated, using a mixed methods quasi-experimental design. the proposed study hypothesizes that the chatbot-assisted clinical education has the potential to promote clinical performance by improving nursing knowledge, general self-efficacy, and self-reflection and insight. The proposed study will be conducted in two phases. Phase 1: Design and develop a chatbot prototype which can provide evidence-based clinical information, engaging in interactive dialogue, and offering learning resources tailored to different clinical scenarios. The chatbot will be accessible through a web-based platform or mobile application. Phase 2: Conduct a mixed-methods quasi-experimental design and focus group interviews to evaluate the impact of the chatbot intervention on clinical learning. A convenience sample of 174 nursing students enrolled in a nursing program from two institutions will be recruited and assigned to either the experimental group, which will receive chatbot-supported clinical learning, or the control group, which will receive traditional clinical education without chatbot integration. Quantitative data will be collected from the two groups at three-time points: before-intervention at baseline (T0), 4 weeks after baseline (week 4, T1) and after intervention (week 12; T2) to assess students' nursing knowledge, general self-efficacy, and self-reflection and insight. Qualitative data will be collected through focus group interviews to explore students' perceptions, satisfaction, and experiences with chatbot-assisted learning. Quantitative data will be analyzed using MANOVA to compare the outcomes between and within groups. Thematic analysis will be conducted for qualitative data to identify emerging themes and patterns related to student perceptions and experiences using chatbot. Participants in the Experimental Group (EG) will receive an access code for the self-developed web-based chatbot platform called CALMO, a gender-neutral name as an acronym for Chatbot-Assisted Learning Model, which is planned to be developed during the first phase of this proposed study. The intervention will be available throughout the 12-week study.Participants in the Control Group (CG) will receive their usual clinical education support during the intervention period.A project management team will be formed to evaluate the intervention and ensure the quality and consistency of study intervention via regular face-to-face or online meetings. The fidelity of the study will be ensured by developing mechanisms for tracking participant logins, session completion rates, and engagement with intervention materials. Regular checks of the chatbot functionality will be performed by the platform developer with provision of regular service reports. Communication channels will be established for participants to provide feedback, report technical issues, or seek support from the research team during the intervention. Data will be analyzed using a mixed-methods approach, combining both quantitative and qualitative data. The statistical analysis using Statistical Package for the Social Sciences (SPSS), Version 26. The baseline characteristics of the participants in the two arms will be compared to identify any clinically meaningful differences using Chi-square test (for nominal data), Mann-Witney U test (for ordinal or non-normality data) or t-test (for normality data). Any variables considered to be significantly different will be further evaluated (e.g., included as covariates in the statistical analyses) and adjusted to account for their potential confounding effects on the outcomes. To test the hypothesis and to cater for the clustering effect with repeated measures of the outcome variables, a multivariate approach, such as a Generalized Estimating Equation (GEE), with adjustment of the baseline outcome variables to test for between-group differences will be employed. If students drop out between baseline and final data collection, intention to treat analysis will be applied. Both the Time x Group effect will be tested using a two-sided type I error rate of 5% for all tests of significance. Thematic analysis will be conducted for qualitative data to identify emerging themes and patterns related to students' perceptions and experiences.

• Study Type: Interventional
• Enrollment (Estimated): 174
• Phase: Not Applicable

Contacts and Locations

• Study Contact: Name: Winnie LS CHENG, PhD; Phone Number: 852-37024434; Email: wcheng@sfu.edu.hk

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Adult; Older Adult
• Accepts Healthy Volunteers: Yes

Description

Inclusion Criteria:

• Baccalaureate nursing students participating in their final clinical placement
• have sufficient knowledge of English language
• own a smartphone with internet access
• possess a valid Hong Kong phone number
• possess a valid email address

Exclusion Criteria:

• those who have been involved in the pilot test of the proposed chatbot intervention

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Supportive Care
• Allocation: Non-Randomized
• Interventional Model: Parallel Assignment
• Masking: Single

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Experimental: Chatbot assisted learning; The participants will recevie a self-developed web-based chatbot platform which is accessible via smartphones. It provides real-time dialogue with the chatbot through texts, visual aids, and audio files. The intervention will be available throughout the 12-week study. the chatbot is characterized by a gender-neutral persona, assumes the role of a friendly and empathetic clinical mentor. Its communication style can be described as calm, tolerant, supportive, and appreciative. Participants will receive weekly Short Message Service (SMS) notifications to remind them to consult with the chatbot.	Behavioral: chatbot-assisted clinical learning; The participants will recevie a self-developed web-based chatbot platform which is accessible via smartphones. It provides real-time dialogue with the chatbot through texts, visual aids, and audio files. The intervention will be available throughout the 12-week study. the chatbot is characterized by a gender-neutral persona, assumes the role of a friendly and empathetic clinical mentor. Its communication style can be described as calm, tolerant, supportive, and appreciative. Participants will receive weekly SMS notifications to remind them to consult with the chatbot.
No Intervention: control group _usual care; Participants in the Control Group will receive their usual clinical education support during the intervention period

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
nursing knowledge	multiple-choice questionnaire (MCQ) consisting of 50 items will be administered to evaluate students' nursing knowledge. the score ranges from 0 to 50. the higher the score, the better nursing knowledge.	before-intervention at baseline (T0), 4 weeks after baseline (week 4, T1) and after intervention (week 12; T2).
General self-efficacy	A 10-item General Self-Efficacy Scale to assess a general sense of perceived self-efficacy to manage hassles. It is scored with possible responses, 1=not at all true; 2=hardly true; 3=moderately true; and 4=exactly true, yielding a total score between 10 and 40. The higher the score, the higher the optimistic to cope with daily hassles.	before-intervention at baseline (T0), 4 weeks after baseline (week 4, T1) and after intervention (week 12; T2).

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Self-reflection and insight	A 12-item Self-Reflection and Insight Scale to assess the need for self-reflection and engagement in self-reflection. The scale is scored from 6 (highly matched) to 1 (least matched), yielding a total score between 12 and 72. Higher scores indicated greater self-reflection and insight.	: before-intervention at baseline (T0), 4 weeks after baseline (week 4, T1) and after intervention (week 12; T2).
Qualitative Comments on the Perceptions of Chatbot-assisted technology	it is a narrative feedback from the participants using focus group interviews to explore users' experiences with the chatbot during their clinical learning. A set of semi-structured interview questions encompassing questions to describe the overall experience, identify useful features, and assess the impact on understanding and retention of clinical concepts. Usability and user-friendliness of the chatbot interface will also be evaluated. For example: Can you describe your experience using the chatbot for clinical learning? What specific features or functionalities did you find useful or engaging? How do you think the chatbot has contributed to your understanding and retention of clinical concepts? Are there any limitations or challenges you encountered while using the chatbot? Did you perceive any differences in your learning outcomes when using the chatbot compared to other learning methods or resources? If so, what were those differences?	at the end of the 12-week study

Collaborators and Investigators

• Sponsor: Caritas Institute of Higher Education
• Collaborators: Tung Wah College
• Investigators: Principal Investigator: Winnie LS CHENG, PhD, Saint Francis University

Study record dates

Study Major Dates

• Study Start (Estimated): January 11, 2025
• Primary Completion (Estimated): September 30, 2026
• Study Completion (Estimated): December 30, 2026

Study Registration Dates

• First Submitted: January 11, 2025
• First Submitted That Met QC Criteria: January 15, 2025
• First Posted (Actual): March 25, 2025

Study Record Updates

• Last Update Posted (Actual): March 25, 2025
• Last Update Submitted That Met QC Criteria: January 15, 2025
• Last Verified: January 1, 2025

More Information

Terms related to this study

• Keywords: Nursing students; Chatbot; Focus group interview; Quasi-experimental design; Clinical learning
• Other Study ID Numbers: ISG230111

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO
• IPD Plan Description: I will not publish in Medical Journal Editors

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No