Evaluating AI for the Translation of KM Resources

Background

Knowledge mobilization (KM) aims to ensure that research and evidence are effectively shared, applied and integrated into decision-making processes [1, 2]. KM resources serve different end-users (e.g., policymakers, healthcare providers, patients, caregivers) and can take various forms depending on their target audience (e.g., resources for patients include plain language summaries, videos, or infographics). By translating complex health information into user-friendly formats and evaluating dissemination strategies, this approach enhances awareness, engagement, and accessibility. In the context of child health, KM helps parents and caregivers access, understand, and use research findings to make informed healthcare decisions [3].

Our team has successfully co-created KM resources with families, integrating research, lived experience, and art to resonate with parents [4-10]. Our resources reflect diverse identities, including gender, family structure, ability, appearance, and socioeconomic status. The investigators have extensively evaluated accessible formats (e.g., video, interactive infographics) and effective dissemination strategies (e.g., social media) to maximize engagement [11-13]. Interactive infographics offer broad accessibility, easily shared across various platforms (e.g., websites, social media) to reach diverse audiences, regardless of geographic location (urban, rural, remote) or socioeconomic background (e.g., gender, race, culture, education, income) [14].

Communication challenges due to language barriers (specifically understanding or accessing healthcare information) have been shown to impact quality of care and lead to poor health outcomes [15]. While KM resources are meant to be widely accessible, they can be challenging for those whose first language is not English. According to the 2021 Canadian census, immigrants make up 23% of the population. This proportion is growing and is expected to reach 29-34% by 2041 [16, 17]. The proportion in Alberta is consistent with Canada overall at 23%, with the top places of birth of immigrants being the Philippines, India, and China. Therefore, the communities of interest in this proposal also represent common non-English languages spoken at home in Alberta: Mandarin, Punjabi, Tagalog, Urdu [18].

With increasing diversity in Alberta, the complexities associated with supporting groups with different ethnocultural backgrounds also increases. In situations where immigrants may face challenges in adjusting to new sociopolitical environments and health systems, KM in health is critical to address their knowledge and information needs [19, 20]. Translating KM resources into multiple languages can mitigate disparities in healthcare access and quality, ensuring that immigrants receive the information they need to navigate and engage with the health system effectively [15]. However, language translation of KM resources by a professional (i.e., human) translator can be resource intensive, costly, and time consuming. Artificial intelligence (AI) applications (e.g., ChatGPT, DeepL, Google Translate) have the potential for more efficient translation [21]. Yet, there may be disadvantages with using AI for translation. For instance, translations involve not only the literal meaning of words but also the cultural implications and social connotations attached to them [22, 23]. AI may lack the ability to recognize cultural subtleties, which could result in insensitive or inappropriate interpretations [22]. Nevertheless, as technology rapidly advances, AI is likely to be used for healthcare communications to enhance access and quality of care for those who are not proficient in English [24]. Empirical assessments of AI translation are therefore essential for understanding its capabilities and limitations.

Our previous work has demonstrated that parent-informed and parent-mediated KM resources can increase knowledge and influence decision-making and expectations regarding healthcare seeking; [25, 26] in turn, expectations can impact the care delivered by health professionals. A key tenet of KM is that resources are accessible to the intended end users. An important barrier to accessing KM resources is difficulty understanding the information which may arise due to language limitations. With nearly 25% of Alberta's population reporting an immigrant mother tongue, it is essential to assess the utility of English versus non-English resources to promote equitable access to research and support informed healthcare decisions. Evaluating professional versus AI translated resources will provide crucial information to support efficiencies in KM. This project has the potential for substantial impact in dissemination and uptake of research.

Objective

The study objective is to explore two key research questions (RQs) regarding the language translation of child health KM resources:

RQ1: How does language translation of a pediatric KM resource influence the understanding, accessibility, usability, satisfaction, and preferences of parents whose first language is not English?

RQ2: How do outcomes (understanding, usability, readability, and acceptability) compare between professional and AI translated versions?

Methods

Trial design: This study will use a multi-methods approach, combining both quantitative (non-pharmacological randomized controlled trial [RCT]) and qualitative (interview) data to provide a comprehensive understanding of our research questions [27]. RCTs are the most rigorous design to compare interventions and will provide estimates of differences between the versions of a KM resource. The qualitative interviews will help us gain a greater understanding of any observed differences and parent preferences.

Trial setting: The trial will be conducted entirely online within Canada. Participants will complete the study remotely using their personal electronic devices (e.g., computers, tablets) with internet access. All materials (study link, instructions, questionnaires, interviews) will be delivered electronically.

Patient and public involvement: The investigators will involve parents as partners in this work to help refine the study design, interpret results, and disseminate study findings. This will consist of seeking feedback from our Pediatric Parents' Advisory Group (P-PAG) that was established by our research team in 2016 [28].

Reporting guidelines: The study protocol is reported using the SPIRIT (Standard Protocol Items: Recommendations for Interventional Trials) [29]. The investigators will follow established guidance for reporting RCTs and qualitative studies: CONSORT (Consolidated Standards of Reporting Trials) [30] and SRQR (Standards for Reporting Qualitative Research) [31].

Recruitment: The recruitment approach will be through convenience sampling. The investigators will co-develop a recruitment strategy with our target end users through our established P-PAG. Parents will be recruited from the communities of interest using traditional and non-traditional approaches (e.g., newsletters, posters, list serv, social media) in partnership with our collaborators (Our Kids Health, Translating Emergency Knowledge for Kids).

Study screening, enrollment and consent: Parents will begin the screening and study enrollment process by clicking the study link (hosted by the REDCap platform) provided on the recruitment materials [32, 33]. After accessing the study link, parents will complete a short set of eligibility questions to self-identify as eligible to participate in the study, and review the study information letter. Parents can read the letter and will have the option to download a copy. They will then be stratified by language, randomized and directed within the same study link to complete the online study tasks (e.g., review KM resource[s], questionnaire). Implied consent by overt action will be sought for this portion of the study.

Participants will self-identify as being interested in participating in the optional one-on-one interview at the end of the questionnaire. Interested participants will then be contacted to arrange an interview. Verbal consent will be obtained on Zoom prior to starting the interview and before any data collection. This will be documented in the consent log.

Design and study flow: After enrollment (as described above), participants will be stratified by language (Mandarin, Punjabi, Tagalog, Urdu). Within each language stratum, they will be randomized (computer-generated sequence developed by a statistician) in a 2:1 ratio into one of two groups aligned with the study's research questions:

• RQ1 Group: English vs. Professional Translation
• RQ2 Group: Professional Translation vs. AI Translation

Within each RQ group, participants will undergo a second randomization (1:1 ratio, using a random number generator) to determine which version of the KM resource they receive.

RQ1 Group: Participants will be shown either the English or the professionally translated online KM resource in their language via REDCap. They will then be directed to complete a series of 16 questions that will be used to measure the outcomes: understandability, accessibility, usability, satisfaction. After this, they will be shown the alternate version (English or professionally translated) and asked a series of 3 questions about their preference, followed by a series of 10-12 demographic questions to describe the study population (age, education level, country of birth, time in Canada, including self-reported health literacy).

RQ2 Group: Participants will receive either the professionally translated or the AI-translated online KM resource in their language via REDCap. They will then be directed to complete a series of 20 questions that will be used to measure the outcomes: understandability, readability, acceptability, followed by a series of 10-12 demographic questions to describe the study population (age, education level, country of birth, time in Canada, including self-reported health literacy).

Reviewing the resources and completing the questionnaire is expected to take 10-20 minutes via REDCap, depending on whether participants are randomized to RQ1 or RQ2. Data will be collected and managed using the REDCap survey platform for this portion of the study.

Blinding: Participants will be blinded to the study hypothesis and to their group allocation. Team members involved in data collection and analysis, as well as the Investigators, will remain blinded during data interpretation and manuscript drafting. General group labels will be used until group allocation is revealed.

Intervention (KM resource): For the KM resource, the investigators selected an interactive infographic that was originally created in English with funding from Women and Children's Health Research Institute (WCHRI) and the Stollery Children's Hospital Foundation (SCHF). The topic was identified as a priority by local interest holders at the Stollery Children's Hospital. Further, the investigators have already been asked by the Multicultural Health Brokers Cooperative in Edmonton to translate this resource into Ukrainian and Russian due to interest from these communities (translations completed in 2024). The infographic provides information about what to expect when taking an ill or injured child to the emergency department (ED). The infographic contains 750 words and takes on average 3 minutes to read. For this study, the infographic will be professionally and AI-translated into Mandarin, Punjabi, Tagalog and Urdu. ChatGPT (the latest version available at time of translation) will be used, as it is widely used and easily accessible, increasing the generalizability of our results. ChatGPT has also demonstrated strong performance metrics, process efficiency, and cost-effectiveness [34].

Interviews (optional): Participants will be invited to take part in a one-on-one semi-structured online (optional, via Zoom, 30-60 minutes) qualitative interview with a member of the research team to provide deeper insights, exploring their experiences, perceptions, barriers, trust and language preference related to the resources (e.g., what about it being in their language was helpful or unhelpful) [27]. The interviews will occur concurrently with the online portion so that there is minimal time lapse between when the participant completes the questionnaire and participates in an interview. The investigators will purposively select for the interviews to ensure representation across languages and health literacy levels. Interviews will follow a semi-structured guide (8-10 questions) to collect additional information about the main quantitative variables and explore reasons for responses (25, 35). The interview questions will be tailored based on how participants were randomized and their questionnaire responses. Interviews will be conducted in English, professionally transcribed, de-identified, and imported into NVivo for analysis.

Quantitative analysis: Descriptive statistics (count, frequencies, means with standard deviations) will be used for demographic data. The investigators will determine overall understanding by calculating a score (ranging from 0 to 7) for each participant based on the number of questions answered correctly. The investigators will calculate a mean score for each group and compare groups (English vs. non-English and professional vs. AI translated) using a two-sample t-test assuming equal variances. Percentage of parents who preferred English vs. non-English versions will be compared using a chi-squared test. Other secondary outcomes will be analyzed based on differences in mean scores and two-sample t-tests. Statistical uncertainties will be expressed with 95% confidence intervals; p< 0.05 will indicate statistical significance. Analyses will be conducted based on intention-to-treat [36].

Quantitative sample size: The investigators aim to recruit 144 parents per language group across four language groups (total = 576). Participants will be randomized in a 2:1 ratio, with 376 allocated to RQ1 (English vs. non English) and 200 to RQ2 (professional vs. AI translated).

The sample size for RQ1 is based on our primary outcome of understanding. Assuming a correct response rate of 60% for the English version and 75% for the non English version, with α = 0.05 and β = 0.8, 240 participants (120 per arm) will be required to reject the null hypothesis of no difference in understanding between versions. To account for a 15% non completion rate, the investigators plan to recruit 376 participants, which corresponds to 94 participants from each of the four language groups.

The sample size for RQ2 follows guidance for exploratory and pilot studies, which do not require a formal sample size calculation [37, 38]. Based on previous research, the investigators plan to recruit 200 parents (50 per language group). This will provide estimates of potential differences while allowing us to explore variation across languages.

Qualitative analysis: Thematic content analysis will be used to analyze qualitative interview data. The process underpinning data analysis will be constant comparison, specifically pattern-matching and explanation building within, across, and between cases [39]. The phases of inductive analysis that will be employed are coding, categorizing, and developing themes [40].

Qualitative sample size: The investigators aim to conduct interviews with 12-15 parents who were randomized to each RQ (total n=24-30). This sample size is based on qualitative research methodology for one-on-one interviews, [41] and is often determined by data saturation which will be assessed concurrently throughout data collection and analysis to assess data comprehensiveness, variation, richness, and redundancy [42].

Expected outcomes: This study will assess how translation influences understanding and preferences for a child health KM resource among linguistically diverse parents. By exploring the question, 'Is translation enough?' the findings will help shape future KM resource development for culturally and linguistically diverse communities. While translation is a crucial step in making health information accessible, it may not be sufficient on its own to ensure effective communication, understanding, and uptake. If parents struggle to grasp key health concepts due to inadequate translation, their ability to make informed decisions about their child's care may be compromised. This innovative study will also provide empirical data on the use of AI to translate a child health KM resource. By evaluating translation methods, especially the role of AI, this study will provide insights into efficient approaches for scaling KM resources to increase accessibility and address health inequities.

Timelines: A realistic 24-month timeline has been set based on our experience and understanding of the required resources and activities (25,26,28). The first 6 months will involve: finalizing the study protocol and submitting for ethics approval; developing and refining with end user input the data collection instruments (questionnaires, interview guides) and recruitment strategy; and developing and pilot testing the online REDCap database. The investigators will allow 12 months for recruitment and data collection with quantitative and qualitative data collection occurring simultaneously. The final 6 months will involve data cleaning, analysis, interpretation, and report preparation.

Compensation: Participants will be offered a $5 CDN electronic gift card for completing the online questionnaire and a $25 CDN electronic gift card for completing an interview.

Trial monitoring: No on-site monitoring is planned, as all data collection and oversight will occur remotely. The research team will review REDCap data weekly and data will be downloaded regularly (e.g., weekly) for quality control, cleaning and analysis. Recruitment and completion rates will also be monitored to inform outreach/recruitment strategies.

Ancillary and Post-Trial Care: Although this trial involves minimal risk to participants, any concerns or adverse effects identified during the study will be addressed promptly. Participants will be provided with contact information for the study team to report any issues.

Dissemination: Our team plans to publish this trial in a high-impact peer-reviewed journal and present the results and national and international meetings.

Confidentiality: This study is designed to maintain participant confidentiality at all times. All data will be collected through a secure online platform (i.e., REDCap) and stored on password-protected systems accessible only to the research team. No identifying information will be collected unless a participant voluntarily provides it (e.g., to be contacted for an interview, to receive a gift card), in which case this information will be separated and stored separately from the questionnaire responses. All data on the secure online platform will be deleted upon study completion. De-identified data will be kept for a minimum of 5 years and stored securely on an encrypted password protected network drive at the University of Alberta.

Potential harms: This study involves minimal risk in which the magnitude of possible harms implied by participation is no greater than those encountered by participants in those aspects of their everyday life that relate to the research.