Teaching Medical Students to Verify AI-Generated Citation Hallucinations (SARA)

Effect of the SARA Citation-Verification Framework on Medical Students' Verification of AI-Generated Citation Hallucinations: A Randomized Controlled Trial With an Explanatory Sequential Mixed-Methods Component

Generative artificial intelligence (GenAI) tools are increasingly used by medical students for literature searching, scientific writing, and research. However, AI-generated citations may contain errors. Some references may not exist or may contain incorrect bibliographic details, while other references may be genuine but may not actually support the claim for which they are cited.

This randomized educational study will evaluate whether a structured citation-verification framework called SARA can improve medical students' ability to identify and correct these problems. SARA teaches students to assess four elements of a citation: Source authenticity, Accuracy, Relevance, and Alignment between the cited evidence and the claim being made.

Year 3-5 medical students will be randomly assigned to either a 75-minute SARA training session or an attention-matched session on responsible use of GenAI in academic work. Participants will complete citation-verification assessments before training, immediately after training, and 12 weeks later. The main outcome will assess whether students can independently apply the verification approach to an unstructured literature-review task 12 weeks after training, without being prompted to use SARA.

The study will also examine which types of citation errors remain difficult, whether students initiate appropriate verification, the quality of their verification, handling of valid citations, and confidence in their judgments. A subgroup of participants will take part in think-aloud and follow-up interviews to explore how they approach citation verification and transfer the strategy to new tasks.

Study Overview

• Status: Not yet recruiting
• Conditions: Medical Education; Generative Artificial Intelligence
• Intervention / Treatment: Behavioral: SARA Citation-Verification Training; Behavioral: Responsible GenAI Education Intervention Description

Detailed Description

AI-generated citation errors can occur at different levels. Some are source-level errors, such as fabricated references or incorrect bibliographic details. Others are more difficult to detect because the cited publication is genuine, but the evidence may be irrelevant, may not support the direction, magnitude, population, or conclusion of the claim, or may be presented without important qualifiers or limitations. In this study, these problems are examined across a five-level citation-error taxonomy ranging from non-existent sources to contextual omission or overgeneralization.

The SARA framework was developed as an educational scaffold to make citation-verification reasoning explicit. It prompts learners to ask whether the source exists, whether the citation details are accurate, whether the source is relevant to the specific claim, and whether the evidence is aligned with the claim as written. The intervention uses modelling, coached practice, feedback, articulation, reflection, and progressive removal of support so that learners move from guided verification toward independent judgment.

A key feature of the study is assessment of delayed transfer rather than immediate post-training performance alone. At 12 weeks, participants will review a realistic literature-review passage containing both valid and problematic citation-claim relationships without being told how many errors are present or being prompted to use SARA. This is intended to evaluate whether the verification strategy has become sufficiently internalized to be applied independently in a new scholarly context.

The explanatory qualitative component will explore why some learners successfully transfer verification skills while others continue to miss deeper evidence-claim problems. Think-aloud and stimulated-recall interviews will examine what triggers verification, how learners decide that evidence is sufficient, when they stop checking, and how they distinguish source existence from evidentiary support.

• Study Type: Interventional
• Enrollment (Estimated): 320
• Phase: Not Applicable

Contacts and Locations

• Study Contact: Name: Sara Reza; Phone Number: +92-333-4603388; Email: sara.qmc@qamc.edu.pk

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Child; Adult; Older Adult
• Accepts Healthy Volunteers: Yes
• Genders Eligible for Study: All

Description

Inclusion Criteria:

• Enrolled as an MBBS student in Year 3, Year 4, or Year 5 at Quaid-e-Azam Medical College, Bahawalpur.
• Has received basic curricular exposure to research methodology or scientific literature searching.
• Has not previously received formal structured training specifically in verification of AI-generated citations.
• Willing and able to provide written informed consent.
• Meets the age eligibility specified in the approved study protocol.

Exclusion Criteria:

• Previous participation in development, cognitive interviewing, piloting, or validation testing of the SARA assessment materials.
• Previous formal training in the SARA citation-verification framework before enrolment.
• Unwilling or unable to provide informed consent.

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Other
• Allocation: Randomized
• Interventional Model: Parallel Assignment
• Masking: Single

Number of Arms

2

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Experimental: SARA Citation-Verification Training; Participants assigned to this arm will receive a standardized 75-minute educational intervention using the SARA citation-verification framework: Source authenticity, Accuracy, Relevance, and Alignment. Training includes facilitator modelling and think-aloud demonstration, coached practice with feedback, use of a one-page verification scaffold, articulation and reflection, and progressive fading of prompts before independent practice. The intervention is designed to teach students to distinguish source-level citation errors from deeper evidence-claim errors and to apply structured verification independently.	Behavioral: SARA Citation-Verification Training; A standardized 75-minute educational intervention using the SARA framework: Source authenticity, Accuracy, Relevance, and Alignment. The session includes facilitator modelling and think-aloud demonstration, coached practice with feedback, a one-page verification scaffold, learner articulation and reflection, and progressive fading of prompts before independent practice. The intervention is designed to teach medical students to identify both source-level citation errors and deeper evidence-claim errors and to apply verification independently.
Active Comparator: Responsible GenAI Education Control; Participants assigned to this arm will receive an attention-matched 75-minute educational session on responsible use of generative artificial intelligence in academic work. The session covers privacy, disclosure, authorship, academic integrity, and general awareness that GenAI may produce errors. It does not teach the SARA framework or another structured citation-verification process. The session is matched to the intervention for duration, group size, facilitator interaction, and delivery format.	Behavioral: Responsible GenAI Education Intervention Description; A standardized 75-minute attention-matched educational session on responsible use of generative artificial intelligence in academic work. The session covers privacy, disclosure, authorship, academic integrity, and general awareness that GenAI may produce errors. It does not teach the SARA framework or another structured citation-verification process. The session is matched to the SARA intervention for duration, group size, facilitator interaction, and delivery format.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Independent Citation-Verification Performance on an Unstructured Scholarly Transfer Task	Participants will review a novel approximately 650-word AI-generated literature-review passage containing eight embedded citation-claim relationships. The passage includes both valid and erroneous citations spanning the study's five-level citation-error taxonomy. Participants will not be prompted to use the SARA framework and will not be told how many or what types of errors are present. Each citation-claim relationship will be scored against a prespecified gold standard for the appropriateness of the participant's final disposition. One point will be awarded for each appropriate disposition, producing a total score from 0 to 8, with higher scores indicating better independent citation-verification performance. Responses will be scored by outcome assessors blinded to intervention allocation.	12 weeks after the intervention

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Structured Citation-Verification Performance	Citation-verification performance will be measured using parallel 24-item structured assessment forms. Each item requires an appropriate final disposition of a citation-claim pair against a prespecified gold standard. One point will be awarded for each correct disposition, giving a total score from 0 to 24. Higher scores indicate better structured citation-verification performance.	Immediately after the intervention and 12 weeks after the intervention
Citation-Verification Accuracy by Citation-Error Condition	Accuracy will be assessed separately for valid/aligned citations and for Levels 1-5 of the prespecified citation-error taxonomy: non-existent citations, altered bibliographic details, irrelevant sources, evidence-claim misalignment, and contextual omission or overgeneralization. The outcome will be expressed as the proportion of citation-claim decisions with an appropriate final disposition within each condition.	Immediately after the intervention and 12 weeks after the intervention
Verification Initiation	Verification initiation will be measured as the proportion of citation-claim decisions for which the participant initiates an external verification action, such as consulting a bibliographic database, search engine, publisher source, abstract, or full text. Higher proportions indicate more frequent initiation of verification.	Immediately after the intervention and 12 weeks after the intervention
High-Quality Citation Verification	High-quality verification will be defined as examining the source element necessary to resolve the specific citation problem. Merely locating an article will not be considered adequate for evidence-claim errors if the participant does not assess whether the evidence supports the claim. The outcome will be expressed as the proportion of citation-claim decisions meeting the prespecified high-quality verification criterion.	12 weeks after the intervention
Unnecessary Alteration or Rejection of Valid Citations	Participants will encounter valid citation-claim relationships within the transfer task. The number of valid citations that are unnecessarily altered, rejected, replaced, or deleted will be recorded. The score ranges from 0 to 2, with lower values indicating more appropriate retention of valid evidence.	12 weeks after the intervention
Confidence Calibration in Citation-Verification Judgments	Participants will rate confidence in each final citation-verification judgment from 0% to 100%. Calibration error will be calculated by comparing confidence with correctness of the corresponding judgment. Lower absolute calibration error indicates closer correspondence between confidence and actual performance.	12 weeks after the intervention
Retention of Structured Citation-Verification Performance	Retention will be assessed using the 24-item structured verification score at 12 weeks and compared with baseline and immediate post-intervention performance. Scores range from 0 to 24, with higher scores indicating better retained structured citation-verification performance.	Baseline, immediately after the intervention, and 12 weeks after the intervention

Other Outcome Measures

Outcome Measure	Measure Description	Time Frame
Qualitative Exploration of Citation-Verification Strategies and Transfer Processes	A purposively selected subgroup of participants will complete think-aloud verification of novel citation-claim pairs followed by stimulated-recall interviews. Data will explore what triggers verification, how participants decide what evidence is sufficient, why verification stops, how relevance is distinguished from evidentiary support, how confidence changes, and whether SARA has become an internalized strategy or remains an explicit checklist. Interview transcripts will be analyzed using reflexive thematic analysis.	After completion of the 12-week assessment

Collaborators and Investigators

• Sponsor: Quaid-e-Azam Medical College

Study record dates

Study Major Dates

• Study Start (Estimated): September 26, 2026
• Primary Completion (Estimated): January 16, 2027
• Study Completion (Estimated): February 1, 2027

Study Registration Dates

• First Submitted: September 21, 2026
• First Submitted That Met QC Criteria: September 21, 2026
• First Posted (Actual): September 28, 2026

Study Record Updates

• Last Update Posted (Actual): September 28, 2026
• Last Update Submitted That Met QC Criteria: September 21, 2026
• Last Verified: September 1, 2026

More Information

Terms related to this study

• Keywords: Medical Students; Generative Artificial Intelligence; Citation Hallucination; Citation Verification; Research Integrity; Evidence Appraisal
• Other Study ID Numbers: QAMC-SARA-2026-01

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: YES

IPD Plan Description

De-identified individual participant quantitative data underlying the results reported in the primary publication will be made available. A data dictionary and relevant supporting documentation will also be provided. Qualitative interview transcripts will not be shared publicly because of the risk of participant re-identification.

IPD Sharing Time Frame

Beginning 6 months after publication of the primary study results and available for 5 years.

IPD Sharing Access Criteria

Data will be available to qualified researchers who submit a methodologically sound proposal for legitimate academic research. Requests will be reviewed by the study investigators and, where required, the institution. Access will be subject to applicable ethics, institutional, and data-use requirements.

• IPD Sharing Supporting Information Type: STUDY_PROTOCOL; SAP; ANALYTIC_CODE

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No