Scientific Validity Assessment and Optimization of AI-Generated A3/A4 Type Questions for the Chinese Medical Licensing Examination

Scientific Validity Assessment and Optimization of AI-Generated A3/A4 Type Questions for the Chinese Medical Licensing Examination: An Empirical Analysis Based on the "Answering-Generating" Closed Loop and Transformation of Competency Assessment

This is a cross-sectional study that primarily employs quantitative analysis, supplemented by qualitative assessment. The research is conducted in two stages: Phase I consists of a model performance comparison experiment, and Phase II involves an item quality evaluation experiment. The entire study adheres to the principles of single-blinding, randomization, and standardization to ensure scientific rigor and reproducibility.

The single-blind design is implemented during the "standardized testing" phase, where the system intersperses AI-generated items with those authored by human experts. Participants remain blinded to the source of each item (AI-generated vs. human-authored) throughout the testing and scoring processes, thereby ensuring the objectivity of the evaluation results.

Study Overview

• Status: Recruiting
• Conditions: AI (Artificial Intelligence)

• Study Type: Observational
• Enrollment (Estimated): 20

Contacts and Locations

• Study Contact: Name: Zhuoyi Chen MD; Phone Number: +86 18737552662; Email: 18737552662@163.com

Study Locations

• China
  • Guangdong
    • Guangzhou, Guangdong, China
      • Recruiting
      • Guangdong Provincial People's Hospital
      • Contact: Zhuoyi Chen MD; Phone Number: +86 18737552662; Email: 18737552662@163.com

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Adult
• Accepts Healthy Volunteers: Yes

• Sampling Method: Non-Probability Sample

Study Population

Medical students currently enrolled in a Standardized Residency Training (SRT) program.

Description

Inclusion Criteria:

• 1.Professional Status: Medical students currently enrolled in a Standardized Residency Training (SRT) program.

  2.Educational Background: Holders of a Bachelor of Medicine degree or higher, with foundational clinical knowledge.

  3.Informed Consent: Voluntarily participate in the study and provide written informed consent.

  4.Technical Competency: Proficient in using digital platforms to complete assessments and scoring.

Exclusion Criteria:

• 1.Conflict of Interest: Individuals involved in the AI model training, prompt engineering, or the creation of the human-authored question bank for this study.

  2.Inability to Complete: Presence of visual/auditory impairments or severe illness that precludes completion of the assessment within the specified time.

  3.Investigator's Discretion: Any other condition that, in the opinion of the investigator, renders the participant unsuitable for the study.

Study Plan

How is the study designed?

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Comparative Analysis of Item Accuracy	The investigators will evaluate the average accuracy rates coefficients of AI-generated items compared to human-generated items	Baseline

Collaborators and Investigators

• Sponsor: Guangdong Provincial People's Hospital

Study record dates

Study Major Dates

• Study Start (Actual): October 1, 2025
• Primary Completion (Estimated): March 30, 2026
• Study Completion (Estimated): March 30, 2026

Study Registration Dates

• First Submitted: March 22, 2026
• First Submitted That Met QC Criteria: March 29, 2026
• First Posted (Actual): April 1, 2026

Study Record Updates

• Last Update Posted (Actual): April 1, 2026
• Last Update Submitted That Met QC Criteria: March 29, 2026
• Last Verified: March 1, 2026

More Information

Terms related to this study

• Other Study ID Numbers: KY2025-1298-02

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No