AI-Assisted Chest-CT Reporting for Enhanced Speed and Quality (The DOUBLE-ACE Study) (DOUBLE-ACE)

Here we provide a summary of the study's methodological framework, including a description of the AI system under evaluation, key quality control measures, and the data analysis plan. Comprehensive details regarding the full protocol, including eligibility criteria and outcome measures, can be found in the other modules of the study protocol.

1. Background on the AI System: The study evaluates a clinically deployed AI-assisted reporting system, built upon an advanced multimodal foundation model trained on a large-scale chest CT dataset. Its performance, reliability, and generalizability have been established through rigorous validation on extensive internal and external datasets. Prior reader studies have demonstrated its clinical utility by significantly reducing reporting time through automated draft generation while maintaining or improving report quality, supporting its integration into real-world workflow for this evaluation.
2. Quality Control: To ensure objective assessment, report quality will be scored by a panel of at least two independent, blinded thoracic radiologists using a standardized rubric, with inter-rater reliability calculated. A study-specific data dictionary and Standard Operating Procedures (SOPs) for data handling and analysis will be implemented to ensure reproducibility and auditability.
3. Data Analysis Plan: A comparative statistical analysis between pre- and post-implementation groups is planned. Appropriate statistical tests (e.g., Mann-Whitney U test, mixed-effects models) will be applied based on data distribution and variable type. A sample size calculation will be conducted to ensure the study is adequately powered to detect a clinically meaningful difference in the primary efficiency endpoint. The primary analysis will be a paired comparison of outcomes (e.g., report time, quality) between the two phases for the participants who complete both. To address potential attrition (e.g., radiologist turnover during the study year) and the influence of radiologist experience, the analysis plan includes: (1) Accounting for and reporting any participant dropout between phases. (2) Conducting pre-specified subgroup or stratified analyses based on radiologist seniority (e.g., junior vs. senior) to examine its effect on the outcomes.
4. Confounding Factor Control: To minimize potential bias, the study may identify collective variables (radiologists' sex, years of relevant professional experience, etc.) that may be considered potential confounding factors according to external experts' judgements. Certain patient-related information, such as diagnosis (infection, malignancy, cardiovascular disease, etc.) and clinical scenario (e.g., inpatient, outpatient, emergency), may also be collected retrospectively, where necessary, to evaluate model performance within specific diagnostic subgroups. The study will adopt multiple possible approaches for confounding factor adjustment or analysis, which may include stratified analyses and other related statistical methods.
5. Potential Adjustments in Study Protocol or Analysis Methods: As the study may involve multiple centers, in case of ethical or administrative restrictions at certain time at specific sites, AI assistance may be temporarily suspended to approximate the scenario without AI assistance at those sites. In such cases, the corresponding results may be reported as the with-AI and without-AI phases, rather than labeling them as baseline and AI-available phases. Furthermore, radiologists who decline to provide demographic or occupational information (e.g., years of professional experience or sex)-variables that may serve as potential confounders-will be excluded from adjusted and stratified analyses that require such covariates. These approaches may need to be incorporated into confounding factor or stratified analyses, and we may update related conditions accordingly when necessary.