Testing an AI Tool to Help Primary Care Clinicians With Specialty Consultation Questions (SAGE)
Physician Usability and Output Quality Evaluation of a Retrieval-Augmented Language Model System for Specialty Medical Consultation: A Human-Computer Interaction Study
The goal of this study is to test an artificial intelligence (AI) tool called SAGE. SAGE helps primary care doctors with questions that often need a specialist. Primary care doctors are the doctors people usually see first. SAGE reviews a case and suggests what a specialist might advise.
The main questions it aims to answer are:
- Do doctors make sound, timely care decisions when they use SAGE?
- Do those decisions match what a specialist would advise?
Researchers will compare the decisions doctors make with and without SAGE.
Doctors in the study will:
- Review made-up patient cases (these are not real patients)
- Make a decision for each case, their usual way and with SAGE
Studieoversigt
Status
Status
Betingelser
Betingelser
Intervention / Behandling
Intervention / Behandling
Undersøgelsestype
Undersøgelsestype
Tilmelding (Anslået)
Tilmelding
Fase
Fase
- Ikke anvendelig
Kontakter og lokationer
Studiesteder
-
-
California
-
Stanford, California, Forenede Stater, 94305
- Stanford University
-
Kontakt:
- Jonathan Chen, MD, PhD
- Telefonnummer: 650-725-3655
- E-mail: jonc101@stanford.edu
-
-
Deltagelseskriterier
Berettigelseskriterier
Berettigelseskriterier
Aldre berettiget til at studere
- Voksen
- Ældre voksen
Tager imod sunde frivillige
Beskrivelse
Inclusion Criteria:
- Physician practicing in primary care internal medicine and/or family medicine
- Attending physician, or resident physician at postgraduate year 2 (PGY-2) or higher
- Stanford-affiliated (faculty, staff, or trainee)
- Able to complete a single, approximately 60-minute remote (Zoom) session conducted in English
Exclusion Criteria:
- Not currently practicing in primary care internal medicine or family medicine
- Member of the study team or otherwise involved in the development of SAGE
- Unable to complete the remote study session
Studieplan
Hvordan er undersøgelsen tilrettelagt?
Design detaljer
- Primært formål: Sundhedstjenesteforskning
- Tildeling: N/A
- Interventionel model: Enkelt gruppeopgave
- Maskning: Ingen (Åben etiket)
Antal våben
Våben og indgreb
Deltagergruppe / ArmDeltagergruppe / Arm |
Intervention / BehandlingIntervention / Behandling |
|---|---|
|
Eksperimentel: Primary care physicians
Primary care physicians make management decisions on synthetic patient cases.
Each physician reviews four cases in random order: two with SAGE (AI clinical decision support) and two using a standard eConsult approach.
|
SAGE is an artificial intelligence (AI) based clinical decision support tool.
It generates specialty-informed recommendations for a primary care consultation question.
Physicians review the SAGE output while making a management decision for the synthetic case.
A standard electronic consultation (eConsult) format, without SAGE.
Physicians review this information while making a management decision for the synthetic case.
|
Hvad måler undersøgelsen?
Primære resultatmål
Primære resultatmål
Resultatmål |
Foranstaltningsbeskrivelse |
Tidsramme |
|---|---|---|
|
Clinical actionability of the clinician's decision
Tidsramme: Post-session independent review, through study completion (up to approximately 2 months)
|
For each case, independent reviewers judge whether the physician's decision is clinically actionable, meaning the physician can act on it now rather than wait for a specialist.
The outcome is the proportion of cases judged actionable, measured separately for decisions made with SAGE and for decisions made the standard way.
|
Post-session independent review, through study completion (up to approximately 2 months)
|
|
Concordance of the clinician's decision with the specialist reference answer
Tidsramme: Post-session independent review, through study completion (up to approximately 2 months)
|
Agreement between the physician's management decision and a specialist reference answer, rated by independent reviewers, with inter-rater agreement summarized.
Assessed for decisions made with SAGE and for decisions made the standard way.
|
Post-session independent review, through study completion (up to approximately 2 months)
|
Sekundære resultatmål
Sekundære resultatmål
Resultatmål |
Foranstaltningsbeskrivelse |
Tidsramme |
|---|---|---|
|
System usability (System Usability Scale, SUS)
Tidsramme: End of study session (Day 1, approximately 60 minutes)
|
Perceived usability of SAGE measured with the 10-item System Usability Scale (SUS), scored from 0 to 100.
|
End of study session (Day 1, approximately 60 minutes)
|
|
Order acceptance in the SAGE condition
Tidsramme: During study session (Day 1)
|
Proportion of SAGE-suggested orders that the clinician accepts, assessed in the SAGE condition.
|
During study session (Day 1)
|
|
Time to complete case review
Tidsramme: During study session (Day 1)
|
Time (in minutes) taken to review each case and reach a management decision, compared between the SAGE and standard eConsult conditions.
|
During study session (Day 1)
|
Samarbejdspartnere og efterforskere
Sponsor
Sponsor
Efterforskere
Efterforskere
- Ledende efterforsker: Jonathan Chen, MD, PhD, Stanford University
Datoer for undersøgelser
Studer store datoer
Studiestart (Anslået)
Studiestart
Primær færdiggørelse (Anslået)
Primær færdiggørelse
Studieafslutning (Anslået)
Studieafslutning
Datoer for studieregistrering
Først indsendt
Først indsendt
Først indsendt, der opfyldte QC-kriterier
Først indsendt, der opfyldte QC-kriterier
Først opslået (Faktiske)
Først opslået
Opdateringer af undersøgelsesjournaler
Sidste opdatering sendt (Faktiske)
Sidste opdatering sendt
Sidste opdatering indsendt, der opfyldte kvalitetskontrolkriterier
Sidste opdatering indsendt, der opfyldte kvalitetskontrolkriterier
Sidst verificeret
Sidst verificeret
Mere information
Begreber relateret til denne undersøgelse
Nøgleord
Andre undersøgelses-id-numre
Andre undersøgelses-id-numre
- 85430
Plan for individuelle deltagerdata (IPD)
Planlægger du at dele individuelle deltagerdata (IPD)?
Lægemiddel- og udstyrsoplysninger, undersøgelsesdokumenter
Studerer et amerikansk FDA-reguleret lægemiddelprodukt
Studerer et amerikansk FDA-reguleret enhedsprodukt
Disse oplysninger blev hentet direkte fra webstedet clinicaltrials.gov uden ændringer. Hvis du har nogen anmodninger om at ændre, fjerne eller opdatere dine undersøgelsesoplysninger, bedes du kontakte register@clinicaltrials.gov. Så snart en ændring er implementeret på clinicaltrials.gov, vil denne også blive opdateret automatisk på vores hjemmeside .