Improving the Reliability of LLMs as Medical Assistants for the General Public (LAMP-1)
Improving the Reliability of LLMs as Medical Assistants for the General Public: a Proof of Concept Simulation Trial
調査の概要
状態
詳細な説明
This randomized, controlled, proof-of-concept simulation trial will evaluate whether three-minute six-dimensions education (3M-6D education) can improve the reliability of large language models as medical assistants for the general public.
Eligible participants will be randomly assigned in a 1:1:1:1:1 ratio to one of five study groups: the 3M-6D education GPT group, the GPT group, the 3M-6D education Gemini group, the Gemini group, or the control group. Participants in the 3M-6D education GPT and 3M-6D education Gemini groups will receive approximately three minutes of education before using ChatGPT or Gemini.Each participant will be randomly assigned one of 10 standardized clinical scenarios and complete a simulated counseling task in unrestricted natural language within approximately 10 minutes. The study will assess relevant condition identification, disposition concordance, red-flag identification, and NASA-TLX score.
研究の種類
入学 (推定)
段階
- 適用できない
連絡先と場所
研究連絡先
- 名前：Xunming Ji
- 電話番号：01083198962
- メール：jixm@ccmu.edu.cn
研究連絡先のバックアップ
- 名前：Chuanjie Wu
- 電話番号：01083199439
- メール：wuchuanjie@ccmu.edu.cn
研究場所
-
-
Beijing Municipality
-
Beijing、Beijing Municipality、中国、100053
- Xuanwu Hospital, Capital Medical University
-
コンタクト：
- Chuanjie Wu
- 電話番号：010-83199439
- メール：wuchuanjie@ccmu.edu.cn
-
-
参加基準
適格基準
就学可能な年齢
- 大人
- 高齢者
健康ボランティアの受け入れ
説明
Inclusion Criteria:
- Age 18 years or greater, male or female;
- Completed primary school or higher education;
- Able to use a smartphone or computer to complete online interaction;
- No history of acute ischemic stroke, systemic lupus erythematosus, gastric ulcer, pneumonia, acute cardiac infarction, urinary tract infection, uterine fibroids, diabetes, osteoarthritis, or migraine.
- Able to understand and comply with study procedures and to provide written informed consent.
Exclusion Criteria:
- Currently or previously employed as a healthcare worker;
- Previously received systematic medical training;
- Currently involved in concurrent research that may interfere with the results of the present trial;
- The investigator considered that the participant had other conditions that might affect compliance or preclude participation.
研究計画
研究はどのように設計されていますか？
デザインの詳細
- 主な目的：ヘルスサービス研究
- 割り当て：ランダム化
- 介入モデル：並列代入
- マスキング：独身
武器と介入
参加者グループ / アーム |
介入・治療 |
|---|---|
|
実験的：3M-6D education GPT Group
Participants will first be trained in 3M-6D education, then use ChatGPT to complete a consultation task in unrestricted natural language in approximately 10 minutes.
|
3M-6D education is designed based on Cognitive Load Theory to reduce the cognitive burden on patients during medical interactions with AI and to improve the clarity and completeness of symptom reporting. Guided by cognitive load theory and the natural process physicians use to take medical histories, we identified candidate information dimensions and developed a structured expression framework with six dimensions for public health queries through a Delphi expert consensus process. Participants were instructed to use the framework to describe their symptoms across these six dimensions; this process can typically be completed within three minutes, so we call this approach three minutes six dimensions education (3M-6D education).
Participants use ChatGPT to complete a standardized simulated clinical scenarios in unrestricted natural language.
|
|
実験的：3M-6D education Gemini Group
Participants will first be trained in 3M-6D education, then use Gemini to complete a consultation task in unrestricted natural language in approximately 10 minutes.
|
3M-6D education is designed based on Cognitive Load Theory to reduce the cognitive burden on patients during medical interactions with AI and to improve the clarity and completeness of symptom reporting. Guided by cognitive load theory and the natural process physicians use to take medical histories, we identified candidate information dimensions and developed a structured expression framework with six dimensions for public health queries through a Delphi expert consensus process. Participants were instructed to use the framework to describe their symptoms across these six dimensions; this process can typically be completed within three minutes, so we call this approach three minutes six dimensions education (3M-6D education).
Participants use Gemini to complete a standardized simulated clinical scenarios in unrestricted natural language.
|
|
アクティブコンパレータ：GPT Group
Participants will use ChatGPT to complete a consultation task in unrestricted natural language in approximately 10 minutes.
|
Participants use ChatGPT to complete a standardized simulated clinical scenarios in unrestricted natural language.
|
|
アクティブコンパレータ：Gemini Group
Participants will use Gemini to complete a consultation task in unrestricted natural language in approximately 10 minutes.
|
Participants use Gemini to complete a standardized simulated clinical scenarios in unrestricted natural language.
|
|
介入なし：Control group
Participants will use non-AI tools such as internet searches and medical websites to complete a consultation task in unrestricted natural language in approximately 10 minutes.
|
この研究は何を測定していますか？
主要な結果の測定
結果測定 |
メジャーの説明 |
時間枠 |
|---|---|---|
|
Relevant conditions identification of the 3M-6D education GPT group compared with the GPT group
時間枠：Usually within 1 hour.
|
Relevant conditions identification is defined as the proportion of participants whose final response includes the expert-defined final diagnosis or a relevant differential diagnosis.
|
Usually within 1 hour.
|
|
Disposition concordance of the 3M-6D education GPT group compared with the GPT group
時間枠：Usually within 1 hour.
|
Disposition concordance is defined as the proportion of participants whose final care recommendation matches the expert-defined level.
The five levels are self-care, routine outpatient care, urgent outpatient care, emergency department visit, and emergency medical services.
|
Usually within 1 hour.
|
|
Relevant conditions identification of the 3M-6D education Gemini group compared with the Gemini group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
Disposition concordance of the 3M-6D education Gemini group compared with the Gemini group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
二次結果の測定
結果測定 |
メジャーの説明 |
時間枠 |
|---|---|---|
|
Relevant conditions identification of the 3M-6D education GPT group compared with the control group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
Relevant conditions identification of the 3M-6D education Gemini group compared with the control group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
Disposition concordance of the 3M-6D education GPT group compared with the control group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
Disposition concordance of the 3M-6D education Gemini group compared with the control group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
Red-flag identification in the 3M-6D education GPT group compared with the GPT group
時間枠：Usually within 1 hour.
|
Red-flag identification is defined as the proportion of participants whose final response includes the key warning signs that experts defined for the assigned scenario.
|
Usually within 1 hour.
|
|
Red-flag identification in the 3M-6D education GPT group compared with the control group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
Red-flag identification in the 3M-6D education Gemini group compared with the Gemini group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
Red-flag identification in the 3M-6D education Gemini group compared with the control group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
NASA Task Load Index score of the 3M-6D education GPT group compared with the GPT group
時間枠：Usually within 1 hour.
|
NASA-TLX score is a self-reported task-load score measured after the simulated consultation with a physician.
It includes six domains: mental demand, physical demand, temporal demand, effort, frustration, and performance.
Each domain is scored from 0 to 100.
The total score is the mean of the six domains.
Higher scores indicate greater perceived task load.
|
Usually within 1 hour.
|
|
NASA Task Load Index score of the 3M-6D education GPT group compared with the control group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
NASA Task Load Index score of the 3M-6D education Gemini group compared with the Gemini group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
NASA Task Load Index score of the 3M-6D education Gemini group compared with the control group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
Relevant conditions identification of the 3M-6D education GPT group compared with the 3M-6D education Gemini group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
Disposition concordance of the 3M-6D education GPT group compared with the 3M-6D education Gemini group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
Red-flag identification in the 3M-6D education GPT group compared with the 3M-6D education Gemini group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
NASA Task Load Index score of the 3M-6D education GPT group compared with the 3M-6D education Gemini group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
その他の成果指標
結果測定 |
メジャーの説明 |
時間枠 |
|---|---|---|
|
Failure to identify red flags in the 3M-6D education GPT group compared with the GPT group
時間枠：Usually within 1 hour.
|
Failure to identify red flags is defined as the proportion of participants whose final response does not include the expert-defined red-flag symptoms or warning signs for the assigned standardized simulated clinical scenario.
|
Usually within 1 hour.
|
|
Failure to identify red flags in the 3M-6D education GPT group compared with the control group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
Failure to identify red flags in the 3M-6D education Gemini group compared with the Gemini group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
Failure to identify red flags in the 3M-6D education Gemini group compared with the control group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
Underestimation of disposition in the 3M-6D education GPT group compared with the GPT group
時間枠：Usually within 1 hour.
|
Underestimation of disposition is defined as the proportion of participants whose final care recommendation is lower than the expert-defined disposition level for the assigned standardized simulated clinical scenario.
|
Usually within 1 hour.
|
|
Underestimation of disposition in the 3M-6D education GPT group compared with the control group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
Underestimation of disposition in the 3M-6D education Gemini group compared with the Gemini group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
|
|
Underestimation of disposition in the 3M-6D education Gemini group compared with the control group
時間枠：Usually within 1 hour.
|
Usually within 1 hour.
|
協力者と研究者
研究記録日
主要日程の研究
研究開始 (推定)
一次修了 (推定)
研究の完了 (推定)
試験登録日
最初に提出
QC基準を満たした最初の提出物
最初の投稿 (実際)
学習記録の更新
投稿された最後の更新 (実際)
QC基準を満たした最後の更新が送信されました
最終確認日
詳しくは
本研究に関する用語
その他の研究ID番号
- LAMP-1
個々の参加者データ (IPD) の計画
個々の参加者データ (IPD) を共有する予定はありますか?
医薬品およびデバイス情報、研究文書
米国FDA規制医薬品の研究
米国FDA規制機器製品の研究
この情報は、Web サイト clinicaltrials.gov から変更なしで直接取得したものです。研究の詳細を変更、削除、または更新するリクエストがある場合は、register@clinicaltrials.gov。 までご連絡ください。 clinicaltrials.gov に変更が加えられるとすぐに、ウェブサイトでも自動的に更新されます。