Improving the Reliability of LLMs as Medical Assistants for the General Public (LAMP-1)
Improving the Reliability of LLMs as Medical Assistants for the General Public: a Proof of Concept Simulation Trial
Descripción general del estudio
Estado
Estado
Condiciones
Condiciones
Intervención / Tratamiento
Intervención / Tratamiento
Descripción detallada
This randomized, controlled, proof-of-concept simulation trial will evaluate whether three-minute six-dimensions education (3M-6D education) can improve the reliability of large language models as medical assistants for the general public.
Eligible participants will be randomly assigned in a 1:1:1:1:1 ratio to one of five study groups: the 3M-6D education GPT group, the GPT group, the 3M-6D education Gemini group, the Gemini group, or the control group. Participants in the 3M-6D education GPT and 3M-6D education Gemini groups will receive approximately three minutes of education before using ChatGPT or Gemini.Each participant will be randomly assigned one of 10 standardized clinical scenarios and complete a simulated counseling task in unrestricted natural language within approximately 10 minutes. The study will assess relevant condition identification, disposition concordance, red-flag identification, and NASA-TLX score.
Tipo de estudio
Tipo de estudio
Inscripción (Actual)
Inscripción
Fase
Fase
- No aplica
Contactos y Ubicaciones
Estudio Contacto
Estudio Contacto
- Nombre: Xunming Ji
- Número de teléfono: 01083198962
- Correo electrónico: jixm@ccmu.edu.cn
Copia de seguridad de contactos de estudio
- Nombre: Chuanjie Wu
- Número de teléfono: 01083199439
- Correo electrónico: wuchuanjie@ccmu.edu.cn
Ubicaciones de estudio
-
-
Beijing Municipality
-
Beijing, Beijing Municipality, Porcelana
- Beijing Ctiy
-
-
Criterios de participación
Criterio de elegibilidad
Criterio de elegibilidad
Edades elegibles para estudiar
- Adulto
- Adulto Mayor
Acepta Voluntarios Saludables
Descripción
Inclusion Criteria:
- Age 18 years or greater, male or female;
- Completed primary school or higher education;
- Able to use a smartphone or computer to complete online interaction;
- No history of acute ischemic stroke, systemic lupus erythematosus, gastric ulcer, pneumonia, acute cardiac infarction, urinary tract infection, uterine fibroids, diabetes, osteoarthritis, or migraine.
- Able to understand and comply with study procedures and to provide written informed consent.
Exclusion Criteria:
- Currently or previously employed as a healthcare worker;
- Previously received systematic medical training;
- Currently involved in concurrent research that may interfere with the results of the present trial;
- The investigator considered that the participant had other conditions that might affect compliance or preclude participation.
Plan de estudios
¿Cómo está diseñado el estudio?
Detalles de diseño
- Propósito principal: Investigación de servicios de salud
- Asignación: Aleatorizado
- Modelo Intervencionista: Asignación paralela
- Enmascaramiento: Único
Número de brazos
Armas e Intervenciones
Grupo de participantes/brazoGrupo de participantes/brazo |
Intervención / TratamientoIntervención / Tratamiento |
|---|---|
|
Experimental: 3M-6D education GPT Group
Participants will first be trained in 3M-6D education, then use ChatGPT to complete a consultation task in unrestricted natural language in approximately 10 minutes.
|
Participants use ChatGPT to complete a standardized simulated clinical scenarios in unrestricted natural language.
3M-6D education is designed based on Cognitive Load Theory to reduce the cognitive burden on patients during medical interactions with AI and to improve the clarity and completeness of symptom reporting. Guided by cognitive load theory and the natural process physicians use to take medical histories, the investigators identified candidate information dimensions and developed a structured expression framework with six dimensions for public health queries through a Delphi expert consensus process. Participants were instructed to use the framework to describe their symptoms across these six dimensions; this process can typically be completed within three minutes, so the investigators call this approach three minutes six dimensions education (3M-6D education). |
|
Experimental: 3M-6D education Gemini Group
Participants will first be trained in 3M-6D education, then use Gemini to complete a consultation task in unrestricted natural language in approximately 10 minutes.
|
Participants use Gemini to complete a standardized simulated clinical scenarios in unrestricted natural language.
3M-6D education is designed based on Cognitive Load Theory to reduce the cognitive burden on patients during medical interactions with AI and to improve the clarity and completeness of symptom reporting. Guided by cognitive load theory and the natural process physicians use to take medical histories, the investigators identified candidate information dimensions and developed a structured expression framework with six dimensions for public health queries through a Delphi expert consensus process. Participants were instructed to use the framework to describe their symptoms across these six dimensions; this process can typically be completed within three minutes, so the investigators call this approach three minutes six dimensions education (3M-6D education). |
|
Comparador activo: GPT Group
Participants will use ChatGPT to complete a consultation task in unrestricted natural language in approximately 10 minutes.
|
Participants use ChatGPT to complete a standardized simulated clinical scenarios in unrestricted natural language.
|
|
Comparador activo: Gemini Group
Participants will use Gemini to complete a consultation task in unrestricted natural language in approximately 10 minutes.
|
Participants use Gemini to complete a standardized simulated clinical scenarios in unrestricted natural language.
|
|
Sin intervención: Control group
Participants will use non-AI tools such as internet searches and medical websites to complete a consultation task in unrestricted natural language in approximately 10 minutes.
|
¿Qué mide el estudio?
Medidas de resultado primarias
Medidas de resultado primarias
Medida de resultado |
Medida Descripción |
Periodo de tiempo |
|---|---|---|
|
Relevant conditions identification of the 3M-6D education GPT group compared with the GPT group
Periodo de tiempo: 1 hour.
|
Relevant conditions identification is defined as the proportion of participants whose final response includes the expert-defined final diagnosis or a relevant differential diagnosis.
|
1 hour.
|
|
Disposition concordance of the 3M-6D education GPT group compared with the GPT group
Periodo de tiempo: 1 hour.
|
Disposition concordance is defined as the proportion of participants whose final care recommendation matches the expert-defined level.
The five levels are self-care, routine outpatient care, urgent outpatient care, emergency department visit, and emergency medical services.
|
1 hour.
|
|
Relevant conditions identification of the 3M-6D education Gemini group compared with the Gemini group
Periodo de tiempo: 1 hour.
|
1 hour.
|
|
|
Disposition concordance of the 3M-6D education Gemini group compared with the Gemini group
Periodo de tiempo: 1 hour.
|
1 hour.
|
Medidas de resultado secundarias
Medidas de resultado secundarias
Medida de resultado |
Medida Descripción |
Periodo de tiempo |
|---|---|---|
|
Relevant conditions identification of the 3M-6D education GPT group compared with the control group
Periodo de tiempo: 1 hour.
|
1 hour.
|
|
|
Relevant conditions identification of the 3M-6D education Gemini group compared with the control group
Periodo de tiempo: 1 hour.
|
1 hour.
|
|
|
Disposition concordance of the 3M-6D education GPT group compared with the control group
Periodo de tiempo: 1 hour.
|
1 hour.
|
|
|
Disposition concordance of the 3M-6D education Gemini group compared with the control group
Periodo de tiempo: 1 hour.
|
1 hour.
|
|
|
Red-flag identification in the 3M-6D education GPT group compared with the GPT group
Periodo de tiempo: 1 hour.
|
Red-flag identification is defined as the proportion of participants whose final response includes the key warning signs that experts defined for the assigned scenario.
|
1 hour.
|
|
Red-flag identification in the 3M-6D education GPT group compared with the control group
Periodo de tiempo: 1 hour.
|
1 hour.
|
|
|
Red-flag identification in the 3M-6D education Gemini group compared with the Gemini group
Periodo de tiempo: 1 hour.
|
1 hour.
|
|
|
Red-flag identification in the 3M-6D education Gemini group compared with the control group
Periodo de tiempo: 1 hour.
|
1 hour.
|
|
|
NASA Task Load Index score of the 3M-6D education GPT group compared with the GPT group
Periodo de tiempo: 1 hour.
|
NASA-TLX score is a self-reported task-load score measured after the simulated consultation with a physician.
It includes six domains: mental demand, physical demand, temporal demand, effort, frustration, and performance.
Each domain is scored from 0 to 100.
The total score is the mean of the six domains.
Higher scores indicate greater perceived task load.
|
1 hour.
|
|
NASA Task Load Index score of the 3M-6D education GPT group compared with the control group
Periodo de tiempo: 1 hour.
|
NASA-TLX score is a self-reported task-load score measured after the simulated consultation with a physician.
It includes six domains: mental demand, physical demand, temporal demand, effort, frustration, and performance.
Each domain is scored from 0 to 100.
The total score is the mean of the six domains.
Higher scores indicate greater perceived task load.
|
1 hour.
|
|
NASA Task Load Index score of the 3M-6D education Gemini group compared with the Gemini group
Periodo de tiempo: 1 hour.
|
NASA-TLX score is a self-reported task-load score measured after the simulated consultation with a physician.
It includes six domains: mental demand, physical demand, temporal demand, effort, frustration, and performance.
Each domain is scored from 0 to 100.
The total score is the mean of the six domains.
Higher scores indicate greater perceived task load.
|
1 hour.
|
|
NASA Task Load Index score of the 3M-6D education Gemini group compared with the control group
Periodo de tiempo: 1 hour.
|
NASA-TLX score is a self-reported task-load score measured after the simulated consultation with a physician.
It includes six domains: mental demand, physical demand, temporal demand, effort, frustration, and performance.
Each domain is scored from 0 to 100.
The total score is the mean of the six domains.
Higher scores indicate greater perceived task load.
|
1 hour.
|
|
Relevant conditions identification of the 3M-6D education GPT group compared with the 3M-6D education Gemini group
Periodo de tiempo: 1 hour.
|
1 hour.
|
|
|
Disposition concordance of the 3M-6D education GPT group compared with the 3M-6D education Gemini group
Periodo de tiempo: 1 hour.
|
1 hour.
|
|
|
Red-flag identification in the 3M-6D education GPT group compared with the 3M-6D education Gemini group
Periodo de tiempo: 1 hour.
|
1 hour.
|
|
|
NASA Task Load Index score of the 3M-6D education GPT group compared with the 3M-6D education Gemini group
Periodo de tiempo: 1 hour.
|
NASA-TLX score is a self-reported task-load score measured after the simulated consultation with a physician.
It includes six domains: mental demand, physical demand, temporal demand, effort, frustration, and performance.
Each domain is scored from 0 to 100.
The total score is the mean of the six domains.
Higher scores indicate greater perceived task load.
|
1 hour.
|
Otras medidas de resultado
Otras medidas de resultado
Medida de resultado |
Medida Descripción |
Periodo de tiempo |
|---|---|---|
|
Failure to identify red flags in the 3M-6D education GPT group compared with the GPT group
Periodo de tiempo: 1 hour.
|
Failure to identify red flags is defined as the proportion of participants whose final response does not include the expert-defined red-flag symptoms or warning signs for the assigned standardized simulated clinical scenario.
|
1 hour.
|
|
Failure to identify red flags in the 3M-6D education GPT group compared with the control group
Periodo de tiempo: 1 hour.
|
1 hour.
|
|
|
Failure to identify red flags in the 3M-6D education Gemini group compared with the Gemini group
Periodo de tiempo: 1 hour.
|
1 hour.
|
|
|
Failure to identify red flags in the 3M-6D education Gemini group compared with the control group
Periodo de tiempo: 1 hour.
|
1 hour.
|
|
|
Underestimation of disposition in the 3M-6D education GPT group compared with the GPT group
Periodo de tiempo: 1 hour.
|
Underestimation of disposition is defined as the proportion of participants whose final care recommendation is lower than the expert-defined disposition level for the assigned standardized simulated clinical scenario.
|
1 hour.
|
|
Underestimation of disposition in the 3M-6D education GPT group compared with the control group
Periodo de tiempo: 1 hour.
|
1 hour.
|
|
|
Underestimation of disposition in the 3M-6D education Gemini group compared with the Gemini group
Periodo de tiempo: 1 hour.
|
1 hour.
|
|
|
Underestimation of disposition in the 3M-6D education Gemini group compared with the control group
Periodo de tiempo: 1 hour.
|
1 hour.
|
Colaboradores e Investigadores
Patrocinador
Patrocinador
Colaboradores
Colaboradores
Fechas de registro del estudio
Fechas importantes del estudio
Inicio del estudio (Actual)
Inicio del estudio
Finalización primaria (Actual)
Finalización primaria
Finalización del estudio (Actual)
Finalización del estudio
Fechas de registro del estudio
Enviado por primera vez
Enviado por primera vez
Primero enviado que cumplió con los criterios de control de calidad
Primero enviado que cumplió con los criterios de control de calidad
Publicado por primera vez (Actual)
Publicado por primera vez
Actualizaciones de registros de estudio
Última actualización publicada (Actual)
Última actualización publicada
Última actualización enviada que cumplió con los criterios de control de calidad
Última actualización enviada que cumplió con los criterios de control de calidad
Última verificación
Última verificación
Más información
Términos relacionados con este estudio
Palabras clave
Otros números de identificación del estudio
Otros números de identificación del estudio
- LAMP-1
Plan de datos de participantes individuales (IPD)
¿Planea compartir datos de participantes individuales (IPD)?
Información sobre medicamentos y dispositivos, documentos del estudio
Estudia un producto farmacéutico regulado por la FDA de EE. UU.
Estudia un producto de dispositivo regulado por la FDA de EE. UU.
Esta información se obtuvo directamente del sitio web clinicaltrials.gov sin cambios. Si tiene alguna solicitud para cambiar, eliminar o actualizar los detalles de su estudio, comuníquese con register@clinicaltrials.gov. Tan pronto como se implemente un cambio en clinicaltrials.gov, también se actualizará automáticamente en nuestro sitio web. .