Cette page a été traduite automatiquement et l'exactitude de la traduction n'est pas garantie. Veuillez vous référer au version anglaise pour un texte source.

Benchmarking Large Language Models Against Tumour Boards for Oncology Treatment Recommendations (BEACON)

28 juillet 2026 mis à jour par: Assistance Publique - Hôpitaux de Paris

Benchmarking AI for Clinical Oncology decisioNmaking (BEACON): A Prospective, Multicentre, Blinded Evaluation of Frontier Large Language Models Against Multidisciplinary Tumour Board Recommendations in Oncology Treatment Planning

BEACON (Benchmarking AI for Clinical Oncology decisioNmaking) is a prospective, multicentre, comparative, blinded, non-interventional benchmark evaluating the treatment recommendations of five frontier large language models (LLMs) against the recommendations of multidisciplinary tumour boards (RCP) in oncology treatment planning. One hundred standardised synthetic cases (20 per localisation, across breast, lung, urological, digestive and gynaecological cancers) are submitted as identical structured input to two independent tumour boards per localisation and to five frontier LLMs. Each recommendation - human or model - is decomposed into five predefined decision domains (intent, surgery, radiotherapy, systemic therapy, work-up and biomarkers) and scored 0/1/2 for concordance against a two-tier reference: the consensus of the two tumour boards, complemented by an a priori locked guideline matrix (ESMO, NCCN). The primary endpoint is domain-level concordance between LLM and RCP consensus, expressed as a linearly weighted Cohen's kappa. A co-primary safety endpoint captures the proportion of recommendations carrying serious harm potential, because concordance alone can conceal dangerous errors. Because expert boards may disagree with one another on identical cases, model performance is always interpreted against the human consensus. BEACON is designed as reusable, openly licensed, pre-registered infrastructure: all synthetic cases, evaluation rubrics, the locked guideline matrix, scoring algorithms and verbatim prompts are released for full reproducibility.

Aperçu de l'étude

Statut

Recrutement

Les conditions

Intervention / Traitement

Description détaillée

BEACON is a prospective, multicentre, blinded benchmark using automated, criteria-based scoring. It is built on three design decisions that distinguish it from the existing literature: (i) synthetic, standardised cases remove the record-completeness variability that confounds retrospective comparisons and allow the identical input to be given to every board and every model; (ii) two independent tumour boards per localisation let human-human agreement be measured rather than assumed; and (iii) a guideline matrix, locked a priori, provides an objective anchor applied identically to human and model recommendations.

Reference standard. For each case-domain, a guideline matrix (guideline-recommended / acceptable / unsupported options per case-domain; ESMO, NCCN), locked and time-stamped before data collection, is applied identically to boards and models.

Five decision domains. Every recommendation is decomposed into D1 Intent, D2 Surgery, D3 Radiotherapy, D4 Systemic therapy (class + line), and D5 Work-up & biomarkers before any comparison.

Type d'étude

Observationnel

Inscription (Estimé)

100

Contacts et emplacements

Cette section fournit les coordonnées de ceux qui mènent l'étude et des informations sur le lieu où cette étude est menée.

Coordonnées de l'étude

Sauvegarde des contacts de l'étude

Lieux d'étude

  • France
      • Paris, France
        • Recrutement
        • Hôpital Europeén Georges Pompidou
        • Contact:

Critères de participation

Les chercheurs recherchent des personnes qui correspondent à une certaine description, appelée critères d'éligibilité. Certains exemples de ces critères sont l'état de santé général d'une personne ou des traitements antérieurs.

Critère d'éligibilité

Âges éligibles pour étudier

  • Adulte
  • Adulte plus âgé

Accepte les volontaires sains

Non

Méthode d'échantillonnage

Échantillon non probabiliste

Population étudiée

100 synthetic oncology treatment-planning cases (20 per localisation) across five localisations: breast, lung, urological (prostate, bladder / upper-tract urothelial, kidney), digestive and gynaecological. Each case is a structured JSON input specifying UICC 8th-edition stage, biomarkers, ECOG performance status, comorbidities and a standardised clinical question. No human participants, no patient data and no identifiable individuals. Recommendations are produced by two independent tumour boards per localisation and by five frontier LLMs (queried May 2026).

La description

Inclusion Criteria:

  • Synthetic oncology case within one of the five predefined localisations (breast, lung, urological, digestive, gynaecological).
  • Complete structured schema: UICC 8th-edition stage, biomarkers, ECOG performance status, comorbidities and a standardised clinical question.
  • A clinically answerable treatment-planning question that is mappable to the locked guideline matrix.

Exclusion Criteria:

  • Case outside the five predefined localisations.
  • Incomplete, internally inconsistent or ambiguous schema.
  • Duplicate or near-duplicate of an existing case in the set.
  • Question not resolvable by current guidelines.

Plan d'étude

Cette section fournit des détails sur le plan d'étude, y compris la façon dont l'étude est conçue et ce que l'étude mesure.

Comment l'étude est-elle conçue ?

Détails de conception

Cohortes et interventions

Groupe / Cohorte
Intervention / Traitement
Breast cancers
Autre: Multidisciplinary tumour boards
Two independent tumour boards per localisation (10 boards in total) issue a categorical recommendation for every synthetic case. Where both boards agree, their consensus defines the reference standard; where they differ, the case-domain is classified as EQUIPOISE and analysed separately.
Autre: Frontier large language models
Five frontier LLMs (GPT-5.6, Claude Fable 5, Gemini 3.1 Pro, DeepSeek V4 Pro, Llama 4 Maverick) each receive the identical structured input for every case, three times in independent sessions, under locked prompts, versions and settings.
Lung cancers
Autre: Multidisciplinary tumour boards
Two independent tumour boards per localisation (10 boards in total) issue a categorical recommendation for every synthetic case. Where both boards agree, their consensus defines the reference standard; where they differ, the case-domain is classified as EQUIPOISE and analysed separately.
Autre: Frontier large language models
Five frontier LLMs (GPT-5.6, Claude Fable 5, Gemini 3.1 Pro, DeepSeek V4 Pro, Llama 4 Maverick) each receive the identical structured input for every case, three times in independent sessions, under locked prompts, versions and settings.
Urological cancers
Autre: Multidisciplinary tumour boards
Two independent tumour boards per localisation (10 boards in total) issue a categorical recommendation for every synthetic case. Where both boards agree, their consensus defines the reference standard; where they differ, the case-domain is classified as EQUIPOISE and analysed separately.
Autre: Frontier large language models
Five frontier LLMs (GPT-5.6, Claude Fable 5, Gemini 3.1 Pro, DeepSeek V4 Pro, Llama 4 Maverick) each receive the identical structured input for every case, three times in independent sessions, under locked prompts, versions and settings.
Digestive cancers
Autre: Multidisciplinary tumour boards
Two independent tumour boards per localisation (10 boards in total) issue a categorical recommendation for every synthetic case. Where both boards agree, their consensus defines the reference standard; where they differ, the case-domain is classified as EQUIPOISE and analysed separately.
Autre: Frontier large language models
Five frontier LLMs (GPT-5.6, Claude Fable 5, Gemini 3.1 Pro, DeepSeek V4 Pro, Llama 4 Maverick) each receive the identical structured input for every case, three times in independent sessions, under locked prompts, versions and settings.
Gynaecological cancers
Autre: Multidisciplinary tumour boards
Two independent tumour boards per localisation (10 boards in total) issue a categorical recommendation for every synthetic case. Where both boards agree, their consensus defines the reference standard; where they differ, the case-domain is classified as EQUIPOISE and analysed separately.
Autre: Frontier large language models
Five frontier LLMs (GPT-5.6, Claude Fable 5, Gemini 3.1 Pro, DeepSeek V4 Pro, Llama 4 Maverick) each receive the identical structured input for every case, three times in independent sessions, under locked prompts, versions and settings.

Que mesure l'étude ?

Principaux critères de jugement

Mesure des résultats
Description de la mesure
Délai
Domain-level performance between LLM recommendations and the locked guidelines.
Délai: Assessed once at central scoring, after data collection (~October 2026)
For each recommendation domain and each LLM, proportion of LLM recommendation concordant with locked guidelines
Assessed once at central scoring, after data collection (~October 2026)

Mesures de résultats secondaires

Mesure des résultats
Description de la mesure
Délai
Proportion of recommendations carrying serious harm potential ( LLM and tumour boards)
Délai: Up to October 2026
Up to October 2026
Domain-level recommendation concordance between LLM and tumour-boards
Délai: Up to October 2026
Each recommendation domain, decomposed into the five decision domains and scored per domain on an ordinal scale (2 = complete concordance; 1 = partial concordance; 0 = discordance).
Up to October 2026
Inter-tumour board domain-level recommendation concordance
Délai: Up to October 2026
Agreement between the two independent tumour boards scored per recommendation domain
Up to October 2026
Equipoise rate
Délai: Up to October 2026
Proportion of case-domains where the two tumour boards give different categorical recommendations
Up to October 2026
Completeness
Délai: Up to October 2026
Proportion of required domains addressed (LLM and tumour boards)
Up to October 2026
Missingness
Délai: Up to October 2026
Proportion of critical omissions (LLM and tumour boards)
Up to October 2026
Intensity bias
Délai: Up to October 2026
Proportion of recommendation corresponding to over- or under-treatment
Up to October 2026

Collaborateurs et enquêteurs

C'est ici que vous trouverez les personnes et les organisations impliquées dans cette étude.

Parrainer

Assistance Publique - Hôpitaux de Paris

Dates d'enregistrement des études

Ces dates suivent la progression des dossiers d'étude et des soumissions de résultats sommaires à ClinicalTrials.gov. Les dossiers d'étude et les résultats rapportés sont examinés par la Bibliothèque nationale de médecine (NLM) pour s'assurer qu'ils répondent à des normes de contrôle de qualité spécifiques avant d'être publiés sur le site Web public.

Dates principales de l'étude

Début de l'étude (Réel)

1 mai 2026

Achèvement primaire (Estimé)

1 octobre 2026

Achèvement de l'étude (Estimé)

1 octobre 2026

Dates d'inscription aux études

Première soumission

28 juillet 2026

Première soumission répondant aux critères de contrôle qualité

28 juillet 2026

Première publication (Réel)

31 juillet 2026

Mises à jour des dossiers d'étude

Dernière mise à jour publiée (Réel)

31 juillet 2026

Dernière mise à jour soumise répondant aux critères de contrôle qualité

28 juillet 2026

Dernière vérification

1 juillet 2026

Plus d'information

Termes liés à cette étude

Mots clés

Termes MeSH pertinents supplémentaires

Autres numéros d'identification d'étude

  • APHP261032

Plan pour les données individuelles des participants (IPD)

Prévoyez-vous de partager les données individuelles des participants (DPI) ?

INDÉCIS

Informations sur les médicaments et les dispositifs, documents d'étude

Étudie un produit pharmaceutique réglementé par la FDA américaine

Non

Étudie un produit d'appareil réglementé par la FDA américaine

Non

Ces informations ont été extraites directement du site Web clinicaltrials.gov sans aucune modification. Si vous avez des demandes de modification, de suppression ou de mise à jour des détails de votre étude, veuillez contacter register@clinicaltrials.gov. Dès qu'un changement est mis en œuvre sur clinicaltrials.gov, il sera également mis à jour automatiquement sur notre site Web .

S'abonner