Using Surveys to Examine the Association of Exposure to ML Mortality Risk Predictions With Medical Oncologists' Prognostic Accuracy and Decision-making

Nearly half of cancer patients in the US will receive care that is inconsistent with their wishes prior to death. Early advanced care planning (ACP) and palliative care improve goal-concordant care and symptoms and reduce unnecessary utilization. A promising strategy to increase ACP and palliative care is to identify patients at risk of mortality earlier in the disease course in order to target these services. Machine learning (ML) algorithms have been used in various industries, including medicine, to accurately predict risk of adverse outcomes and direct earlier resources. "Human-machine collaborations" - systems that leverage both ML and human intuition - have been shown to improve predictions and decision-making in various situations, but it is not known whether human-machine collaborations can improve prognostic accuracy and lead to greater and earlier ACP and palliative care. In this study, we contacted a national sample of medical oncologists and invited them complete a vignette-based survey. Our goal was to examine the association of exposure to ML mortality risk predictions with clinicians' prognostic accuracy and decision-making. We presented a series of six vignettes describing three clinical scenarios specific to a patient with advanced non-small cell lung cancer (aNSCLC) that differ by age, gender, performance status, smoking history, extent of disease, symptoms and molecular status. We will use these vignette-based surveys to examine the association of exposure to ML mortality risk predictions with medical oncologists' prognostic accuracy and decision-making.

Study Overview

• Status: Completed
• Conditions: Oncology
• Intervention / Treatment: Other: Survey

• Study Type: Observational
• Enrollment (Actual): 51

Contacts and Locations

Study Locations

• United States
  • Pennsylvania
    • Philadelphia, Pennsylvania, United States, 19104
      • Abramson Cancer Center of the University of Pennsylvania

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: Child; Adult; Older Adult
• Accepts Healthy Volunteers: No

• Sampling Method: Non-Probability Sample

Study Population

The study population consisted of a convenience sample of practicing medical oncologists who treated lung cancer in the US. We recruited medical oncologists through direct emails to the Principal Investigator's personal contacts (n=29); direct messages via Doximity, an online networking service for medical professionals (n=17), using a "thoracic oncology" ; and direct messages via X (formerly Twitter) (n=4). Efforts were taken to sample equally from 4 US geographic regions (Northeast, South, Midwest, West).

Description

Inclusion Criteria:

• Medical oncologists who treat lung cancer

Exclusion Criteria:

• Medical oncologists who do not see lung cancer patients

Study Plan

How is the study designed?

Number of groups / cohorts

6

Cohorts and Interventions

Group / Cohort	Intervention / Treatment
1A 2B 3C; 1. Intermediate; 1.A. Reference dependent; 2. Poor; 2.B. Absolute prognosis; 3. Good; 3.C. Both	Other: Survey; The study consisted of a 3 × 3 online factorial experiment employing a survey instrument hosted via Qualtrics presenting describing three patient vignettes. The three patient vignettes varied by various clinical characteristics including age, gender, performance status, smoking history, extent of disease, symptoms and molecular status. Each patient had advanced non-small cell lung cancer (aNSCLC). Each vignette had two parts: Part 1 described the case history for one of the three patients, after which prognostic estimates and medical decision-making was assessed (i.e. 1, 2, 3). Part 2 immediately followed and described the same vignette from the same patient with added information from a hypothetical ML predictive algorithm (i.e. A, B, C). The order of the vignettes in each survey was randomized with regard to presentation strategies for the ML risk predictions, so that there were 6 versions of the survey to which each participant was randomized.
1A 2C 3B; 1. Intermediate; 1.A. Reference dependent; 2. Poor; 2.C. Both; 3. Good; 3.B. Absolute prognosis	Other: Survey; The study consisted of a 3 × 3 online factorial experiment employing a survey instrument hosted via Qualtrics presenting describing three patient vignettes. The three patient vignettes varied by various clinical characteristics including age, gender, performance status, smoking history, extent of disease, symptoms and molecular status. Each patient had advanced non-small cell lung cancer (aNSCLC). Each vignette had two parts: Part 1 described the case history for one of the three patients, after which prognostic estimates and medical decision-making was assessed (i.e. 1, 2, 3). Part 2 immediately followed and described the same vignette from the same patient with added information from a hypothetical ML predictive algorithm (i.e. A, B, C). The order of the vignettes in each survey was randomized with regard to presentation strategies for the ML risk predictions, so that there were 6 versions of the survey to which each participant was randomized.
1B 2A 3C; 1. Intermediate; 1.B. Absolute; 2. Poor; 2.A. Reference dependent; 3. Good; 3.C. Both	Other: Survey; The study consisted of a 3 × 3 online factorial experiment employing a survey instrument hosted via Qualtrics presenting describing three patient vignettes. The three patient vignettes varied by various clinical characteristics including age, gender, performance status, smoking history, extent of disease, symptoms and molecular status. Each patient had advanced non-small cell lung cancer (aNSCLC). Each vignette had two parts: Part 1 described the case history for one of the three patients, after which prognostic estimates and medical decision-making was assessed (i.e. 1, 2, 3). Part 2 immediately followed and described the same vignette from the same patient with added information from a hypothetical ML predictive algorithm (i.e. A, B, C). The order of the vignettes in each survey was randomized with regard to presentation strategies for the ML risk predictions, so that there were 6 versions of the survey to which each participant was randomized.
1B 2C 3A; 1. Intermediate; 1.B. Absolute; 2. Poor; 2.C. Both; 3. Good; 3.A. Reference dependent	Other: Survey; The study consisted of a 3 × 3 online factorial experiment employing a survey instrument hosted via Qualtrics presenting describing three patient vignettes. The three patient vignettes varied by various clinical characteristics including age, gender, performance status, smoking history, extent of disease, symptoms and molecular status. Each patient had advanced non-small cell lung cancer (aNSCLC). Each vignette had two parts: Part 1 described the case history for one of the three patients, after which prognostic estimates and medical decision-making was assessed (i.e. 1, 2, 3). Part 2 immediately followed and described the same vignette from the same patient with added information from a hypothetical ML predictive algorithm (i.e. A, B, C). The order of the vignettes in each survey was randomized with regard to presentation strategies for the ML risk predictions, so that there were 6 versions of the survey to which each participant was randomized.
1C 2A 3B; 1. Intermediate; 1.C. Both; 2. Poor; 2.A. Reference dependent; 3. Good; 3.B. Absolute	Other: Survey; The study consisted of a 3 × 3 online factorial experiment employing a survey instrument hosted via Qualtrics presenting describing three patient vignettes. The three patient vignettes varied by various clinical characteristics including age, gender, performance status, smoking history, extent of disease, symptoms and molecular status. Each patient had advanced non-small cell lung cancer (aNSCLC). Each vignette had two parts: Part 1 described the case history for one of the three patients, after which prognostic estimates and medical decision-making was assessed (i.e. 1, 2, 3). Part 2 immediately followed and described the same vignette from the same patient with added information from a hypothetical ML predictive algorithm (i.e. A, B, C). The order of the vignettes in each survey was randomized with regard to presentation strategies for the ML risk predictions, so that there were 6 versions of the survey to which each participant was randomized.
1C 2B 3A; 1. Intermediate; 1.C. Both; 2. Poor; 2.B. Absolute; 3. Good; 3.A. Reference dependent	Other: Survey; The study consisted of a 3 × 3 online factorial experiment employing a survey instrument hosted via Qualtrics presenting describing three patient vignettes. The three patient vignettes varied by various clinical characteristics including age, gender, performance status, smoking history, extent of disease, symptoms and molecular status. Each patient had advanced non-small cell lung cancer (aNSCLC). Each vignette had two parts: Part 1 described the case history for one of the three patients, after which prognostic estimates and medical decision-making was assessed (i.e. 1, 2, 3). Part 2 immediately followed and described the same vignette from the same patient with added information from a hypothetical ML predictive algorithm (i.e. A, B, C). The order of the vignettes in each survey was randomized with regard to presentation strategies for the ML risk predictions, so that there were 6 versions of the survey to which each participant was randomized.

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Prognostic accuracy as assessed via survey	Prognostic estimates were measured using two items administered after Parts 1 and 2 of each of the 3 vignettes: What is your anticipated life expectancy for this patient, in months?; What do you think is the likelihood that she will die within 12 months? Please provide a percentage on a scale of 0% to 100%. Accurate prognoses were defined as whether the reported life expectancy estimate was within 33% of the LCPI estimate, as modified after the focus groups. Participants answered the first question in months and the second question as a percentage between 0-100%.	Up to 3 months

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Advance care planning decisions as assessed via survey	ACP decision-making was assessed using the following item administered after Parts 1 and 2 of each of the 3 vignettes: 1) Would you have a discussion about advance care planning at this point in her disease course? Each question was operationalized as a Yes/No answer and was followed by a free response box asking, "Please share your reason for this decision."	Up to 3 months
Palliative care referral as assessed via survey	Palliative care referral was assessed using the following item administered after Parts 1 and 2 of each of the 3 vignettes: 1) Would you refer him/her to a palliative care specialist at this point in her disease course? Each question was operationalized as a Yes/No answer and was followed by a free response box asking, "Please share your reason for this decision."	Up to 3 months

Collaborators and Investigators

• Sponsor: Abramson Cancer Center at Penn Medicine

Study record dates

Study Major Dates

• Study Start (Actual): March 14, 2023
• Primary Completion (Actual): June 14, 2023
• Study Completion (Actual): June 14, 2023

Study Registration Dates

• First Submitted: April 30, 2024
• First Submitted That Met QC Criteria: June 12, 2024
• First Posted (Actual): June 18, 2024

Study Record Updates

• Last Update Posted (Actual): June 18, 2024
• Last Update Submitted That Met QC Criteria: June 12, 2024
• Last Verified: June 1, 2024

More Information

Terms related to this study

• Additional Relevant MeSH Terms: Neoplasms
• Other Study ID Numbers: UPCC 10524; 850382 (Other Identifier: Penn IRB)

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: NO

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No