Project 3 Example: Human-AI Collaboration Tester (HAICT) Exp. 7

The study is one part of a "bundle" of experiments that constitute Project Three of a National Eye Institute grant. Project Three includes a series of experiments that investigate how changing the input from a simulated AI can affect the decisions made by human observers in a two-alternative forced choice task (like the decision to recall a woman for further examination in mammography). HAICT 7, the experiment described here, investigates how changing prevalence affects human performance when AI is used as a Second Reader.

Study Overview

• Status: Recruiting
• Conditions: Decision Making; Computer Aided Diagnosis
• Intervention / Treatment: Behavioral: Simulated Second Reader AI; Behavioral: Target Prevalence

Detailed Description

This text is the text of the pre-registration for the HAICT 7 experiment as described on the Open Science Framework. https://osf.io/hngu4/

NOTE: This study is representative of studies conducted in Project 3 of this grant. There are multiple experiments in the bundle of experiments represented by Project 3 but it is not possible to register a bundle of studies on CT.gov.

NOTE: Since the pronoun comment is advisory, we will leave it for now.

Human-AI Collaboration Tester (HAICT) Exp. 7 (lightly edited from OSF)

1. Data collection. Have any data been collected for this study already? (Yes/No)

   yes

2. Hypothesis. What's the main question being asked or hypothesis being tested in this study?

Background: In a variety of search experiments, both basic and clinical, the data have been consistent with a situation where the variability of the signal (or target) is greater than the variability of the noise (distractors). The classic sign of this is a zROC function with a slope < 1 - typically around 0.6. A slope of 1.0 is indicative of an equal variance 2AFC task. For the HAICT task that we have been testing, we would expect equal variance, but we think it would be worth checking so we will systematically vary prevalence which will shift criterion. That will sweep out an ROC curve that we can examine.

We will also test the Second Reader faux-AI in order to determine if low prevalence makes Second Reader worse.

• (H1): We expect to replicate the finding that human criteria become more conservative as prevalence declines.
• (H2): We predict that the slope of the resulting zROC will be 1.0.

• (H3): We hypothesize that low prevalence will make Second Reader AI less effective because the positive predictive value of its comments will be low.

  1. Dependent variable. Describe the key dependent variable(s) specifying how they will be measured.

     The main dependent variables of interest are accuracy (and the signal detection derivatives of accuracy, d' and c), reaction time, and subjective ratings on the survey following each block.

  2. Conditions. How many and which conditions will participants be assigned to?

This series of experiments investigates how changing the input from a simulated AI can affect the decisions made by human observers in a two-alternative forced choice task (like the decision to recall a woman for further examination in mammography). We have developed a paradigm called the Human-AI Collaboration Tester (HAICT) that allows for efficient testing of interactions between a human and a simulated AI.

The observers' task in all conditions is to give a 2AFC decision about whether a stimulus is "bad" or "not bad." To use language roughly mimicking a medical diagnosis, each stimulus is referred to as a "case." Observers are asked to make a 2AFC decision about arrays of colored shapes. The decision is made based on the predominant color of the case. The number of elements of each color are drawn from one of two normal distributions, one for positive (bad) stimuli and the other for negative (not bad) stimuli.

The results from previous HAICT experiments (3 and 4) showed that human performance in the Second Reader condition drops off significantly at low prevalence. Performance in the Second Reader condition was better than Baseline when the prevalence of bad cases was 50% but was significantly worse than Baseline when prevalence was only 10%. In this experiment, we manipulate the prevalence of "bad" cases in the Second Reader and Baseline conditions. Four different prevalence rates will be tested - 10%, 33%, 67%, and 90%. Observers will complete 8 blocks (2 AI rules x 4 prevalence rates), and block order is random.

AI rules to be tested:

1. Baseline - No AI input. Observer classifies each case as "bad" or "not" bad on their own.

2. Second Reader - The observer makes an initial decision about every case. The AI silently classifies stimuli using a conservative criterion (c = 0.5). The logic for the conservative criterion is that the second reader is being used to cut down on false positive responses and so it is intended to question positive human responses that might be marginal. If the observer and AI disagree, then the AI informs the human observer. The observer is then given a chance to either change their response or go with their first opinion.

   As in Experiments 1-5, the AI d-prime is fixed at 2.2. Feedback is known to increase the prevalence effect, so feedback will be given in both the practice and the test trials. Observers will complete 20 practice trials and 200 test trials in each block. Immediately after each block is completed, observers will be shown a summary of their performance. After the Second Reader blocks, they will also be asked to answer three subjective questions about the usefulness of the AI (see "Files" for more details).

3. Analyses. Specify exactly which analyses you will conduct to examine the main question/hypothesis.

   First, we summarize the number of hits, true negatives, misses, and false alarms in each block. From this, we can calculate the accuracy, the positive predictive value, sensitivity (d-prime), and the criterion for each observer under each of the different conditions. Given measures of performance at 4 levels of prevalence, we can estimate the ROC curve (pHit x pFA) and the zROC function (zHit x zFA). We will test the hypothesis that the slope of the zROC is equal to 1 (the consequence of an equal variance 2AFC task).

4. More analyses. Any secondary analyses?

   We will look to see if the observers' subjective opinions about the AI are correlated with variables such as the empirical d-prime, or the positive predictive value.

5. Sample size. How many observations will be collected or what will determine sample size? No need to justify decision, but be precise about exactly how the number will be determined.

   We will test 12 observers. This is consistent with the sample sizes of previous experiments.

6. Other. Is there anything else that you would like to pre-register? (e.g., data exclusions, variables collected for exploratory purposes, unusual analyses planned?)

N/A

• Study Type: Interventional
• Enrollment (Estimated): 15
• Phase: Not Applicable

Contacts and Locations

• Study Contact: Name: Jeremy M Wolfe, PhD; Phone Number: 6178511166; Email: jwolfe@bwh.harvard.edu

Study Locations

• United States
  • Massachusetts
    • Boston, Massachusetts, United States, 02215
      • Recruiting
      • Visual Attention Lab / Brigham and Women's Hospital
      • Contact: Jeremy M Wolfe; Phone Number: 617-851-1166; Email: jwolfe@bwh.harvard.edu

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 18 years and older (Adult, Older Adult)
• Accepts Healthy Volunteers: Yes

Description

Inclusion Criteria:

• - All welcome to enroll on line

Exclusion Criteria:

• Must pass the Ishihara color vision screening test
• 20/25 vision (with correction)

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Basic Science
• Allocation: N/A
• Interventional Model: Single Group Assignment
• Masking: None (Open Label)

Number of Arms

1

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Experimental: Experiment; All participants are tested in all conditions of this experiment.	Behavioral: Simulated Second Reader AI; In this experiment, in some conditions, the participant makes their decision in the presence of information about a simulated artificial intelligence decision.; Behavioral: Target Prevalence; The frequency with which targets are presented varies from 10% to 90%; Other Names: Base Rate

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
D'	D' (d-prime) is the signal detection theory measure of the level of performance on a task.	Up to one week
Criterion	Criterion is the signal detection theory measure of the bias ("liberal" or "conservative") of observers' decisions	Up to one week

Secondary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Reaction Time	This is the measure of how long it takes to make a response.	Up to one week

Collaborators and Investigators

• Sponsor: Brigham and Women's Hospital
• Investigators: Principal Investigator: Jeremy M Wolfe, PhD, Brigham and Women's Hospital

Study record dates

Study Major Dates

• Study Start (Actual): January 1, 2020
• Primary Completion (Estimated): August 1, 2024
• Study Completion (Estimated): January 1, 2025

Study Registration Dates

• First Submitted: February 18, 2022
• First Submitted That Met QC Criteria: February 28, 2022
• First Posted (Actual): March 9, 2022

Study Record Updates

• Last Update Posted (Actual): July 27, 2023
• Last Update Submitted That Met QC Criteria: July 25, 2023
• Last Verified: July 1, 2023

More Information

Terms related to this study

• Keywords: BESH; Signal detection; VIsual Perception
• Additional Relevant MeSH Terms: Pathologic Processes; Disease
• Other Study ID Numbers: 2007P000646-B

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: YES
• IPD Plan Description: De-identified raw data will be posted on the experiment's OSF page and will also be available on request to the PI.
• IPD Sharing Time Frame: Materials will be available when requested
• IPD Sharing Access Criteria: essentially unrestricted
• IPD Sharing Supporting Information Type: STUDY_PROTOCOL; SAP; ICF

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No