Brief Title: Project 4 Example: Mixed Hybrid Checklist

Project 4 Example: Mixed Hybrid Checklist

In "Mixed Hybrid Search", participants look for 3 specific target (e.g. this boot, this cat, this hat) and 3 categorical items (ANY fruit, ANY car, ANY game). In this task, participants tend to miss many categorical items. This is analogous to radiologists missing "incidental findings" when reading medical images. In this experiment, participants were given a checklist to help them to find categorical targets.

Study Overview

• Status: Completed
• Conditions: Incidental Findings; Visual Attention; Visual Search; Hybrid Search
• Intervention / Treatment: Behavioral: Stimulus Type; Behavioral: Target Presence; Behavioral: Checklist

Detailed Description

This experiment is one member of a 'bundle' of BESH experiments that were registered as part of the grant application process. It is not possible or meaningful to register a bundle of experiments in a single CT.gov filing so "Mixed Hybrid Checklist" is being registered as an example of this bundle.

This is the text of the pre-registration of Mixed Hybrid Checklist that was originally placed on the Open Science Framework: https://osf.io/j6h4s

NOTE: Since the pronoun comment is advisory, we will leave this for later. The website is, indeed, a functioning OSF page.

========

What's the main question being asked or hypothesis being tested in this study?

In real world search (and, notably, in radiology), it can be important to keep "an eye out" for items of interest that are not the primary subject of the search. In radiology these are known as "incidental findings," and they are a significant issue because clinicians are expected to detect items of clinical importance even they are answering a different question (e.g. you need to detect pneumonia even if your task was to screen for lung cancer). We have developed a "model system," in the form of a 'mixed hybrid search' task that can be used to investigate the fundamental cognitive processes that lie behind incidental finding errors. In this mixed hybrid search task, observers search for any one of multiple specific targets (e.g., this rabbit, this truck, and this spoon) and multiple categorical targets (e.g. any masks, furniture, and plants). In the all of the 'mixed hybrid search' experiments conducted thus far, the categorical targets are missed at a much higher rate than the specific targets. The categorical targets, thus, become the analog of the incidental error and our goal is to reduce those errors in our model system in the hope that this will provide guidance as to how to reduce errors in real-world tasks.

We have tested two strategies intended to reduce the number of missed categorical targets. In Experiment 5, we reminded observers about the categorical targets via non-search trials in which categorical targets were clearly marked. In Experiment 6, we forced observers to respond twice on each trial: Once to confirm the presence or absence of specific targets, and once to confirm the presence or absence of categorical targets. Both of these interventions failed.

This 'Hybrid Checklist' experiment is another attempt to develop a strategy that helps us reduce these categorical errors. Like the previous experiments, observers begin by memorizing three specific targets and three categorical targets. After viewing the targets, observers must pass an old/new recognition test with 100% correct responses to move to the search phase. Each memory test consists of 2×N items where N is the memory set size, and 50% of the items are old (targets) and 50% are new. Having passed the memory test, observers search for their targets in displays with 4, 8, or 12 items. Observers must indicate whether each target in the memory set is present or absent in the current display by clicking with a mouse. There are 6 targets (3 specific, 3 categorical), so observers make 6 responses on each trial. Error and reaction time will be recorded for each click, and for each trial.

Hypothesis (H1):

- We hypothesize that this checklist procedure will improve accuracy and reduce miss errors. However, given that there are 6 responses on each trial, the time per trial (or time per case for radiologists) is likely to increase significantly.

Describe the key dependent variable(s) specifying how they will be measured.

Miss errors and reaction times are the key dependent variables. These will be averaged across trials of the same condition (specific target, categorical target, and absent trials) and visual set size (4, 8, 12). Note that these measures will be compared to results from previous experiments rather than a within-observer design.

How many and which conditions will participants be assigned to?

In this experiment, participants are assigned to a single, 'checklist' condition. All observers memorize a set of three specific and three categorical targets. They are then required to pass a memory test before moving on to perform 20 practice trials and 300 experimental trials of the visual search task. Of these 300 trials:

70% are target-present trials 45% of target-present trials will have one target

• 80% of one-target trials have one specific target
• 20% of one-target trials have one categorical target 25% of target-present trials will have two targets
• 65% of two-target trials have two specific targets
• 5% of two-target trials have two categorical targets
• 30% of two-target trials have one specific and one categorical target The remaining 30% of trials will have no targets

Specify exactly which analyses you will conduct to examine the main question/hypothesis.

Reaction time and error rate will be compared to those in the previous mixed hybrid search experiment - a between-subjects design (Wolfe, Soce, and Schill, 2017). Analyses will be performed on both reaction time and miss error rate. For reaction time, a repeated measures ANOVA will be run with two factors: condition (specific target, categorical target, and absent) and visual set size. For the miss error rate, statistical analysis of errors will be performed on arcsine transformed error rates (Hogg & Craig, 1995).

Any secondary analyses?

We will explore the data for other patterns of interest: E.g.

- Order effects: Does a categorical target on trial N influence the probability of detecting such a target on N+1.

How many observations will be collected or what will determine sample size? No need to justify decision, but be precise about exactly how the number will be determined.

To be consistent with the previous 'Mixed Hybrid Search' experiments, we will collect data from 12 observers. Participants are excluded if they are not between the ages of 18 and 55, if their visual acuity is worse than 20/25, or if they do not pass the Ishihara Color Test.

Anything else you would like to pre-register? (e.g., data exclusions, variables collected for exploratory purposes, unusual analyses planned?)

N/A

• Study Type: Interventional
• Enrollment (Actual): 12
• Phase: Not Applicable

Contacts and Locations

Study Locations

• United States
  • Massachusetts
    • Boston, Massachusetts, United States, 02215
      • Visual Attention Lab / Brigham and Women's Hospital

Participation Criteria

Eligibility Criteria

• Ages Eligible for Study: 18 years and older (Adult, Older Adult)
• Accepts Healthy Volunteers: Yes

Description

Inclusion Criteria:

• All welcome to enroll

Exclusion Criteria:

• excluded if Participant fails Ishihara color vision test
• excluded if Participant has acuity worse than 20/25 with correction

Study Plan

How is the study designed?

Design Details

• Primary Purpose: Basic Science
• Allocation: N/A
• Interventional Model: Single Group Assignment
• Masking: None (Open Label)

Number of Arms

1

Arms and Interventions

Participant Group / Arm	Intervention / Treatment
Experimental: Main Experiment; This is a single arm study where all participants are tested on 1-2 hours of visual search.	Behavioral: Stimulus Type; Targets can be specific or categorical; Behavioral: Target Presence; There is or is not a target present on each trial; Behavioral: Checklist; In fact, there is a checklist on every trial but these results will be compared to a non-checklist condition from another experiment

What is the study measuring?

Primary Outcome Measures

Outcome Measure	Measure Description	Time Frame
Accuracy	Did the Participant get the right answer?	Within 1 hour
Response Time (RT)	How long did it take to respond on each trial?	Within 1 hour

Collaborators and Investigators

• Sponsor: Brigham and Women's Hospital

Study record dates

Study Major Dates

• Study Start (Actual): January 17, 2018
• Primary Completion (Actual): January 1, 2024
• Study Completion (Actual): January 1, 2024

Study Registration Dates

• First Submitted: February 18, 2022
• First Submitted That Met QC Criteria: February 28, 2022
• First Posted (Actual): March 9, 2022

Study Record Updates

• Last Update Posted (Actual): March 25, 2025
• Last Update Submitted That Met QC Criteria: February 16, 2025
• Last Verified: February 1, 2025

More Information

Terms related to this study

• Keywords: errors; visual perception; BESH; visual search
• Other Study ID Numbers: 2007P000646-C

Plan for Individual participant data (IPD)

• Plan to Share Individual Participant Data (IPD)?: YES
• IPD Plan Description: De-identified raw data and other materials will be posted on the OSF site for the experiment
• IPD Sharing Time Frame: All materials will be available whenever requested.

IPD Sharing Access Criteria

Any reasonable request will be honored.

NOTE: The URL does reach our OSF page, successfully.

• IPD Sharing Supporting Information Type: STUDY_PROTOCOL; ICF

Drug and device information, study documents

• Studies a U.S. FDA-regulated drug product: No
• Studies a U.S. FDA-regulated device product: No