Dissociable signatures of visual salience and behavioral relevance across attentional priority maps in human cortex

Abstract

Computational models posit that visual attention is guided by activity within spatial maps that index the image-computable salience and the behavioral relevance of objects in the scene. These spatial maps are theorized to be instantiated as activation patterns across a series of retinotopic visual regions in occipital, parietal, and frontal cortex. Whereas previous research has identified sensitivity to either the behavioral relevance or the image-computable salience of different scene elements, the simultaneous influence of these factors on neural "attentional priority maps" in human cortex is not well understood. We tested the hypothesis that visual salience and behavioral relevance independently impact the activation profile across retinotopically organized cortical regions by quantifying attentional priority maps measured in human brains using functional MRI while participants attended one of two differentially salient stimuli. We found that the topography of activation in priority maps, as reflected in the modulation of region-level patterns of population activity, independently indexed the physical salience and behavioral relevance of each scene element. Moreover, salience strongly impacted activation patterns in early visual areas, whereas later visual areas were dominated by relevance. This suggests that prioritizing spatial locations relies on distributed neural codes containing graded representations of salience and relevance across the visual hierarchy. NEW & NOTEWORTHY We tested a theory which supposes that neural systems represent scene elements according to both their salience and their relevance in a series of "priority maps" by measuring functional MRI activation patterns across human brains and reconstructing spatial maps of the visual scene. We found that different regions indexed either the salience or the relevance of scene items, but not their interaction, suggesting an evolving representation of salience and relevance across different visual areas.

Keywords: computational neuroimaging; fMRI; inverted encoding model; priority map; visual spatial attention.

Figures

Fig. 1.

Identifying maps of visual salience and behavioral relevance. A: we define a salience map as a map of the visual scene where each position on the map indexes the importance of the corresponding location within the scene based on image-computable features, such as contrast, motion, or distinctness from the background or other scene items. Accordingly, activity within a salience map should scale with the visual salience of local scene elements. For an example scene, in which two stimuli of differing contrast are presented and a participant is cued to attend to one (dashed yellow circle), a salience map would show higher activation at visual field positions corresponding to higher contrast, even if those elements of the scene are not relevant for behavior. B: for a given location in the scene, activation in a pure salience map would scale only with image-computable features, such as contrast (shown). C: we define a relevance map as a map of the visual scene where each position on the map indexes the behavioral relevance of the corresponding location. In this example scene, the relevance map would only show activation at locations relevant to the behavior of the observer, independent of their visual salience. This requires that visual locations corresponding to highly salient but irrelevant stimuli are not reflected in a pure relevance map. D: a location within a pure relevance map would show high activity when the corresponding position is relevant for behavior and low activity when it is irrelevant. Importantly, a cortical map of visual space can reflect a combination of both visual salience and behavioral relevance by virtue of its position within the visual processing hierarchy.

Fig. 2.

Measuring spatial maps of visual salience and behavioral relevance. A: on each trial, a fixation cue (red or blue dot) indicated whether participants (n = 8) should attend the clockwise (CW) or counterclockwise (CCW) of 2 stimuli that appeared 500 ms after cue onset. We presented stimuli at 3.5° eccentricity separated by 144° or 72° polar angle, and for each trial we randomly rotated the display such that each polar angle was equally likely to be stimulated on a given trial. CW and CCW are with respect to the polar angle between the 2 stimuli (this was always unambiguous, because the stimuli were never 180° apart). Stimuli, which consisted of randomly oriented light and dark lines, flickered at 15 Hz for 3,000 ms (left inset). On every trial, a target appeared within each stimulus stream (colored border in left inset). Targets (1,000 ms) were variable-coherence spirals called “Type 1” (line angle CCW from radial) or Type 2 (shown; line angle CW from radial) (see right inset). Participants responded with a button press to indicate which type of spiral appeared in the cued stimulus. ITI, intertrial interval. B: we presented the attended stimulus (yellow dashed circle) and unattended stimulus each at 1 of 3 logarithmically spaced contrast levels, fully crossed. C: on each trial, 2 stimuli appeared at 3.5° eccentricity and could be either 72° or 144° polar angle apart (inset). The attended stimulus (yellow for this example arrangement) could be any of the 5 “base” positions. The attended stimulus could appear CW (solid lines) or CCW (dashed lines) with respect to the unattended stimulus. The entire stimulus display was rotated randomly about fixation on each trial, so each and every trial involved a distinct stimulus display. D and E: behavioral performance (D) and response time (E) did not reliably vary as a function of attended or unattended stimulus contrast, or their interaction (2-way permuted repeated-measures ANOVA, P values for main effect of attended contrast, unattended contrast, and interaction for accuracy: 0.359, 0.096, 0.853; for response time: 0.926, 0.143, 0.705). F: average coherence used for each contrast to achieve performance in B and C. Although qualitatively the coherence decreases with increasing contrast, there is no main effect of target contrast on mean coherence (1-way permuted repeated-measures ANOVA, P = 0.153). Error bars are SE across participants (n = 8). Attn, attended; Unattn, unattended.

Fig. 3.

Measuring salience and relevance maps with an inverted encoding model. A: we estimated a spatial encoding model using data from a spatial mapping task in which participants viewed flickering checkerboard disks presented across a triangular grid of positions spanning a hexagonal region of the screen with spatial jitter (see Sprague et al. 2016). We modeled each voxel’s spatial sensitivity (i.e., receptive field, RF) as a weighted sum of smooth functions centered at each point in a triangular grid spanning the stimulus display visible inside the scanner. By knowing the stimulus on each trial (contrast mask) and the sensitivity profile of each modeled spatial “information channel” (spatial filter), we could predict how each modeled information channel should respond to each stimulus. We then solve for the contribution of each channel to the measured signal in each voxel. This step amounts to solving the standard general linear model (GLM) typical in functional MRI (fMRI) analyses and is univariate (each voxel can be estimated independently). B: we then used all voxels within a region to compute an inverted encoding model (IEM). This step is multivariate; all voxels contribute to the IEM. This IEM allows us to transform activation patterns measured during the main spatial attention task (Fig. 2A) to activations of modeled information channels. We then compute a weighted sum of information channels on the basis of their computed activation in the spatial attention task. These images of the visual field index the activation across the entire cortical region transformed into visual field coordinates. To quantify activation in the map, we extract the mean signal over the map area corresponding to the known stimulus positions on that trial (red dashed circle: attended stimulus position; black dashed circle: unattended stimulus position). For visualization of spatial maps averaged across trials, we rotated reconstructions as though stimuli were presented at positions indicated in cartoons (e.g., Fig. 2C). See Sprague et al. (2016) for detailed methods on image alignment and coregistration.

Fig. 4.

Reconstructed spatial maps index stimulus salience and relevance. Image reconstructions are computed for each stimulus separation condition (grouped rows) and each stimulus contrast pair (entries within each 3 × 3 group of reconstructions). Within each group of reconstructions, the diagonals (top left to bottom right) are trials with matched contrasts and are useful for visualizing the qualitative effect of behavioral relevance (spatial attention) on map profiles. Even in V1, locations near attended stimuli are represented more strongly than those near unattended stimuli. Going from top to bottom (unattended stimulus) and left to right (attended stimulus) within each group of reconstructions steps through increasing stimulus contrast levels and can be used to infer the sensitivity of each visual field map to visual salience. Color mapping scaled within each region of interest (ROI) independently (see inset, left center). Range subtended by vertical line for each ROI indicates color range (see color bar) for that ROI. BOLD, blood oxygen level dependent; V1–hV4, visual cortex areas; IPS0–IPS3, intraparietal sulcus regions; sPCS, superior precentral sulcus.

Fig. 5.

Quantifying sensitivity to visual salience and behavioral relevance across cortex. For each stimulus, we averaged the reconstruction activation on all trials where each stimulus appeared at a given contrast (see also Fig. 6). In visual cortex areas V1, V2, V3, and hV4, reconstructed map activation increased with stimulus salience, regardless of whether the stimulus was attended. In all regions of interest (ROIs) except for intraparietal sulcus regions IPS2 and IPS3 and superior precentral sulcus (sPCS), reconstructed map activation also reflected behavioral relevance such that the attended location was more active than the unattended location. Error bars are SE across participants (n = 8). Lines indicate best-fit linear contrast response function, and all data are plotted on a log scale. All P values are available in Table 1. Black symbols indicate significant main effects or interaction (corrected via false discovery rate across all ROIs), and gray symbols indicate trend (P ≤ 0.05, uncorrected). BOLD, blood oxygen level dependent; attn, attended; unattn, unattended.

Fig. 6.

Attended stimulus representation does not depend on distractor stimulus contrast. A: reconstructed map activation for attended stimulus location across all salience combinations. Within each plot, linked points correspond to the contrast of the attended stimulus, and the individual linked points indicate the attended stimulus activation at each unattended stimulus contrast level (20%, 40%, 80%). Error bars are SE across participants. We observed no significant interactions [2-way permuted repeated-measures ANOVA within each region of interest (ROI), factors of attended and unattended stimulus contrast, P ≥ 0.041, minimum P value V3A), so we collapse across unattended stimulus salience within each attended stimulus salience level (Fig. 5). B: same as A, but for unattended stimulus activation, as sorted by attended stimulus contrast within each unattended stimulus contrast level. Again, we did not observe any significant interactions (P ≥ 0.026, minimum P value V3A), so these are collapsed (Fig. 5). Error bars are SE. All P values are available in Table 2. Black symbols indicate significant main effects or interactions, corrected for multiple comparisons via false discovery rate across regions of interest. Gray symbols indicate trends, defined as P ≤ 0.05, uncorrected. BOLD, blood oxygen level dependent; Attn, attended; Unattn, unattended.