Dissociable signatures of visual salience and behavioral relevance across attentional priority maps in human cortex
Thomas C Sprague, Sirawaj Itthipuripat, Vy A Vo, John T Serences, Thomas C Sprague, Sirawaj Itthipuripat, Vy A Vo, John T Serences
Abstract
Computational models posit that visual attention is guided by activity within spatial maps that index the image-computable salience and the behavioral relevance of objects in the scene. These spatial maps are theorized to be instantiated as activation patterns across a series of retinotopic visual regions in occipital, parietal, and frontal cortex. Whereas previous research has identified sensitivity to either the behavioral relevance or the image-computable salience of different scene elements, the simultaneous influence of these factors on neural "attentional priority maps" in human cortex is not well understood. We tested the hypothesis that visual salience and behavioral relevance independently impact the activation profile across retinotopically organized cortical regions by quantifying attentional priority maps measured in human brains using functional MRI while participants attended one of two differentially salient stimuli. We found that the topography of activation in priority maps, as reflected in the modulation of region-level patterns of population activity, independently indexed the physical salience and behavioral relevance of each scene element. Moreover, salience strongly impacted activation patterns in early visual areas, whereas later visual areas were dominated by relevance. This suggests that prioritizing spatial locations relies on distributed neural codes containing graded representations of salience and relevance across the visual hierarchy. NEW & NOTEWORTHY We tested a theory which supposes that neural systems represent scene elements according to both their salience and their relevance in a series of "priority maps" by measuring functional MRI activation patterns across human brains and reconstructing spatial maps of the visual scene. We found that different regions indexed either the salience or the relevance of scene items, but not their interaction, suggesting an evolving representation of salience and relevance across different visual areas.
Keywords: computational neuroimaging; fMRI; inverted encoding model; priority map; visual spatial attention.
Figures
References
- Balan PF, Gottlieb J. Integration of exogenous input into a dynamic salience map revealed by perturbing attention. J Neurosci 26: 9239–9249, 2006. doi:10.1523/JNEUROSCI.1898-06.2006.
- Benjamini Y, Yekutieli D. The control of the false discovery rate in multiple testing under dependency. Ann Stat 29: 1165–1188, 2001. doi:10.1214/aos/1013699998.
- Bertleff S, Fink GR, Weidner R. The role of top-down focused spatial attention in preattentive salience coding and salience-based attentional capture. J Cogn Neurosci 28: 1152–1165, 2016. doi:10.1162/jocn_a_00964.
- Betz T, Wilming N, Bogler C, Haynes JD, König P. Dissociation between saliency signals and activity in early visual cortex. J Vis 13: 6, 2013. doi:10.1167/13.14.6.
- Bichot NP, Schall JD. Effects of similarity and history on neural mechanisms of visual selection. Nat Neurosci 2: 549–554, 1999. doi:10.1038/9205.
- Bisley JW, Goldberg ME. Neuronal activity in the lateral intraparietal area and spatial attention. Science 299: 81–86, 2003a. doi:10.1126/science.1077395.
- Bisley JW, Goldberg ME. Attention, intention, and priority in the parietal lobe. Annu Rev Neurosci 33: 1–21, 2010. doi:10.1146/annurev-neuro-060909-152823.
- Bogler C, Bode S, Haynes JD. Decoding successive computational stages of saliency processing. Curr Biol 21: 1667–1671, 2011. doi:10.1016/j.cub.2011.08.039.
- Bogler C, Bode S, Haynes JD. Orientation pop-out processing in human visual cortex. Neuroimage 81: 73–80, 2013. doi:10.1016/j.neuroimage.2013.05.040.
- Brainard DH. The Psychophysics Toolbox. Spat Vis 10: 433–436, 1997. doi:10.1163/156856897X00357.
- Brouwer GJ, Heeger DJ. Decoding and reconstructing color from responses in human visual cortex. J Neurosci 29: 13992–14003, 2009. doi:10.1523/JNEUROSCI.3577-09.2009.
- Buracas GT, Boynton GM. The effect of spatial attention on contrast response functions in human visual cortex. J Neurosci 27: 93–97, 2007. doi:10.1523/JNEUROSCI.3162-06.2007.
- Burrows BE, Moore T. Influence and limitations of popout in the selection of salient visual stimuli by area V4 neurons. J Neurosci 29: 15169–15177, 2009. doi:10.1523/JNEUROSCI.3710-09.2009.
- Carrasco M. Visual attention: the past 25 years. Vision Res 51: 1484–1525, 2011. doi:10.1016/j.visres.2011.04.012.
- Chelazzi L, Eštočinová J, Calletti R, Lo Gerfo E, Sani I, Della Libera C, Santandrea E. Altering spatial priority maps via reward-based learning. J Neurosci 34: 8594–8604, 2014. doi:10.1523/JNEUROSCI.0277-14.2014.
- Constantinidis C, Steinmetz MA. Posterior parietal cortex automatically encodes the location of salient stimuli. J Neurosci 25: 233–238, 2005. doi:10.1523/JNEUROSCI.3379-04.2005.
- de Haas B, Schwarzkopf DS, Anderson EJ, Rees G. Perceptual load affects spatial tuning of neuronal populations in human early visual cortex. Curr Biol 24: R66–R67, 2014. doi:10.1016/j.cub.2013.11.061.
- Dumoulin SO, Wandell BA. Population receptive field estimates in human visual cortex. Neuroimage 39: 647–660, 2008. doi:10.1016/j.neuroimage.2007.09.034.
- Egeth HE, Yantis S. Visual attention: control, representation, and time course. Annu Rev Psychol 48: 269–297, 1997. doi:10.1146/annurev.psych.48.1.269.
- Engel SA, Rumelhart DE, Wandell BA, Lee AT, Glover GH, Chichilnisky EJ, Shadlen MN. fMRI of human visual cortex. Nature 369: 525, 1994. doi:10.1038/369525a0.
- Ester EF, Sprague TC, Serences JT. Parietal and frontal cortex encode stimulus-specific mnemonic representations during visual working memory. Neuron 87: 893–905, 2015. doi:10.1016/j.neuron.2015.07.013.
- Fecteau JH, Munoz DP. Salience, relevance, and firing: a priority map for target selection. Trends Cogn Sci 10: 382–390, 2006. doi:10.1016/j.tics.2006.06.011.
- Folk CL, Leber AB, Egeth HE. Made you blink! Contingent attentional capture produces a spatial blink. Percept Psychophys 64: 741–753, 2002. doi:10.3758/BF03194741.
- Folk CL, Remington RW, Johnston JC. Involuntary covert orienting is contingent on attentional control settings. J Exp Psychol Hum Percept Perform 18: 1030–1044, 1992. doi:10.1037/0096-1523.18.4.1030.
- Foster JJ, Sutterer DW, Serences JT, Vogel EK, Awh E. Alpha-band oscillations enable spatially and temporally resolved tracking of covert spatial attention. Psychol Sci 28: 929–941, 2017. doi:10.1177/0956797617699167.
- Freeman J, Brouwer GJ, Heeger DJ, Merriam EP. Orientation decoding depends on maps, not columns. J Neurosci 31: 4792–4804, 2011. doi:10.1523/JNEUROSCI.5160-10.2011.
- Freeman J, Heeger DJ, Merriam EP. Coarse-scale biases for spirals and orientation in human visual cortex. J Neurosci 33: 19695–19703, 2013. doi:10.1523/JNEUROSCI.0889-13.2013.
- Gandhi SP, Heeger DJ, Boynton GM. Spatial attention affects brain activity in human primary visual cortex. Proc Natl Acad Sci USA 96: 3314–3319, 1999. [Erratum in Proc Natl Acad Sci USA 96: 7610, 1999.] 10.1073/pnas.96.6.3314.
- Garcia JO, Srinivasan R, Serences JT. Near-real-time feature-selective modulations in human cortex. Curr Biol 23: 515–522, 2013. doi:10.1016/j.cub.2013.02.013.
- Gottlieb JP, Kusunoki M, Goldberg ME. The representation of visual salience in monkey parietal cortex. Nature 391: 481–484, 1998. doi:10.1038/35135.
- Gouws AD, Alvarez I, Watson DM, Uesaki M, Rodgers J, Morland AB. On the role of suppression in spatial attention: evidence from negative BOLD in human subcortical and cortical structures. J Neurosci 34: 10347–10360, 2014. doi:10.1523/JNEUROSCI.0164-14.2014.
- Hagler DJ Jr, Sereno MI. Spatial maps in frontal and prefrontal cortex. Neuroimage 29: 567–577, 2006. doi:10.1016/j.neuroimage.2005.08.058.
- Ipata AE, Gee AL, Gottlieb J, Bisley JW, Goldberg ME. LIP responses to a popout stimulus are reduced if it is overtly ignored. Nat Neurosci 9: 1071–1076, 2006. doi:10.1038/nn1734.
- Itthipuripat S, Cha K, Byers A, Serences JT. Two different mechanisms support selective attention at different phases of training. PLoS Biol 15: e2001724, 2017. doi:10.1371/journal.pbio.2001724.
- Itthipuripat S, Ester EF, Deering S, Serences JT. Sensory gain outperforms efficient readout mechanisms in predicting attention-related improvements in behavior. J Neurosci 34: 13384–13398, 2014a. doi:10.1523/JNEUROSCI.2277-14.2014.
- Itthipuripat S, Garcia JO, Rungratsameetaweemana N, Sprague TC, Serences JT. Changing the spatial scope of attention alters patterns of neural gain in human cortex. J Neurosci 34: 112–123, 2014b. doi:10.1523/JNEUROSCI.3943-13.2014.
- Itti L, Koch C. Computational modelling of visual attention. Nat Rev Neurosci 2: 194–203, 2001. doi:10.1038/35058500.
- Jerde TA, Merriam EP, Riggall AC, Hedges JH, Curtis CE. Prioritized maps of space in human frontoparietal cortex. J Neurosci 32: 17382–17390, 2012. doi:10.1523/JNEUROSCI.3810-12.2012.
- Johannes S, Münte TF, Heinze HJ, Mangun GR. Luminance and spatial attention effects on early visual processing. Brain Res Cogn Brain Res 2: 189–205, 1995. doi:10.1016/0926-6410(95)90008-X.
- Kahnt T, Park SQ, Haynes J-D, Tobler PN. Disentangling neural representations of value and salience in the human brain. Proc Natl Acad Sci USA 111: 5000–5005, 2014. doi:10.1073/pnas.1320189111.
- Kastner S, Pinsk MA, De Weerd P, Desimone R, Ungerleider LG. Increased activity in human visual cortex during directed attention in the absence of visual stimulation. Neuron 22: 751–761, 1999. doi:10.1016/S0896-6273(00)80734-5.
- Katsuki F, Constantinidis C. Early involvement of prefrontal cortex in visual bottom-up attention. Nat Neurosci 15: 1160–1166, 2012. doi:10.1038/nn.3164.
- Kay KN, Weiner KS, Grill-Spector K. Attention reduces spatial uncertainty in human ventral temporal cortex. Curr Biol 25: 595–600, 2015. doi:10.1016/j.cub.2014.12.050.
- Kay KN, Winawer J, Mezer A, Wandell BA. Compressive spatial summation in human visual cortex. J Neurophysiol 110: 481–494, 2013. doi:10.1152/jn.00105.2013.
- Kay KN, Yeatman JD. Bottom-up and top-down computations in word- and face-selective cortex. eLife 6: e22341, 2017. doi:10.7554/eLife.22341.
- Kim YJ, Grabowecky M, Paller KA, Muthu K, Suzuki S. Attention induces synchronization-based response gain in steady-state visual evoked potentials. Nat Neurosci 10: 117–125, 2007. doi:10.1038/nn1821.
- Klein BP, Harvey BM, Dumoulin SO. Attraction of position preference by spatial attention throughout human visual cortex. Neuron 84: 227–237, 2014. doi:10.1016/j.neuron.2014.08.047.
- Klyszejko Z, Rahmati M, Curtis CE. Attentional priority determines working memory precision. Vision Res 105: 70–76, 2014. doi:10.1016/j.visres.2014.09.002.
- Koch C, Ullman S. Shifts in selective visual attention: towards the underlying neural circuitry. Hum Neurobiol 4: 219–227, 1985.
- Kusunoki M, Gottlieb J, Goldberg ME. The lateral intraparietal area as a salience map: the representation of abrupt onset, stimulus motion, and task relevance. Vision Res 40: 1459–1468, 2000. doi:10.1016/S0042-6989(99)00212-6.
- Mackey WE, Winawer J, Curtis CE. Visual field map clusters in human frontoparietal cortex. eLife 6: e22974, 2017. doi:10.7554/eLife.22974.
- MacLean MH, Diaz GK, Giesbrecht B. Irrelevant learned reward associations disrupt voluntary spatial attention. Atten Percept Psychophys 78: 2241–2252, 2016. doi:10.3758/s13414-016-1103-x.
- MacLean MH, Giesbrecht B. Irrelevant reward and selection histories have different influences on task-relevant attentional selection. Atten Percept Psychophys 77: 1515–1528, 2015a. doi:10.3758/s13414-015-0851-3.
- MacLean MH, Giesbrecht B. Neural evidence reveals the rapid effects of reward history on selective attention. Brain Res 1606: 86–94, 2015b. doi:10.1016/j.brainres.2015.02.016.
- Masciocchi CM, Mihalas S, Parkhurst D, Niebur E. Everyone knows what is interesting: salient locations which should be fixated. J Vis 9: 25, 2009. doi:10.1167/9.11.25.
- Mazer JA, Gallant JL. Goal-related activity in V4 during free viewing visual search. Evidence for a ventral stream visual salience map. Neuron 40: 1241–1250, 2003. doi:10.1016/S0896-6273(03)00764-5.
- Mo C, He D, Fang F. Attention priority map of face images in human early visual cortex. J Neurosci 38: 149–157, 2018.
- Murray SO. The effects of spatial attention in early human visual cortex are stimulus independent. J Vis 8: 2.1–11, 2008. doi:10.1167/8.10.2.
- Parkhurst D, Law K, Niebur E. Modeling the role of salience in the allocation of overt visual attention. Vision Res 42: 107–123, 2002. doi:10.1016/S0042-6989(01)00250-4.
- Pelli DG. The VideoToolbox software for visual psychophysics: transforming numbers into movies. Spat Vis 10: 437–442, 1997. doi:10.1163/156856897X00366.
- Pestilli F, Carrasco M, Heeger DJ, Gardner JL. Attentional enhancement via selection and pooling of early sensory responses in human visual cortex. Neuron 72: 832–846, 2011. doi:10.1016/j.neuron.2011.09.025.
- Poltoratski S, Ling S, McCormack D, Tong F. Characterizing the effects of feature salience and top-down attention in the early visual system. J Neurophysiol 118: 564–573, 2017. doi:10.1152/jn.00924.2016.
- Rahmati M, Saber GT, Curtis CE. Population dynamics of early visual cortex during working memory. J Cogn Neurosci 30: 219–233, 2018. doi:10.1162/jocn_a_01196.
- Saber GT, Pestilli F, Curtis CE. Saccade planning evokes topographically specific activity in the dorsal and ventral streams. J Neurosci 35: 245–252, 2015. doi:10.1523/JNEUROSCI.1687-14.2015.
- Samaha J, Sprague TC, Postle BR. Decoding and reconstructing the focus of spatial attention from the topography of alpha-band oscillations. J Cogn Neurosci 28: 1090–1097, 2016. doi:10.1162/jocn_a_00955.
- Serences JT, Yantis S. Selective visual attention and perceptual coherence. Trends Cogn Sci 10: 38–45, 2006. doi:10.1016/j.tics.2005.11.008.
- Serences JT, Yantis S. Spatially selective representations of voluntary and stimulus-driven attentional priority in human occipital, parietal, and frontal cortex. Cereb Cortex 17: 284–293, 2007. doi:10.1093/cercor/bhj146.
- Sheremata SL, Silver MA. Hemisphere-dependent attentional modulation of human parietal visual field representations. J Neurosci 35: 508–517, 2015. doi:10.1523/JNEUROSCI.2378-14.2015.
- Silver MA, Ress D, Heeger DJ. Topographic maps of visual spatial attention in human parietal cortex. J Neurophysiol 94: 1358–1371, 2005. doi:10.1152/jn.01316.2004.
- Silver MA, Ress D, Heeger DJ. Neural correlates of sustained spatial attention in human early visual cortex. J Neurophysiol 97: 229–237, 2007. doi:10.1152/jn.00677.2006.
- Somers DC, Dale AM, Seiffert AE, Tootell RB. Functional MRI reveals spatially specific attentional modulation in human primary visual cortex. Proc Natl Acad Sci USA 96: 1663–1668, 1999. doi:10.1073/pnas.96.4.1663.
- Sprague TC, Ester EF, Serences JT. Reconstructions of information in visual spatial working memory degrade with memory load. Curr Biol 24: 2174–2180, 2014. doi:10.1016/j.cub.2014.07.066.
- Sprague TC, Ester EF, Serences JT. Restoring latent visual working memory representations in human cortex. Neuron 91: 694–707, 2016. doi:10.1016/j.neuron.2016.07.006.
- Sprague TC, Saproo S, Serences JT. Visual attention mitigates information loss in small- and large-scale neural codes. Trends Cogn Sci 19: 215–226, 2015. doi:10.1016/j.tics.2015.02.005.
- Sprague TC, Serences JT. Attention modulates spatial priority maps in the human occipital, parietal and frontal cortices. Nat Neurosci 16: 1879–1887, 2013. doi:10.1038/nn.3574.
- Srimal R, Curtis CE. Persistent neural activity during the maintenance of spatial position in working memory. Neuroimage 39: 455–468, 2008. doi:10.1016/j.neuroimage.2007.08.040.
- Swisher JD, Halko MA, Merabet LB, McMains SA, Somers DC. Visual topography of human intraparietal sulcus. J Neurosci 27: 5326–5337, 2007. doi:10.1523/JNEUROSCI.0991-07.2007.
- Theeuwes J. Endogenous and exogenous control of visual selection. Perception 23: 429–440, 1994. doi:10.1068/p230429.
- Theeuwes J. Top-down and bottom-up control of visual selection. Acta Psychol (Amst) 135: 77–99, 2010. doi:10.1016/j.actpsy.2010.02.006.
- Thompson KG, Bichot NP, Schall JD. Dissociation of visual discrimination from saccade programming in macaque frontal eye field. J Neurophysiol 77: 1046–1050, 1997. doi:10.1152/jn.1997.77.2.1046.
- Usher M, Niebur E. Modeling the temporal dynamics of it neurons in visual search: a mechanism for top-down selective attention. J Cogn Neurosci 8: 311–327, 1996. doi:10.1162/jocn.1996.8.4.311.
- Van der Stigchel S, Belopolsky AV, Peters JC, Wijnen JG, Meeter M, Theeuwes J. The limits of top-down control of visual attention. Acta Psychol (Amst) 132: 201–212, 2009. doi:10.1016/j.actpsy.2009.07.001.
- Veale R, Hafed ZM, Yoshida M. How is visual salience computed in the brain? Insights from behaviour, neurobiology and modelling. Philos Trans R Soc Lond B Biol Sci 372: 20160113, 2017. doi:10.1098/rstb.2016.0113.
- Vo VA, Sprague TC, Serences JT. Spatial tuning shifts increase the discriminability and fidelity of population codes in visual cortex. J Neurosci 37: 3386–3401, 2017. doi:10.1523/JNEUROSCI.3484-16.2017.
- White BJ, Berg DJ, Kan JY, Marino RA, Itti L, Munoz DP. Superior colliculus neurons encode a visual saliency map during free viewing of natural dynamic video. Nat Commun 8: 14263, 2017. doi:10.1038/ncomms14263.
- Wolfe JM. Guided Search 2.0 A revised model of visual search. Psychon Bull Rev 1: 202–238, 1994. doi:10.3758/BF03200774.
- Yantis S, Johnson DN. Mechanisms of attentional priority. J Exp Psychol Hum Percept Perform 16: 812–825, 1990. doi:10.1037/0096-1523.16.4.812.
- Yildirim F, Carvalho J, Cornelissen FW. A second-order orientation-contrast stimulus for population-receptive-field-based retinotopic mapping. Neuroimage 164: 183–193, 2018. doi:10.1016/j.neuroimage.2017.06.073.
- Zelinsky GJ, Bisley JW. The what, where, and why of priority maps and their interactions with visual working memory. Ann N Y Acad Sci 1339: 154–164, 2015. doi:10.1111/nyas.12606.
- Zhang X, Zhaoping L, Zhou T, Fang F. Neural activities in V1 create a bottom-up saliency map. Neuron 73: 183–192, 2012. doi:10.1016/j.neuron.2011.10.035.
Source: PubMed