The representational dynamics of task and object processing in humans
Martin N Hebart, Brett B Bankson, Assaf Harel, Chris I Baker, Radoslaw M Cichy, Martin N Hebart, Brett B Bankson, Assaf Harel, Chris I Baker, Radoslaw M Cichy
Abstract
Despite the importance of an observer's goals in determining how a visual object is categorized, surprisingly little is known about how humans process the task context in which objects occur and how it may interact with the processing of objects. Using magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI) and multivariate techniques, we studied the spatial and temporal dynamics of task and object processing. Our results reveal a sequence of separate but overlapping task-related processes spread across frontoparietal and occipitotemporal cortex. Task exhibited late effects on object processing by selectively enhancing task-relevant object features, with limited impact on the overall pattern of object representations. Combining MEG and fMRI data, we reveal a parallel rise in task-related signals throughout the cerebral cortex, with an increasing dominance of task over object representations from early to higher visual areas. Collectively, our results reveal the complex dynamics underlying task and object representations throughout human cortex.
Trial registration: ClinicalTrials.gov NCT00001360.
Keywords: MEG; MEG-fMRI fusion; fMRI; human; multivariate analysis; neuroscience; object processing; task context.
Conflict of interest statement
MH, BB, AH, CB, RC No competing interests declared
Figures
References
- Bankson BB, Hebart MN, Groen II BCI. The temporal evolution of conceptual object representations revealed through models of behavior, semantics and deep neural networks. bioRxiv. 2017 doi: 10.1101/223990.
- Belongie S, Malik J, Puzicha J. Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2002;24:509–522. doi: 10.1109/34.993558.
- Bode S, Haynes JD. Decoding sequential stages of task preparation in the human brain. NeuroImage. 2009;45:606–613. doi: 10.1016/j.neuroimage.2008.11.031.
- Bracci S, Daniels N, Op de Beeck H. Task context overrules object- and category-related representational content in the human parietal cortex. Cerebral Cortex. 2017;84:1-12. doi: 10.1093/cercor/bhw419.
- Bracci S, Op de Beeck H. Dissociations and associations between shape and category representations in the two visual pathways. Journal of Neuroscience. 2016;36:432–444. doi: 10.1523/JNEUROSCI.2314-15.2016.
- Bugatus L, Weiner KS, Grill-Spector K. Task alters category representations in prefrontal but not high-level visual cortex. NeuroImage. 2017;155:437–449. doi: 10.1016/j.neuroimage.2017.03.062.
- Carlson T, Tovar DA, Alink A, Kriegeskorte N. Representational dynamics of object vision: the first 1000 ms. Journal of Vision. 2013;13:1–19. doi: 10.1167/13.10.1.
- Chang C-C, Lin C-J. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology. 2011;2:27.
- Cichy RM, Chen Y, Haynes JD. Encoding the identity and location of objects in human LOC. NeuroImage. 2011;54:2297–2307. doi: 10.1016/j.neuroimage.2010.09.044.
- Cichy RM, Pantazis D, Oliva A. Resolving human object recognition in space and time. Nature Neuroscience. 2014;17:455–462. doi: 10.1038/nn.3635.
- Cichy RM, Pantazis D, Oliva A. Similarity-based fusion of meg and fmri reveals spatio-temporal dynamics in human cortex during visual object recognition. Cerebral Cortex. 2016;26:3563–3579. doi: 10.1093/cercor/bhw135.
- Clarke A, Devereux BJ, Randall B, Tyler LK. Predicting the time course of individual objects with MEG. Cerebral Cortex. 2015;25:3602–3612. doi: 10.1093/cercor/bhu203.
- DiCarlo JJ, Zoccolan D, Rust NC. How does the brain solve visual object recognition? Neuron. 2012;73:415–434. doi: 10.1016/j.neuron.2012.01.010.
- Duncan J. The multiple-demand (MD) system of the primate brain: mental programs for intelligent behaviour. Trends in Cognitive Sciences. 2010;14:172–179. doi: 10.1016/j.tics.2010.01.004.
- Emadi N, Esteky H. Behavioral demand modulates object category representation in the inferior temporal cortex. Journal of Neurophysiology. 2014;112:2628–2637. doi: 10.1152/jn.00761.2013.
- Erez Y, Duncan J. Discrimination of visual categories based on behavioral relevance in widespread regions of frontoparietal cortex. Journal of Neuroscience. 2015;35:12383–12393. doi: 10.1523/JNEUROSCI.1134-15.2015.
- Freedman DJ, Assad JA. Neuronal mechanisms of visual categorization: An abstract view on decision making. Annual Review of Neuroscience. 2016;39:129–147. doi: 10.1146/annurev-neuro-071714-033919.
- Freedman DJ, Riesenhuber M, Poggio T, Miller EK. A comparison of primate prefrontal and inferior temporal cortices during visual categorization. Journal of Neuroscience. 2003;23:5235–5246.
- Greene MR, Baldassano C, Esteva A, Beck DM, Fei-Fei L. Visual scenes are categorized by function. Journal of Experimental Psychology: General. 2016;145:82–94. doi: 10.1037/xge0000129.
- Grill-Spector K, Kushnir T, Edelman S, Avidan G, Itzchak Y, Malach R. Differential processing of objects under various viewing conditions in the human lateral occipital complex. Neuron. 1999;24:187–203. doi: 10.1016/S0896-6273(00)80832-6.
- Groen II, Ghebreab S, Lamme VA, Scholte HS. The time course of natural scene perception with reduced attention. Journal of Neurophysiology. 2016;115:931–946. doi: 10.1152/jn.00896.2015.
- Grootswagers T, Wardle SG, Carlson TA. Decoding dynamic brain patterns from evoked responses: A tutorial on multivariate pattern analysis applied to time series neuroimaging data. Journal of Cognitive Neuroscience. 2017;29:677–697. doi: 10.1162/jocn_a_01068.
- Harel A, Kravitz DJ, Baker CI. Task context impacts visual object processing differentially across the cortex. PNAS. 2014;111:E962–E971. doi: 10.1073/pnas.1312567111.
- Hebart MN, Baker CI. Deconstructing multivariate decoding for the study of brain function. NeuroImage. 2017 doi: 10.1016/j.neuroimage.2017.08.005.
- Hebart MN, Donner TH, Haynes JD. Human visual and parietal cortex encode visual choices independent of motor plans. NeuroImage. 2012;63:1393–1403. doi: 10.1016/j.neuroimage.2012.08.027.
- Hebart MN, Görgen K, Haynes JD. The Decoding Toolbox (TDT): a versatile software package for multivariate analyses of functional imaging data. Frontiers in Neuroinformatics. 2014;8:88. doi: 10.3389/fninf.2014.00088.
- Isik L, Meyers EM, Leibo JZ, Poggio T. The dynamics of invariant object recognition in the human visual system. Journal of Neurophysiology. 2014;111:91–102. doi: 10.1152/jn.00394.2013.
- Jehee JF, Brady DK, Tong F. Attention improves encoding of task-relevant features in the human visual cortex. Journal of Neuroscience. 2011;31:8210–8219. doi: 10.1523/JNEUROSCI.6153-09.2011.
- Jeong SK, Xu Y. Behaviorally relevant abstract object identity representation in the human parietal cortex. The Journal of Neuroscience. 2016;36:1607–1619. doi: 10.1523/JNEUROSCI.1016-15.2016.
- Kaiser D, Azzalini DC, Peelen MV. Shape-independent object category responses revealed by MEG and fMRI decoding. Journal of Neurophysiology. 2016;115:2246–2250. doi: 10.1152/jn.01074.2015.
- King JR, Dehaene S. Characterizing the dynamics of mental representations: the temporal generalization method. Trends in Cognitive Sciences. 2014;18:203–210. doi: 10.1016/j.tics.2014.01.002.
- Kok P, Brouwer GJ, van Gerven MA, de Lange FP. Prior expectations bias sensory representations in visual cortex. Journal of Neuroscience. 2013;33:16275–16284. doi: 10.1523/JNEUROSCI.0742-13.2013.
- Kok P, Jehee JF, de Lange FP. Less is more: expectation sharpens representations in the primary visual cortex. Neuron. 2012;75:265–270. doi: 10.1016/j.neuron.2012.04.034.
- Konen CS, Kastner S. Two hierarchically organized neural systems for object information in human visual cortex. Nature Neuroscience. 2008;11:224–231. doi: 10.1038/nn2036.
- Kravitz DJ, Kriegeskorte N, Baker CI. High-level visual object representations are constrained by position. Cerebral Cortex. 2010;20:2916–2925. doi: 10.1093/cercor/bhq042.
- Kriegeskorte N, Mur M, Bandettini P. Representational similarity analysis - connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience. 2008;2:4. doi: 10.3389/neuro.06.004.2008.
- Lowe MX, Gallivan JP, Ferber S, Cant JS. Feature diagnosticity and task context shape activity in human scene-selective cortex. NeuroImage. 2016;125:681–692. doi: 10.1016/j.neuroimage.2015.10.089.
- McKee JL, Riesenhuber M, Miller EK, Freedman DJ. Task dependence of visual and category representations in prefrontal and inferior temporal cortices. Journal of Neuroscience. 2014;34:16065–16075. doi: 10.1523/JNEUROSCI.1660-14.2014.
- Meyers EM, Freedman DJ, Kreiman G, Miller EK, Poggio T. Dynamic population coding of category information in inferior temporal and prefrontal cortex. Journal of Neurophysiology. 2008;100:1407–1419. doi: 10.1152/jn.90248.2008.
- Nastase SA, Connolly AC, Oosterhof NN, Halchenko YO, Guntupalli JS, Visconti di Oleggio Castello M, Gors J, Gobbini MI, Haxby JV. Attention selectively reshapes the geometry of distributed semantic representation. Cerebral Cortex. 2017;27:4277–4291. doi: 10.1093/cercor/bhx138.
- Nichols TE, Holmes AP. Nonparametric permutation tests for functional neuroimaging: a primer with examples. Human Brain Mapping. 2002;15:1–25. doi: 10.1002/hbm.1058.
- Pedhazur EJ. Multiple Regression in Behavioral Research: Explanation and Prediction. Orlando, FL: Harcourt Brace; 1997.
- Peelen MV, Fei-Fei L, Kastner S. Neural mechanisms of rapid natural scene categorization in human visual cortex. Nature. 2009;460:94–97. doi: 10.1038/nature08103.
- Peelen MV, Kastner S. A neural basis for real-world visual search in human occipitotemporal cortex. PNAS. 2011;108:12125–12130. doi: 10.1073/pnas.1101042108.
- Peters B, Bledowski C, Rieder M, Kaiser J. Recurrence of task set-related MEG signal patterns during auditory working memory. Brain Research. 2016;1640:232–242. doi: 10.1016/j.brainres.2015.12.006.
- Proklova D, Kaiser D, Peelen M. MEG sensor patterns reflect perceptual but not categorical similarity of animate and inanimate objects. bioRxiv. 2017 doi: 10.1101/238584.
- Proklova D, Kaiser D, Peelen MV. Disentangling representations of object shape and object category in human visual cortex: The animate-inanimate distinction. Journal of Cognitive Neuroscience. 2016;28:680–692. doi: 10.1162/jocn_a_00924.
- Riesenhuber M, Poggio T. Neural mechanisms of object recognition. Current Opinion in Neurobiology. 2002;12:162–168. doi: 10.1016/S0959-4388(02)00304-5.
- Ritchie JB, Bracci S, Op de Beeck H. Avoiding illusory effects in representational similarity analysis: What (not) to do with the diagonal. NeuroImage. 2017;148:197–200. doi: 10.1016/j.neuroimage.2016.12.079.
- Ritchie JB, Tovar DA, Carlson TA. Emerging object representations in the visual system predict reaction times for categorization. PLOS Computational Biology. 2015;11:e1004316. doi: 10.1371/journal.pcbi.1004316.
- Seibold DR, McPHEE RD. Commonality analysis: A method for decomposing explained variance in multiple regression analyses. Human Communication Research. 1979;5:355–365. doi: 10.1111/j.1468-2958.1979.tb00649.x.
- Serre T, Oliva A, Poggio T. A feedforward architecture accounts for rapid categorization. PNAS. 2007;104:6424–6429. doi: 10.1073/pnas.0700622104.
- Siegel M, Buschman TJ, Miller EK. Cortical information flow during flexible sensorimotor decisions. Science. 2015;348:1352–1355. doi: 10.1126/science.aab0551.
- Sigala N, Kusunoki M, Nimmo-Smith I, Gaffan D, Duncan J. Hierarchical coding for sequential task events in the monkey prefrontal cortex. PNAS. 2008;105:11969–11974. doi: 10.1073/pnas.0802569105.
- Stoet G, Snyder LH. Single neurons in posterior parietal cortex of monkeys encode cognitive set. Neuron. 2004;42:1003–1012. doi: 10.1016/j.neuron.2004.06.003.
- Stokes MG, Kusunoki M, Sigala N, Nili H, Gaffan D, Duncan J. Dynamic coding for cognitive control in prefrontal cortex. Neuron. 2013;78:364–375. doi: 10.1016/j.neuron.2013.01.039.
- Tadel F, Baillet S, Mosher JC, Pantazis D, Leahy RM. Brainstorm: a user-friendly application for MEG/EEG analysis. Computational Intelligence and Neuroscience. 2011;2011:1–13. doi: 10.1155/2011/879716.
- van de Nieuwenhuijzen ME, Backus AR, Bahramisharif A, Doeller CF, Jensen O, van Gerven MA. MEG-based decoding of the spatiotemporal dynamics of visual category perception. NeuroImage. 2013;83:1063–1073. doi: 10.1016/j.neuroimage.2013.07.075.
- VanRullen R, Thorpe SJ. The time course of visual processing: from early perception to decision-making. Journal of Cognitive Neuroscience. 2001;13:454–461. doi: 10.1162/08989290152001880.
- Vaziri-Pashkam M, Xu Y. Goal-directed visual processing differentially impacts human ventral and dorsal visual representations. The Journal of Neuroscience. 2017;37:8767–8782. doi: 10.1523/JNEUROSCI.3392-16.2017.
- Wallis JD, Anderson KC, Miller EK. Single neurons in prefrontal cortex encode abstract rules. Nature. 2001;411:953–956. doi: 10.1038/35082081.
- Waskom ML, Kumaran D, Gordon AM, Rissman J, Wagner AD. Frontoparietal representations of task context support the flexible control of goal-directed cognition. Journal of Neuroscience. 2014;34:10743–10755. doi: 10.1523/JNEUROSCI.5282-13.2014.
- Woolgar A, Thompson R, Bor D, Duncan J. Multi-voxel coding of stimuli, rules, and responses in human frontoparietal cortex. NeuroImage. 2011;56:744–752. doi: 10.1016/j.neuroimage.2010.04.035.
- Çukur T, Nishimoto S, Huth AG, Gallant JL. Attention during natural vision warps semantic representation across the human brain. Nature Neuroscience. 2013;16:763–770. doi: 10.1038/nn.3381.
Source: PubMed