Transferring structural knowledge across cognitive maps in humans and models
Shirley Mark, Rani Moran, Thomas Parr, Steve W Kennerley, Timothy E J Behrens, Shirley Mark, Rani Moran, Thomas Parr, Steve W Kennerley, Timothy E J Behrens
Abstract
Relations between task elements often follow hidden underlying structural forms such as periodicities or hierarchies, whose inferences fosters performance. However, transferring structural knowledge to novel environments requires flexible representations that are generalizable over particularities of the current environment, such as its stimuli and size. We suggest that humans represent structural forms as abstract basis sets and that in novel tasks, the structural form is inferred and the relevant basis set is transferred. Using a computational model, we show that such representation allows inference of the underlying structural form, important task states, effective behavioural policies and the existence of unobserved state-trajectories. In two experiments, participants learned three abstract graphs during two successive days. We tested how structural knowledge acquired on Day-1 affected Day-2 performance. In line with our model, participants who had a correct structural prior were able to infer the existence of unobserved state-trajectories and appropriate behavioural policies.
Conflict of interest statement
The authors declare no competing interests.
Figures
References
- Tolman EC. Maps in your mind. Psychol. Rev. 1948;55:189–208. doi: 10.1037/h0061626.
- Stachenfeld KL, Botvinick MM, Gershman SJ. The hippocampus as a predictive map. Nat. Neurosci. 2017;20:1643–1653. doi: 10.1038/nn.4650.
- Dayan P. Improving generalization for temporal difference learning: the successor representation. Neural Comput. 1993;5:613–624. doi: 10.1162/neco.1993.5.4.613.
- Kemp C, Goodman ND, Tenenbaum JB. Learning to learn causal models. Cogn. Sci. 2010;34:1185–1243. doi: 10.1111/j.1551-6709.2010.01128.x.
- Halford GS, Bain JD, Maybery MT, Andrews G. Induction of relational schemas: common processes in reasoning and complex learning. Cogn. Psychol. 1998;35:201–245. doi: 10.1006/cogp.1998.0679.
- Reeves L, Weisberg RW. The role of content and abstract information in analogical transfer. Psychol. Bull. 1994;115:381–400. doi: 10.1037/0033-2909.115.3.381.
- Strogatz SH. Exploring complex networks. Nature. 2001;410:268–276. doi: 10.1038/35065725.
- Gershman SJ, Niv Y. Learning latent structure: carving nature at its joints. Curr. Opin. Neurobiol. 2010;20:251–256. doi: 10.1016/j.conb.2010.02.008.
- Kemp, C. & Tenenbaum, J. B. The discovery of structural form. Proc. Natl Acad. Sci. USA105, 10687–10692 (2008).
- Girvan M, Newman MEJ. Community structure in social and biological networks. Proc. Natl Acad. Sci. USA. 2002;99:7821–7826. doi: 10.1073/pnas.122653799.
- Collins AM, Quillian MR. Retrieval time from semantic memory. J. Verbal Learning Verbal Behav. 1969;8:240–247. doi: 10.1016/S0022-5371(69)80069-1.
- Zambaldi, V. et al. Relational deep reinforcement learning. arXiv: 1806.01830v2 [cs.LG] (2018).
- Ferguson, K. & Mahadevan, S. Proto-transfer learning in markov decision processes using spectral methods. Proc. ICML Work. Struct. Knowl. Transf. Mach. Learn. 151 (2006).
- Taylor ME, Stone P. Transfer learning for reinforcement learning domains: a survey. J. Mach. Learn. Res. 2009;10:1633–1685.
- Konidaris G, Scheidwasser I, Barto AG. Transfer in reinforcement learning via shared features. J. Mach. Learn. Res. 2012;13:1333–1371.
- Tolman C,E. Introduction and removal of reward, and maze performance in rats. Univ. Calif. Publ. Psychol. 1930;4:257–275.
- Tolman EC, Ritchie BF, Kalish D. Studies in spatial learning: orientation and the short-cut. J. Exp. Psychol. 1946;36:13–24. doi: 10.1037/h0053944.
- Tenenbaum JB, Kemp C, Griffiths TL, Goodman ND. How to grow a mind: statistics, structure, and abstraction. Science. 2011;331:1279–1285. doi: 10.1126/science.1192788.
- Saxe, A. M., Mcclelland, J. L. & Ganguli, S. A mathematical theory of semantic development in deep neural networks. 10.1073/pnas.1820226116 (2019).
- Mahadevan S, Maggioni M. Proto-value functions: a laplacian framework for learning representation and control in markov decision processes. J. Mach. Learn. Res. 2007;8:2169–2231.
- Roweis S, Ghahramani Z. A unifying review of linear gaussian models. Neural Comput. 1999;11:305–345. doi: 10.1162/089976699300016674.
- Rasmussen, C. E. & Ghahramani, Z. Occam’s Razor. Advances in Neural Information Systems 13 (2001).
- Schapiro, A. C., Rogers, T. T., Cordova, N. I., Turk-Browne, N. B. & Botvinick, M. M. Neural representations of events arise from temporal community structure. Nat.Neurosci.16, 486–492 (2013).
- Garvert MM, Dolan RJ, Behrens TEJ. A map of abstract relational knowledge in the human hippocampal–entorhinal cortex. Elife. 2017;6:1–20. doi: 10.7554/eLife.17086.
- Gelman, A., Carlin, J. B., Stern, H. S., Rubin, D. B. & Raton London New York Washington, B. Bayesian Data Analysis 2nd edn (2013).
- Dordek, Y., Soudry, D., Meir, R. & Derdikman, D. Extracting grid cell characteristics from place cell inputs using non-negative principal component analysis. 10.7554/eLife.10094.001 (2016).
- Whittington, J. C. et al. The Tolman-Eichenbaum Machine: Unifying space and relational memory through generalisation in the hippocampal formation. bioRxiv10.1101/770495 (2019).
- Wang, J. X. et al. Prefrontal cortex as a meta-reinforcement learning system. Nat. Neurosci. 10.1038/s41593-018-0147-8 (2018).
- Harlow HF. The formation of learning sets. Psychol. Rev. 1949;56:51–65. doi: 10.1037/h0062474.
- Spectral Graph Theory - Fan R. K. Chung, Fan Chung Graham - Google Books. (2019).
- Hafting T, Fyhn M, Molden S, Moser MB, Moser EI. Microstructure of a spatial map in the entorhinal cortex. Nature. 2005;436:801–806. doi: 10.1038/nature03721.
- Behrens TEJ, et al. What is a cognitive map? Organizing knowledge for flexible behavior. Neuron. 2018;100:490–509. doi: 10.1016/j.neuron.2018.10.002.
- Baram, A. B., Muller, T. H., Whittington, J. C. R. & Behrens, T. E. J. Intuitive planning: global navigation through cognitive maps based on grid-like codes. bioRxiv10.1101/421461 (2018).
- Aronov D, Nevers R, Tank DW. Mapping of a non-spatial dimension by the hippocampal-entorhinal circuit. Nature. 2017;543:719–722. doi: 10.1038/nature21692.
- Bao X, et al. Grid-like neural representations support olfactory navigation of a two-dimensional odor space. Neuron. 2019;102:1066–1075. doi: 10.1016/j.neuron.2019.03.034.
- Constantinescu AO, O’Reilly JX, Behrens TEJ. Organizing conceptual knowledge in humans with a gridlike code. Science. 2016;352:1464–1468. doi: 10.1126/science.aaf0941.
- Yoon, K. et al. Specific evidence of low-dimensional continuous attractor dynamics in grid cells. 10.1038/nn.3450 (2013).
- Burak, Y. & Fiete, I. R. Accurate path integration in continuous attractor network models of grid cells. PLoS Comput. Biol. 5, e1000291 (2009).
- Gardner, R. J., Lu, L., Wernle, T., Moser, M.-B. & Moser, E. I. Correlation structure of grid cells is preserved during sleep. 10.1038/s41593-019-0360-0 (2019).
- Trettel, S. G., Trimper, J. B., Hwaun, E., Fiete, I. R. & Colgin, L. L. Grid cell co-activity patterns during sleep reflect spatial overlap of grid fields during active behaviors. 10.1038/s41593-019-0359-6 (2019).
- Fyhn M, Hafting T, Treves A, Moser MB, Moser EI. Hippocampal remapping and grid realignment in entorhinal cortex. Nature. 2007;446:190–194. doi: 10.1038/nature05601.
- Solstad T, Boccara CN, Kropff E, Moser MB, Moser EI. Representation of geometric borders in the entorhinal cortex. Science. 2008;322:1865–1868. doi: 10.1126/science.1166466.
- Baum L. An inequality and associated maximization technique in statistical estimation of probabilistic functions of a Markov process. Inequalities. 1972;3:1–8.
- Gershman, S. J., Moore, C. D., Todd, M. T., Norman, K. A. & Sederberg, P. B. The successor representation and temporal context. Neural Comput.24, 1553–1568 (2012).
Source: PubMed