Cortical substrates for exploratory decisions in humans
Nathaniel D Daw, John P O'Doherty, Peter Dayan, Ben Seymour, Raymond J Dolan, Nathaniel D Daw, John P O'Doherty, Peter Dayan, Ben Seymour, Raymond J Dolan
Abstract
Decision making in an uncertain environment poses a conflict between the opposing demands of gathering and exploiting information. In a classic illustration of this 'exploration-exploitation' dilemma, a gambler choosing between multiple slot machines balances the desire to select what seems, on the basis of accumulated experience, the richest option, against the desire to choose a less familiar option that might turn out more advantageous (and thereby provide information for improving future decisions). Far from representing idle curiosity, such exploration is often critical for organisms to discover how best to harvest resources such as food and water. In appetitive choice, substantial experimental evidence, underpinned by computational reinforcement learning (RL) theory, indicates that a dopaminergic, striatal and medial prefrontal network mediates learning to exploit. In contrast, although exploration has been well studied from both theoretical and ethological perspectives, its neural substrates are much less clear. Here we show, in a gambling task, that human subjects' choices can be characterized by a computationally well-regarded strategy for addressing the explore/exploit dilemma. Furthermore, using this characterization to classify decisions as exploratory or exploitative, we employ functional magnetic resonance imaging to show that the frontopolar cortex and intraparietal sulcus are preferentially active during exploratory decisions. In contrast, regions of striatum and ventromedial prefrontal cortex exhibit activity characteristic of an involvement in value-based exploitative decision making. The results suggest a model of action selection under uncertainty that involves switching between exploratory and exploitative behavioural modes, and provide a computationally precise characterization of the contribution of key decision-related brain systems to each of these functions.
Figures
![Figure 1](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/2635947/bin/ukmss-3671-f0001.jpg)
![Figure 2](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/2635947/bin/ukmss-3671-f0002.jpg)
![Figure 3](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/2635947/bin/ukmss-3671-f0003.jpg)
Figure 4
Exploration-related activity in intraparietal sulcus.…
Figure 4
Exploration-related activity in intraparietal sulcus. a, Regions of left and right intraparietal sulcus…
- Neuroeconomics: best to go with what you know?Lee D. Lee D. Nature. 2006 Jun 15;441(7095):822-3. doi: 10.1038/441822a. Nature. 2006. PMID: 16778879 No abstract available.
- The neurocomputational bases of explore-exploit decision-making.Hogeveen J, Mullins TS, Romero JD, Eversole E, Rogge-Obando K, Mayer AR, Costa VD. Hogeveen J, et al. Neuron. 2022 Jun 1;110(11):1869-1879.e5. doi: 10.1016/j.neuron.2022.03.014. Epub 2022 Apr 6. Neuron. 2022. PMID: 35390278
- Transcranial Stimulation over Frontopolar Cortex Elucidates the Choice Attributes and Neural Mechanisms Used to Resolve Exploration-Exploitation Trade-Offs.Raja Beharelle A, Polanía R, Hare TA, Ruff CC. Raja Beharelle A, et al. J Neurosci. 2015 Oct 28;35(43):14544-56. doi: 10.1523/JNEUROSCI.2322-15.2015. J Neurosci. 2015. PMID: 26511245 Free PMC article.
- Primate Orbitofrontal Cortex Codes Information Relevant for Managing Explore-Exploit Tradeoffs.Costa VD, Averbeck BB. Costa VD, et al. J Neurosci. 2020 Mar 18;40(12):2553-2561. doi: 10.1523/JNEUROSCI.2355-19.2020. Epub 2020 Feb 14. J Neurosci. 2020. PMID: 32060169 Free PMC article.
- Reward-dependent learning in neuronal networks for planning and decision making.Dehaene S, Changeux JP. Dehaene S, et al. Prog Brain Res. 2000;126:217-29. doi: 10.1016/S0079-6123(00)26016-0. Prog Brain Res. 2000. PMID: 11105649 Review.
- Choice, uncertainty and value in prefrontal and cingulate cortex.Rushworth MF, Behrens TE. Rushworth MF, et al. Nat Neurosci. 2008 Apr;11(4):389-97. doi: 10.1038/nn2066. Epub 2008 Mar 26. Nat Neurosci. 2008. PMID: 18368045 Review.
- Adaptive tuning of human learning and choice variability to unexpected uncertainty.Lee JK, Rouault M, Wyart V. Lee JK, et al. Sci Adv. 2023 Mar 29;9(13):eadd0501. doi: 10.1126/sciadv.add0501. Epub 2023 Mar 29. Sci Adv. 2023. PMID: 36989365 Free PMC article.
- Individuals with problem gambling and obsessive-compulsive disorder learn through distinct reinforcement mechanisms.Suzuki S, Zhang X, Dezfouli A, Braganza L, Fulcher BD, Parkes L, Fontenelle LF, Harrison BJ, Murawski C, Yücel M, Suo C. Suzuki S, et al. PLoS Biol. 2023 Mar 14;21(3):e3002031. doi: 10.1371/journal.pbio.3002031. eCollection 2023 Mar. PLoS Biol. 2023. PMID: 36917567 Free PMC article.
- Diminished reinforcement sensitivity in adolescence is associated with enhanced response switching and reduced coding of choice probability in the medial frontal pole.Waltmann M, Herzog N, Reiter AMF, Villringer A, Horstmann A, Deserno L. Waltmann M, et al. Dev Cogn Neurosci. 2023 Mar 7;60:101226. doi: 10.1016/j.dcn.2023.101226. Online ahead of print. Dev Cogn Neurosci. 2023. PMID: 36905874 Free PMC article.
- Zoom behavior during visual search modulates pupil diameter and reflects adaptive control states.Brunyé TT, Drew T, Kerr KF, Shucard H, Powell K, Weaver DL, Elmore JG. Brunyé TT, et al. PLoS One. 2023 Mar 9;18(3):e0282616. doi: 10.1371/journal.pone.0282616. eCollection 2023. PLoS One. 2023. PMID: 36893083 Free PMC article.
- Local and global reward learning in the lateral frontal cortex show differential development during human adolescence.Wittmann MK, Scheuplein M, Gibbons SG, Noonan MP. Wittmann MK, et al. PLoS Biol. 2023 Mar 2;21(3):e3002010. doi: 10.1371/journal.pbio.3002010. eCollection 2023 Mar. PLoS Biol. 2023. PMID: 36862726 Free PMC article.
- Research Support, Non-U.S. Gov't
- Choice Behavior / physiology
- Decision Making / physiology*
- Exploratory Behavior / physiology
- Gambling
- Humans
- Magnetic Resonance Imaging
- Models, Neurological
- Models, Psychological
- Prefrontal Cortex / physiology*
- Reward
- Uncertainty*
- Full Text Sources
- Other Literature Sources
- Medical
![Figure 4](https://www.ncbi.nlm.nih.gov/pmc/articles/instance/2635947/bin/ukmss-3671-f0004.jpg)
Source: PubMed