A causal role for right frontopolar cortex in directed, but not random, exploration

Abstract

The explore-exploit dilemma occurs anytime we must choose between exploring unknown options for information and exploiting known resources for reward. Previous work suggests that people use two different strategies to solve the explore-exploit dilemma: directed exploration, driven by information seeking, and random exploration, driven by decision noise. Here, we show that these two strategies rely on different neural systems. Using transcranial magnetic stimulation to inhibit the right frontopolar cortex, we were able to selectively inhibit directed exploration while leaving random exploration intact. This suggests a causal role for right frontopolar cortex in directed, but not random, exploration and that directed and random exploration rely on (at least partially) dissociable neural systems.

Keywords: decision making; directed exploration; explore-exploit; frontal pole; human; neuroscience; random exploration; reinforcement learning.

Conflict of interest statement

No competing interests declared.

Figures

Figure 1.. The horizon task.

Participants make a series of decisions between two one-armed bandits that pay out probabilistic rewards with unknown means. At the start of each game, ‘forced-choice’ trials give participants partial information about the mean of each option. We use the forced-choice trials to set up one of two information conditions: (A) an unequal (or [1 3]) condition in which participants see 1 play from one option and 3 plays from the other and (B) an equal (or [2 2]) condition in which participants see 2 plays from both options. A model-free measure of directed exploration is then defined as the change in information seeking with horizon in the unequal condition (A). Likewise a model-free measure of random exploration is defined as the change choosing the low mean option in the equal condition (B).

Figure 2.. The reward-information confound.

The y-axis corresponds to the correlation between the sign of the difference in mean (sgn(μleft−μright)) between options and the sign of difference in the number of times each option has been played (sgn(nleft−nright)). The forced trials are chosen such that the the correlation is approximately zero on the first free-choice trial. After the first trial, however, a positive correlation quickly emerges as participants choose the more rewarding options more frequently. This strong confound between reward and information makes it difficult to dissociate directed and random exploration on later trials.

Figure 3.. Model-free analysis of the first free-choice trial shows that RPFC stimulation affects directed, but not random, exploration.

(A) In the control (vertex) condition, information seeking increases with horizon, consistent with directed exploration. When RFPC is stimulated, directed exploration is reduced, an effect that is entirely driven by changes in horizon 6 (* denotes p<0.02 and ** denotes p<0.005; error bars are ± s.e.m.). (B) Random exploration increases with horizon but is not affected by RFPC stimulation.

Figure 4.. Model-based analysis of the first free-choice trial showing the effect of RFPC stimulation on each of the 13 parameters.

Left column: Posterior distributions over each parameter value for RFPC and vertex stimulation condition. Right column: posterior distributions over the change in each parameter between stimulation conditions. Note that, because information bonus, decision noise and spatial bias are all in units of points, we plot them on the same scale to facilitate comparison of effect size.

Figure 5.. Correlation between TMS-induced changes in information bonus, A, and TMS-induced changes in the prior mean, R0.

(A, B) Samples from the posterior distributions over the TMS-related changes in prior mean, R0, and TMS-related change in information bonus in horizon 1 (A) and horizon 6 (B). In both cases we see a negative correlation between the change in R0 and the change in A consistent with a tradeoff between these variables in the model. (C) Samples from the posterior over the effect of TMS stimulation on the horizon-related change in information bonus, ΔA=A(h=6)-A(h=1) plotted against samples from the TMS-related change in prior mean. Here we see no correlation between variables and the majority of ΔA(vertex)−ΔA(RFPC) samples below zero consistent with an effect of RFPC stimulation on directed exploration.

Figure 6.. Model-free analysis of all trials.

(A, B) Model-free measures of directed (A) and random (B) exploration as a function of trial number suggests a reduction in both directed and random exploration over the course of the game. (C, D) TMS-induced change in measures of directed and random exploration as a function of trial number. This suggests that the reduction in directed exploration on the first free-choice trial, persists into the second trial of the game.

Figure 7.. Correlation between individual differences in the levels of directed and random exploration in a sample of 277 people performing the Horizon Task.

Figure 8.. Graphical representation of the model.

Each variable is represented by a node, with edges denoting the dependence between variables. Shaded nodes correspond to observed variables, that is, the free choices cτ⁢s⁢h⁢u⁢g, forced-trial rewards, 𝐫τ⁢s⁢h⁢u⁢g and forced-trial choices 𝐚τ⁢s⁢h⁢u⁢g. Unshaded nodes correspond to unobserved variables whose values are inferred by the model.

Author response image 1.. No difference in effects between original and replication experiments.

In each panel we plot the model-free measures of directed and random exploration and how they change between stimulation conditions. For example, in Panel A, we plot p(high info) in horizon 1 for vertex stimulation (x-axis) and RFPC stimulation (y-axis). Each point in this plot is a single subject and the diagonal line represents equality. Participants below the diagonal line have a smaller value of p(high info) in the RFPC stimulation condition. From this we can clearly see that there is no effect of RFPC stimulation on directed exploration in horizon 1 (panel A), or random exploration in either horizon (B, D). However, there is a strong effect of RFPC stimulation on directed exploration in horizon 6 with the majority of points lying below the diagonal (C). Moreover, both the original and replication datasets point to the same conclusions in all four panels.

Author response image 2.. Effect of TMS on information bonus in model with bonus proportional to uncertainty.

Author response image 3.. Effect of TMS on decision noise in model in which bonus is a linear function of uncertainty.

Author response image 4.