Explicit neural signals reflecting reward uncertainty

Abstract

The acknowledged importance of uncertainty in economic decision making has stimulated the search for neural signals that could influence learning and inform decision mechanisms. Current views distinguish two forms of uncertainty, namely risk and ambiguity, depending on whether the probability distributions of outcomes are known or unknown. Behavioural neurophysiological studies on dopamine neurons revealed a risk signal, which covaried with the standard deviation or variance of the magnitude of juice rewards and occurred separately from reward value coding. Human imaging studies identified similarly distinct risk signals for monetary rewards in the striatum and orbitofrontal cortex (OFC), thus fulfilling a requirement for the mean variance approach of economic decision theory. The orbitofrontal risk signal covaried with individual risk attitudes, possibly explaining individual differences in risk perception and risky decision making. Ambiguous gambles with incomplete probabilistic information induced stronger brain signals than risky gambles in OFC and amygdala, suggesting that the brain's reward system signals the partial lack of information. The brain can use the uncertainty signals to assess the uncertainty of rewards, influence learning, modulate the value of uncertain rewards and make appropriate behavioural choices between only partly known options.

Figures

Figure 1

Expected reward and risk as a function of the probability of reward. Expected reward, measured as mathematical expectation of reward, increases linearly with the probability of reward p (dashed line). Expected reward is minimal at p=0 and maximal at p=1. Risk, measured as reward variance (or as its square root, standard deviation), follows an inverted U function of probability and is minimal at p=0 and 1 and maximal at p=0.5 (solid curve). Reprinted with permission from Preuschoff et al. (2006). Copyright © Cell Press.

Figure 2

Risk signal in dopamine neurons. (a) Phasic reward value signal reflecting reward prediction (left) and more sustained risk signal during the stimulus–reward interval in a single dopamine neuron. Visual stimuli predicting reward probabilities (i) 0.0, (ii) 0.25, (iii) 0.5, (iv) 0.75 and (v) 1.0 alternated semi-randomly between trials. Both rewarded and unrewarded trials are shown at intermediate probabilities; the longer vertical marks in the rasters indicate delivery of the juice reward. (b) Population histograms of responses shown in (a). Histograms were constructed from every trial in 35–44 neurons per stimulus type (638 total trials at p=0 and 1200–1700 trials for all other probabilities). Both rewarded and unrewarded trials are included at intermediate probabilities. (i) 0.0, (ii) 0.25, (iii) 0.5, (iv) 0.75 and (v) 1.0. (c) Median sustained risk-related activation of dopamine neurons as a function of reward probability. Plots show the sustained activation as inverted U function of reward probability, indicating relationship to risk as opposed to value. Data from different stimulus sets and animals are shown separately. Reprinted with permission from Fiorillo et al. (2003). Copyright © American Association for the Advancement of Science.

Figure 3

Risk signals in human ventral striatum. (a) Sustained BOLD response during 6 s correlated with variance as inverted U function of all-or-none reward probability (random effects, p<0.001; L vst, R vst for left, right ventral striatum). (b) Mean activations (parameter estimates beta with standard error) for 10 probabilities. Neural responses in striatum increased towards intermediate probabilities and decreased towards lower and higher probabilities. (i) Left vst and (ii) right vst. Dotted lines indicate best fit (r2=0.88–0.89, p<0.001). Grey data points at p=0.5 indicate late-onset activation between bet and first card when risk is maximal (p=0.5). Error bars=standard error of the mean (s.e.m). Reprinted with permission from Preuschoff et al. (2006). Copyright © Cell Press.

Figure 4

Relation of human orbitofrontal risk signals to individual risk attitude. (a, b) Risk signal in lateral OFC covarying with increasing risk aversion across participants (e.g. a ‘safety’ or ‘fear’ signal). (b) Correlation of contrast estimates of individual participants with their individual risk aversion (p<0.001, r=0.74; unpaired t-test in seven risk seekers and six risk averters). (c, d) Risk signal in medial OFC covarying with risk seeking (=inverse relation to risk aversion; e.g. a ‘risk seeking’ or ‘gambling’ signal). (d) Risk correlation analogous (r=0.85, p<0.0001) to (b). Abscissae in (b, d) show risk aversion as expressed by preference scores (−4 most risk seeking, +4 most risk aversion). To obtain these graphs, we correlated risk-related BOLD responses to individual risk attitude in two steps. First, we determined in each participant the contrast estimates reflecting the goodness of fit between brain activation and risk (variance as inverted U function of probability). Then, we regressed the contrast estimates of all participants to their individual behavioural risk preference scores and identified brain areas showing positive (a) or negative correlations (c). We plotted the regressions of risk aversion against the contrast estimates in (b, d). Reprinted with permission from Tobler et al. (2007). Copyright © The American Physiological Society.

Figure 5

Ambiguity signals in human OFC. (a) Higher BOLD responses in OFC regions to stimuli-predicting ambiguous outcomes compared with risky outcomes, as identified by random effects analysis (p<0.001, uncorrected; 10 voxels; mean from card deck, knowledge and informed opponent situations). (b) Mean time courses of orbitofrontal BOLD responses to onset of stimuli-predicting ambiguous or risky outcomes (dashed vertical lines are mean decision times; error bars=standard error of the mean, s.e.m.; n=16 participants). (i) Left OFC and (ii) right OFC. Reprinted with permission from Hsu et al. (2005). Copyright © American Association for the Advancement of Science.