A bayesian foundation for individual learning under uncertainty

Abstract

Computational learning models are critical for understanding mechanisms of adaptive behavior. However, the two major current frameworks, reinforcement learning (RL) and Bayesian learning, both have certain limitations. For example, many Bayesian models are agnostic of inter-individual variability and involve complicated integrals, making online learning difficult. Here, we introduce a generic hierarchical Bayesian framework for individual learning under multiple forms of uncertainty (e.g., environmental volatility and perceptual uncertainty). The model assumes Gaussian random walks of states at all but the first level, with the step size determined by the next highest level. The coupling between levels is controlled by parameters that shape the influence of uncertainty on learning in a subject-specific fashion. Using variational Bayes under a mean-field approximation and a novel approximation to the posterior energy function, we derive trial-by-trial update equations which (i) are analytical and extremely efficient, enabling real-time learning, (ii) have a natural interpretation in terms of RL, and (iii) contain parameters representing processes which play a key role in current theories of learning, e.g., precision-weighting of prediction error. These parameters allow for the expression of individual differences in learning and may relate to specific neuromodulatory mechanisms in the brain. Our model is very general: it can deal with both discrete and continuous states and equally accounts for deterministic and probabilistic relations between environmental events and perceptual states (i.e., situations with and without perceptual uncertainty). These properties are illustrated by simulations and analyses of empirical time series. Overall, our framework provides a novel foundation for understanding normal and pathological learning that contextualizes RL within a generic Bayesian scheme and thus connects it to principles of optimality from probability theory.

Keywords: acetylcholine; decision-making; dopamine; hierarchical models; neuromodulation; serotonin; variational Bayes; volatility.

Figures

Figure 1

Overview of the hierarchical generative model. The probability at each level is determined by the variables and parameters at the next highest level. Note that further levels can be added on top of the third. These levels relate to each other by determining the step size (volatility or variance) of a random walk. The topmost step size is a constant parameter ϑ. At the first level, x1 determines the probability of the input u.

Figure 2

Generative model and posterior distributions on hidden states. Left: schematic representation of the generative model as a Bayesian network. x1(k), x2(k), x3(k) are hidden states of the environment at time point k. They generate u(k), the input at time point k, and depend on their immediately preceding values x2(k−1), x3(k−1) and the parameters ϑ, ω, κ. Right: the minimal parametric description q(x) of the posteriors at each level. The distribution parameters μ (posterior expectation) and σ (posterior variance) can be found by approximating the minimal parametric posteriors to the mean-field marginal posteriors. For multidimensional states x, μ is a vector and σ a covariance matrix.

Figure 3

Quadratic approximation to the variational energy. Approximating the variational energy I(x) (blue) by a quadratic function leads (by exponentiation) to a Gaussian posterior. To find our approximation Ĩ(x) (red), we expand I(x) to second order at the preceding posterior expectation μ(k−1). The argmax of Ĩ(x) is then the new posterior expectation μ(k). This generally leads to a different result from the Laplace approximation (dashed), but there is a priori no reason to regard either approximation as more exact than the other.

Figure 4

Interpretation of the variational update equations in terms of Rescorla–Wagner learning. The Rescorla–Wagner update is Δprediction = learning rate × prediction error. Our expectation update equations can be interpreted in these terms.

Figure 5

Reference scenario: ϑ = 0.5, ω = −2.2, κ = 1.4. A simulation of 320 trials. Bottom: the first level. Input u is represented by green dots. In the absence of perceptual uncertainty, this corresponds to x1. The fine black line is the true probability (unknown to the agent) that x1 = 1. The red line shows s(μ2); i.e., the agent's posterior expectation that x1 = 1. Given the input and update rules, the simulation is uniquely determined by the value of the parameters ϑ, ω, and κ. Middle: the second level with the posterior expectation μ2 of x2. Top: the third level with the posterior expectation μ3 of x3. In all three panels, the initial values of the various μ are indicated by circles at trial k = 0.

Figure 6

Reduced ϑ = 0.05 (unchanged ω = −2.2, κ = 1.4). Symbols have the same meaning as in Figure 5. Here, the expected x3 is more stable. The learning rate in x2 is initially unaffected but owing to more stable x3 it no longer increases after the period of increased volatility.

Figure 7

Reduced ω = −4 (unchanged ϑ = 0.5, κ = 1.4). Symbols have the same meaning as in Figure 5. The small learning rate in x2 leads to an extremely stable expected x3. Despite prediction errors, the agent makes only small updates to its beliefs about its environment.

Figure 8

Reduced κ = 0.2 (unchanged ϑ = 0.5, ω = −2.2). Symbols have the same meaning as in Figure 5. x2 and x3 are only weakly coupled. Despite uncertainty about x3, only small updates to μ3 take place. Sensitivity to changes in volatility is reduced. x2 is not affected directly, but its learning rate does not increase with volatility.

Figure 9

Simulations including standard deviations of posterior distributions. Top to bottom: the four scenarios from Figure 5–8. Left: μ2 (bold red line); fine red lines indicate the range of ±σ2 around μ2. Right: μ3 (bold blue line); fine blue lines indicate the range ±σ3 of around μ3. Circles indicate initial values.

Figure 10

A simulation where risk is constant (ϑ = 0.2, ω = −2.3, κ = 1.6). Symbols have the same meanings as in Figure 5. The same basic phenomena shown in Figure 5 can be observed here.

Figure 11

Inference on a continuous-valued state (ϑ = 0.3, ω = −12, κ = 1, α = 2·10−5). Reference scenario for the model of hierarchical Gaussian random walks applied to a continuous-valued state at the bottom level. The state is the value x2 of the U.S. Dollar against the Swiss Franc during the first 180 trading days of the year 2010. Bottom panel: input u representing closing exchange rates (green dots). The bold red line surrounded by two fine red lines indicates the range μ2±σ2. Top panel: The range μ3±σ3 of the log-volatility x3 of x2.

Figure 12

Reduced α = 10−6 (ϑ = 0.3, ω = −12, κ = 1). Reduced perceptual uncertainty α with respect to the reference scenario of Figure 11.

Figure 13

Increased α = 10−4 (ϑ = 0.3, ω = −12, κ = 1). Increased perceptual uncertainty a with respect to the reference scenario of Figure 11.

Figure 14

Reduced ϑ = 0.01 (ω = −12, κ = 1, α = 2·10−5). Reduced ϑ with respect to the reference scenario of Figure 11.