Bonsai trees in your head: how the pavlovian system sculpts goal-directed choices by pruning decision trees
Quentin J M Huys, Neir Eshel, Elizabeth O'Nions, Luke Sheridan, Peter Dayan, Jonathan P Roiser, Quentin J M Huys, Neir Eshel, Elizabeth O'Nions, Luke Sheridan, Peter Dayan, Jonathan P Roiser
Abstract
When planning a series of actions, it is usually infeasible to consider all potential future sequences; instead, one must prune the decision tree. Provably optimal pruning is, however, still computationally ruinous and the specific approximations humans employ remain unknown. We designed a new sequential reinforcement-based task and showed that human subjects adopted a simple pruning strategy: during mental evaluation of a sequence of choices, they curtailed any further evaluation of a sequence as soon as they encountered a large loss. This pruning strategy was Pavlovian: it was reflexively evoked by large losses and persisted even when overwhelmingly counterproductive. It was also evident above and beyond loss aversion. We found that the tendency towards Pavlovian pruning was selectively predicted by the degree to which subjects exhibited sub-clinical mood disturbance, in accordance with theories that ascribe Pavlovian behavioural inhibition, via serotonin, a role in mood disorders. We conclude that Pavlovian behavioural inhibition shapes highly flexible, goal-directed choices in a manner that may be important for theories of decision-making in mood disorders.
Conflict of interest statement
The authors have declared that no competing interests exist.
Figures
References
- Knuth D, Moore R. An Analysis of Alpha-Beta Pruning. Artif Intell. 1975;6:293–326.
- Bonet B, Geffner H. Proc of 16th Int Conf on Automated Planning and Scheduling. 2006; Cumbria, UK. ICAPS 2006. AAAI Press; 2006. Learning depth-first search: A unified approach to heuristic search in deterministic and non-deterministic settings, and its application to MDPs. pp. 142–151.
- Russell S, Norvig P. Artificial Intelligence: A modern approach. Upper Saddle River, NJ: Prentice Hall; 1995.
- Estes W, Skinner B. Some quantitative aspects of anxiety. J Exp Psychol. 1941;29:390–400.
- Tye NC, Everitt BJ, Iversen SD. 5-hydroxytryptamine and punishment. Nature. 1977;268:741–743.
- Bouton ME. Learning and Behavior: A Contemporary Synthesis. USA: Sinauer; 2006.
- Williams DR, Williams H. Auto-maintenance in the pigeon: sustained pecking despite contingent non-reinforcement. J Exp Anal Behav. 1969;12:511–520.
- Dayan P, Niv Y, Seymour B, Daw ND. The misbehavior of value and the discipline of the will. Neural Netw. 2006;19:1153–1160.
- Bolles RC. Species-specific defense reactions and avoidance learning. Psychol Rev. 1970;77:32–48.
- Soubrié P. Reconciling the role of central serotonin neurons in human and animal behaviour. Behav Brain Sci. 1986;9:319–364.
- Boureau YL, Dayan P. Opponency revisited: competition and cooperation between dopamine and serotonin. Neuropsychopharmacology. 2011;36:74–97.
- Cools R, Roberts AC, Robbins TW. Serotoninergic regulation of emotional and behavioural control processes. Trends Cogn Sci. 2008;12:31–40.
- Dayan P, Huys QJM. Serotonin in affective control. Annu Rev Neurosci. 2009;32:95–126.
- Crockett MJ, Clark L, Robbins TW. Reconciling the role of serotonin in behavioral inhibition and aversion: acute tryptophan depletion abolishes punishment-induced inhibition in humans. J Neurosci. 2009;29:11993–11999.
- Robinson OJ, Cools R, Sahakian BJ. Tryptophan depletion disinhibits punishment but not reward prediction: implications for resilience. Psychopharmacology (Berl) 2011;219:599–605.
- Tanaka SC, Samejima K, Okada G, Ueda K, Okamoto Y, et al. Brain mechanism of reward prediction under predictable and unpredictable environmental dynamics. Neural Netw. 2006;19:1233–1241.
- Dayan P, Huys QJM. Serotonin, inhibition, and negative mood. PLoS Comput Biol. 2008;4:e4.
- Daw ND, Niv Y, Dayan P. Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci. 2005;8:1704–1711.
- Watkins C, Dayan P. Q-learning. Mach Learn. 1992;8:279–292.
- Tom SM, Fox CR, Trepel C, Poldrack RA. The neural basis of loss aversion in decisionmaking under risk. Science. 2007;315:515–518.
- Pizzagalli DA, Jahn AL, O'Shea JP. Toward an objective characterization of an anhedonic phenotype: a signal-detection approach. Biol Psychiatry. 2005;57:319–327.
- Huys QJM. Reinforcers and control. Towards a computational ætiology of depression [Ph.D. thesis] Gatsby Computational Neuroscience Unit, UCL, University of London; 2007. [ ]
- Huys QJM, Vogelstein J, Dayan P. Psychiatry: Insights into depression through normative decision-making models. In: Koller D, Schuurmans D, Bengio Y, Bottou L, editors. Advances in Neural Information Processing Systems 21. MIT Press; 2009. pp. 729–736.
- Eshel N, Roiser JP. Reward and punishment processing in depression. Biol Psychiatry. 2010;68:118–124.
- Dickinson A, Balleine B. The role of learning in the operation of motivational systems. In: Gallistel R, editor. Stevens' handbook of experimental psychology, volume 3. New York: Wiley; 2002. pp. 497–534.
- Tversky A, Kahneman D. Loss aversion in riskless choice: A reference-dependent model. Q J Econ. 1991;106:1039.
- Guitart-Masip M, Talmi D, Dolan R. Conditioned associations and economic decision biases. Neuroimage. 2010;53:206–214.
- Cipriani A, Furukawa TA, Salanti G, Geddes JR, Higgins JP, et al. Comparative efficacy and acceptability of 12 new-generation antidepressants: a multiple-treatments meta-analysis. Lancet. 2009;373:746–758.
- Geddes JR, Carney SM, Davies C, Furukawa TA, Kupfer DJ, et al. Relapse prevention with antidepressant drug treatment in depressive disorders: a systematic review. Lancet. 2003;361:653–661.
- Caspi A, Sugden K, Moffitt TE, Taylor A, Craig IW, et al. Influence of life stress on depression: moderation by a polymorphism in the 5-HTT genes. Science. 2003;301:386–89.
- Wankerl M, Wst S, Otte C. Current developments and controversies: does the serotonin transporter gene-linked polymorphic region (5-httlpr) modulate the association between stress and depression? Curr Opin Psychiatry. 2010;23:582–587.
- Ansorge MS, Zhou M, Lira A, Hen R, Gingrich JA. Early-life blockade of the 5-HT transporter alters emotional behavior in adult mice. Science. 2004;306:879–881.
- Roiser JP, Blackwell AD, Cools R, Clark L, Rubinsztein DC, et al. Serotonin transporter polymorphism mediates vulnerability to loss of incentive motivation following acute tryptophan depletion. Neuropsychopharmacology. 2006;31:2264–2272.
- Ruhé HG, Mason NS, Schene AH. Mood is indirectly related to serotonin, norepinephrine and dopamine levels in humans: a meta-analysis of monoamine depletion studies. Mol Psychiatry. 2007;12:331–359.
- Varnäs K, Halldin C, Hall H. Autoradiographic distribution of serotonin transporters and receptor subtypes in human brain. Hum Brain Mapp. 2004;22:246–260.
- Pezawas L, Meyer-Lindenberg A, Drabant EM, Verchinski BA, Munoz KE, et al. 5-HTTLPR polymorphism impacts human cingulate-amygdala interactions: a genetic susceptibility mechanism for depression. Nat Neuosci. 2005;8:828–34.
- Clarke HF, Dalley JW, Crofts HS, Robbins TW, Roberts AC. Cognitive inflexibility after prefrontal serotonin depletion. Science. 2004;304:878–880.
- Amat J, Baratta MV, Paul E, Bland ST, Watkins LR, et al. Medial prefrontal cortex determines how stressor controllability affects behavior and dorsal raphe nucleus. Nat Neurosci. 2005;8:365–71.
- Maier SF, Watkins LR. Stressor controllability and learned helplessness: the roles of the dorsal raphe nucleus, serotonin, and corticotropin-releasing factor. Neurosci Biobehav Rev. 2005;29:829–41.
- Robinson OJ, Sahakian BJ. A double dissociation in the roles of serotonin and mood in healthy subjects. Biol Psychiatry. 2009;65:89–92.
- Roiser JP, Blackwell AD, Cools R, Clark L, Rubinsztein DC, et al. Serotonin transporter polymorphism mediates vulnerability to loss ofincentive motivation following acute tryptophan depletion. Neuropsychopharmacology. 2006;31:2264–2272.
- Neumeister A, Konstantinidis A, Stastny J, Schwarz MJ, Vitouch O, et al. Association between serotonin transporter gene promoter polymorphism (5HTTLPR) and behavioral responses to tryptophan depletion in healthy women with and without family history of depression. Arch Gen Psychiatry. 2002;59:613–20.
- Lasa L, Ayuso-Mateos JL, Vzquez-Barquero JL, Dez-Manrique FJ, Dowrick CF. The use of the Beck Depression Inventory to screen for depression in the general population: a preliminary analysis. J Affect Disord. 2000;57:261–265.
- Beck A, Epstein N, Brown G, Steer R, et al. An inventory for measuring clinical anxiety: Psychometric properties. J Consult Clin Psych. 1988;56:893–897.
- Teasdale J. Cognitive vulnerability to persistent depression. Cognition Emotion. 1988;2:247–274.
- Lewinsohn PM, Allen NB, Seeley JR, Gotlib IH. First onset versus recurrence of depression: differential processes of psychosocial risk. J Abnorm Psychol. 1999;108:483–489.
- Kendler KS, Kessler RC, Neale MC, Heath AC, Eaves LJ. The prediction of major depression in women: toward an integrated etiologic model. Am J Psychiatry. 1993;150:1139–1148.
- Beats BC, Sahakian BJ, Levy R. Cognitive performance in tests sensitive to fronal lobe dysfunction in the elderly depressed. Psychol Med. 1996;26:591–603.
- Elliott R, Sahakian BJ, McKay AP, Herrod JJ, Robbins TW, et al. Neuropsychological impairments in unipolar depression: the role of perceived failure on subsequent performance. Psychol Med. 1996;26:975–89.
- Goodwin GM. Neuropsychological and neuroimaging evidence for the involvement of the frontal lobes in depression. J Psychopharmacol. 1997;11:115–122.
- Williams JMG, Barnhofer T, Crane C, Herman D, Raes F, et al. Autobiographical memory specificity and emotional disorder. Psychol Bull. 2007;133:122–148.
- Elliott R, Sahakian BJ, Herrod JJ, Robbins TW, Paykel ES. Abnormal response to negative feedback in unipolar depression: evidence for a diagnosis-specific impairment. J Neurol Neurosurg Psychiatry. 1997;63:74–82.
- Sheehan DV, Lecrubier Y, Sheehan KH, Amorim P, Janavs J, et al. The mini-international neuropsychiatric interview (m.i.n.i.): the development and validation of a structured diagnostic psychiatric interview for dsm-iv and icd-10. J Clin Psychiatry. 1998;59(Suppl 20):22–33;quiz 34–57.
- Spielberger C, Gorsuch R. STAI manual for the State-trait anxiety inventory (form Y) (“self-evaluation questionnaire”) Palo Alto, CA: Consult Psychol Press; 1970.
- Beck A, Steer R, Brown G. Manual for the Beck Depression Inventory-II. San Antonio, TX: Psychological Corporation; 1996.
- Costa P, McCrae R. The NEO PI-R professional manual. Odessa, Florida, USA: Psychological Assessment Resources; 1992.
- Wechsler D. Wechsler Test of Adult Reading Manual. San Antonio, USA: The Psychological Corporation; 2001.
- Wechsler D. Wechsler Adult Intelligence Scale Revised. New York, USA: The Psychological Corporation; 1981.
- Sutton RS, Barto AG. Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press; 1998.
- Huys QJM, Cools R, Glzer M, Friedel E, Heinz A, et al. Disentangling the roles of approach, activation and valence in instrumental and pavlovian responding. PLoS Comput Biol. 2011;7:e1002028.
- MacKay DJ. Information theory, inference and learning algorithms. Cambridge, UK: CUP; 2003.
- Kass R, Raftery A. Bayes factors. J Am Stat Assoc. 1995;90:773–795.
- Devore JL. Probability and Statistics for Engineering and the Sciences. Duxbury Press, 4th edition; 1995.
Source: PubMed