Dissociable neural representations of reinforcement and belief prediction errors underlie strategic learning

“Such a functional division is consistent with previous proposals in the mentalization literature (18) and may re fl ect differences in the type of information being learned.”

“The responses in the ACCg to the value of behaviors for others and to PEs that were discussed in earlier sections approximate the signals predicted by the MoVE model (Lockwood et al., 2015; Apps and Ramnani, 2014; Zhu et al., 2012; Apps et al., 2012, 2013a, 2015a).”

“…stimuli (e.g., smiling faces) as reward, or learning about reward based on observation or consideration of others’ experiences (Lin et al., 2012; Zhu et al., 2012; Zaki and Mitchell, 2011; Poore et al., 2012; Jones et al., 2011; Izuma et al., 2008; Chang and Sanfey, 2013; see Dunne and…”

“Indeed, studies in primates support the idea that competition triggers model-based counterprediction rather than stochastic choice (Abe and Lee, 2011; Zhu et al., 2012).”

“Receptors that have been identified to serve as host ligands include E-cadherin on epithelial cells, N-cadherin on endothelial cells and the receptor tyrosine kinases, EGF receptor (EGFR) and HEGF receptor 2 (HER2) (Phan et al., 2007; Sun et al., 2010; Zhu et al., 2012).”

“Consistently, the vmPFC is involved in predicting action outcomes [18–21,22 ], suggesting that the vmPFC encodes actionoutcome associations for selecting actions according to reward values.”

“…behavior has been shown previously in a variety of domains, not only reward approach and punishment avoidance, but also the monitoring of perceptual features or social representations (Behrens et al., 2008; Preuschoff et al., 2008; Law and Gold, 2009; den Ouden et al., 2009; Zhu et al., 2012).”

“…2019),learning ‘from’ others (Behrens et al., 2008; Burke et al., 2010; Suzuki et al., 2012; Hill et al., 2017; Lindström et al. , 2019a,b; Zhang and Gläscher, 2020) and learning ‘about’ others (Hampton et al., 2008; Zhu et al., 2012; Will et al., 2017; Lockwood et al., 2018; Yoon et al., 2018).”

“The results from these studies have demonstrated that both humans and animals apply a combination of model-free and model-based reinforcement learning algorithms (Camerer and Ho, 1999; Camerer, 2003; Lee, 2008; Abe et al., 2011; Zhu et al., 2012).”

“Similarly, in a competitive game, selfreferenced reward outcome prediction errors correlate with activity in the ventral striatum, but, critically, belief-based prediction errors about the competitive partners action are encoded in rostral ACC (rACC).(90) Furthermore, in a social decision-making task involving utilizi…”

“Event-related potential (ERP) and neuroimaging studies have shaped our view of how executive mechanisms monitor our behavior and respond to errors so we can improve information processing (Ridderinkhof et al., 2004; Brown and Braver, 2005; Gehring et al., 2012; Sheth et al., 2012; Zhua et al., 2012).”

“…motor performance is impaired and movements become less automatized (Lang and Bastian, 2002), gray matter volume increase in the anterior cingulate gyrus may represent increased attention to the task performance and/ or increased use of cognitive strategies (Rossi et al., 2009; Zhu et al., 2012).”

“…anuscripts E urope PM C Funders A uthor M anuscripts are violated (Apps et al. 2015), whether rewarding outcomes are being delivered to others (Apps et al. 2013a, Zhu et al. 2012), or when painful stimulation is predicted for or delivered to another person (Lamm et al. 2011, Lockwood et al. 2013).”

“Similarly, mPFC activation is involved in strategic learning related to the opponent’s beliefs in an economic decision making game (Zhu et al. 2012).”

“This corresponds to studies in neurotypical cohorts showing that variability in the strength of ACCg signals correlated with social value (Behrens et al., 2008), social group size (Sallet et al., 2011), and the ability to track another individuals actions (Zhu et al., 2012).”

“In a multi-strategy competitive learning task called the “ patent race game ” , a hybrid model integrating both RL and social belief inference best explained behavior (Zhu, Mathewson, & Hsu, 2012).”

“…models from psychology (Rescorla and Wagner, 1972), neuroscience (Montague et al., 1996), and economic exchange games (Camerer and Ho, 1998; Zhu et al., 2012), we investigated behavior in this task using a reinforcement learning (RL) model (Sutton and Barto, 1998; McClure and van den Bos,…”

“Additionally, the rostral ACC has been implicated in belief-based prediction errors during a competitive task (Zhu et al., 2012).”

“the fl ip side, informing the model of an individual ’ s social priors or beliefs about the world [16,90] can speed the learning rate [77,91] and these models fair far better in describing how people actually learn in strategic situations [61].”

“, 2013), and neuroimaging studies have shown that the ACC processes predictions about the value of others’ actions (Behrens et al., 2008; Jones et al., 2011; Zhu et al., 2012; Apps et al., 2013b; Boorman et al., 2013; Apps and Ramnani, 2014).”

“Finally, activation in the rostral ACC correlates with a belief-based prediction error, which is driven by the discrepancy between the anticipation and action of others (30).”

“This region is also implicated in decision processes (Litt et al., 2011; Zhu et al., 2012), although our focus exhibited a greater negative BOLD response for the spatial than the temporal task, with greater threshold change correlating with a decrease of this negative response.”

“…model in the context of prosocial behaviors comes from studies showing that outcomes of social interactions in economic games (Rilling et al., 2004; Zhu et al., 2012), violations of social norms (Klucharev et al., 2009; Xiang et al., 2013), signs of social approval (Jones et al., 2011) and even…”

“In the theory of mind, this is referred to by individuals’ general ability for mentally simulating other people’s mode of reasoning (Hsu & Zhu, 2012).40 40 For a systematic confrontation between theory of mind and game theory, see Schmidt & Livet (2014).”

“Here we address this question by characterizing the computational mechanisms underlying norm-guided behavior in third parties (Behrens et al., 2009; Hsu et al., 2008; Zhu et al., 2012).”

“Confirming this model, studies show that both types of prediction errors are encoded simultaneously by different neural structures (first-order beliefs in the dorsal anterior cingulate cortex (dACC) [41] and second-order beliefs in the temporoparietal junction (TPJ)).”

“Motivated by our previous neuroimaging findings suggesting PFC involvement in belief learning (5), we first examined genes known to regulate prefrontal dopamine levels.”

“dence} × 2{intention type} ANOVA, main effect of the type of intention: (F(1,17) = 8.”

“Activation related to the learning process specified by the WRM model and the RL model was observed (and mostly overlapped) in the VS and the VMPFC, the reward regions identified by previous studies (O’Doherty et al., 2003; Haruno et al., 2004; Rodriguez et al., 2006; Tobler et al., 2006; Jocham et al., 2011; Zhu et a…”

“Nevertheless recent model-based fMRI studies have hypothesized learning mechanisms based on reinforcement learning and beliefbased models (Behrens et al., 2008; Hampton et al., 2008; Haruno and Kawato, 2009; Zhu et al., 2012; Fouragnan et al., 2013; van den Bos et al., 2013).”

“In economics, for a long time there was no axiomatic (epistemic) basis for Nash equilibrium (until Aumann and Brandenburger 1995). Informational efficiency of stock prices, a hypothesis which drove empirical finance for decades starting in 1965 with Gene Fama’s dissertation, did not have clear formal basis until Gross…”

“Learning mechanisms Levels of strategic thinking Neural correlates Belief Prediction error (Zhu et al, 2012) mPFC (Hampton et al, 2008) Low level k=1”

“These models have been repeatedly applied to reward-based learning [46,47] as well as social interactions [48,49].”

“These regions are mainly responsible for social learning processes (King-Casas et al., 2005; Behrens et al., 2008; Hampton et al., 2008; Ho et al., 2008; Zhu et al., 2012).”

“, 2017) and other traits (Hackel et al., 2015; Harris and Fiske, 2010; Zhu et al., 2012) revealed neural activities similar to those in non-social reward learning (for reviews see Chib et al.”

“Theselimitationsexplain why agents basing their behavior only on standard RL models can be exploited by opponents using more sophisticated algorithms 6,29 .”

“Moreover, we used a well-established experimental setup with a population of many anonymously interacting listeners and speakers with low probability of reencounter (18, 23, 26).”

“Efficient learning mechanisms hold in the social domain and are implemented in the medial prefrontal cortex”

“reinforcement learning [15] and belief learning [16] that has been extensively explored in the field of behavioral economics and rather successful in explaining the interactive learning of humans in games [17, 18].”

“…two key parameters capturing (1) the degree to which players are sensitive to actions of others, captured by δ , and (2) learning rate or sensitivity of players to more recent observations relative to past ones, captured by ρ ( Figure 3B ) (Sutton and Barto, 1998; Camerer, 2003; Zhu et al., 2012).”

“Computational modelling using a cognitive hierarchies framework (figure 3 a ) [78 – 80] revealed that individuals engage in more sophisticated reasoning about their rival during attack (versus defence) [62,81].”

“Future trials must incorporate methodologies to investigate any relationship between the subjective pain measures observed in the sham group with their associated motivational or belief neural network (Taylor et al., 2004; Wiech et al., 2008; Zhu et al., 2012; Sterpenich et al., 2014).”

“Examples include subcortical structures such as the striatum (O’Doherty et al., 2003; Seymour et al., 2004; Pessiglione et al., 2006; Glascher et al., 2010; Jocham et al., 2011; Zhu et al., 2012; Eppinger et al., 2013) and nucleus accumbens (Niv et al., 2012).”

“…may share similar Bayesian inference mechanisms (Ahn, Krawitz, Kim, Busemeyer, & Brown, 2013; Aitchison & Lengyel, 2017; Baker & Tenenbaum, 2014; Behrens, Hunt, Woolrich, & Rushworth, 2008; Devaine, Hollard, & Daunizeau, 2014; Khalvati et al., 2019; Robalino & Robson, 2012; Zhu et al., 2012).”

“4a (peak x, y, z = 60, 11, 23).”

“This is in line with the findings through fMRI that there may exist significant tissue loss in gray matter volume in mPFC, which is indicated to be involved in belief-based learning in our previous study (Zhu et al., 2012).”

“Via their web browser, all participants completed the Liebowitz Social Anxiety Scale (LSAS), Liebowitz, 1987), an abbreviated nine-item Ravens’ matrix test to approximate IQ (Bilker et al., 2012), and 80 rounds of a patent race game (Zhu et al., 2012).”

“Accordingly, it has been implicated in strategic reasoning in complex social games [37], and the representation of belief-based prediction errors [3].”

“Our data are also consistent with previous evidence implicating the ACC and DLPFC in strategic decision making based on the response patterns of other agents [Steinbeis et al., 2012; Zhu et al., 2012].”

“Further, mPFC carries value signals specifically related to beliefbased value learning (Burke et al., 2010; Zhu et al., 2012), and associative learning of the reliability of social signals (Behrens et al., 2008).”

“This hybrid approach has been extensively explored in the field of behavioral economics and it has been rather successful in explaining the interactive learning of humans in games [11,12,15,16].”

“Studies in animal models have shown that a competitive setting triggers strategic counterprediction, rather than stochastic behavior, even when acting as random as possible maximizes reward 36–38 .”

“…has been associated with value computation in nonstrategic decision contexts ranging from primary to social rewards for both self and others (Nicolle et al., 2012; Bartra et al., 2013; Clithero and Rangel, 2013) and in choices during competitive games (Hampton et al., 2008; Zhu et al., 2012).”

“…plays a key role in representing the expected value of stimuli (Kuzmanovic et al., 2018), whereas the ventral striatum, which is involved in reward processing, tracks prediction errors related to rewards and punishments, prompting the brain to adjust its expectations accordingly (Zhu et al., 2012).”

“We approached this issue with a computational framework modeling reward and risk prediction errors based on k-level reasoning in belief formation (Zhu et al., 2012; Camerer et al., 2004; Nagel, 1995).”

“…models ( 77 , 78 ) where people use limited steps of thinking, Experience Weighted Attraction where people learn strategies based on pre-dispositions and payoff experience ( 79 , 80 ), and Quantal Response Equilibrium where people are noisy and believe that others are noisy as well ( 81 ).”

“, instead of fixed pairing, repairing participants randomly on a trial-by-trial basis in learning experiments), strategic learning in competitive scenarios sometimes involves a hybrid of belief and reward learning, rather than mental-state learning (Zhu et al., 2012).”

“" Here we review recent efforts, including both our own work (Zhu et al. 2012) and those of others (Behrens et al. 2008; Hampton et al. 2008), toward a more quantitative, model-building approach to understand the neural underpinnings of strategic behavior by studying the adaptation process itself.”

“In social tasks, striatal activation has also been observed in social learning and implicated in social prediction error and belief updating ( Behrens et al., 2008 ; Hampton et al., 2008 ; Hertz et al., 2017 ; Xiang et al., 2012 ; Zhu et al., 2012 ).”