Vicarious reinforcement in rhesus macaques (Macaca Mulatta)

“Stimuli were controlled by a computer running Matlab using PsychToolbox (Brainard, 1997; Pelli, 1997).”

“Stimuli were controlled by a computer running Matlab using PsychToolbox (Brainard, 1997; Pelli, 1997).”

“Reinforcement learning provides a powerful mechanism for associating stimuli and actions with the direct experience of reward and punishment (Rescorla and Wagner, 1972; Schultz et al., 1997; Sutton and Barto, 1998).”

“The “warm glow” hypothesis (Andreoni, 1990) suggests that vicarious reward and punishment motivates individuals to prefer either Edited by: Daeyeol Lee, Yale University School of Medicine, USA Reviewed by: Mehrdad Jazayeri, University of Washington, USA James Rilling, Emory University,…”

“It has been documented that social interaction behaviors increase with familiarity in both humans and monkeys (Preston and de Waal, 2002).”

“Such observations endorse the idea that neural mechanisms supporting vicarious reinforcement are derived specializations of the human brain, which support complex social behavior including observational learning, cooperation, and even altruism (Fehr and Fischbacher, 2003).”

“Others have argued that only those species most closely related to humans, namely chimpanzees and bonobos, possess the derived features of human biology and cognition, in particular “theory of mind” (Call and Tomasello, 2008), express other-regarding preferences, but again the evidence for such behavior in apes remain…”

“First, we examined social status, which is known to influence social behaviors in both young children and non-human animals (Hawley, 1999), and observational learning has been implicated in how monkeys acquire social hierarchical information (Cheney and Seyfarth, 1990).”

“For example, fictive, or counterfactual, learning describes sensitivity to reward outcomes for options that were not chosen, were merely observed, or were even imagined (Byrne, 2002; Lohrenz et al., 2007; Epstude and Roese, 2008).”

“We hypothesize that vicarious experiences are processed as rewarding signals in the brain, and are mediated by neurons in homologous circuits governing social perception and reward learning in non-human primates and humans (Bandura and Rosenthal, 1966; Fehr and Camerer, 2007; Lohrenz et al., 2007; Lee, 2008; Hayden et…”

“In some species, learning to perform a task is facilitated by watching others learn the same task (Zentall and Levine, 1972; Zentall et al., 1996; Drea and Wallen, 1999; Subiaul et al., 2004; Whiten et al., 2009).”

“Observing what happens to others also powerfully shapes human learning and behavior (Berger, 1962; Bandura and McDonald, 1963; Bandura et al., 1963).”

“Observing another mouse receive a shock can drive fear conditioning in the observer, and this observational fear conditioning depends on affective pain circuitry that has been implicated in empathy in humans (Jeon et al., 2010).”

“Given that monkeys generally respond more slowly when they anticipate smaller rewards (Kawagoe et al., 1998; Roesch and Olson, 2004), we inferred the subjective reward value of the four conditions to be R SELF > R BOTH and R OTHER > R NONE .”

“Human social emotions associated with vicarious reward and punishment, such as fairness and envy, appear early in ontogeny, and their derangement in mental disorders like psychopathy can have devastating consequences (Kiehl, 2006).”

“Given that monkeys generally respond more slowly when they anticipate smaller rewards (Kawagoe et al., 1998; Roesch and Olson, 2004), we inferred the subjective reward value of the four conditions to be R SELF > R BOTH and R OTHER > R NONE .”

“, 2009), show rudimentary understanding of the intentions of others (Flombaum and Santos, 2005), care for kin (Maestripieri, 1994), and may give up foods to alleviate pain in conspecifics (Masserman et al.”

“Observing what happens to others also powerfully shapes human learning and behavior (Berger, 1962; Bandura and McDonald, 1963; Bandura et al., 1963).”

“Such vicarious reinforcement may be driven by an intrinsic tendency to observe the experience of others to gather information, as can occur in foraging (Cheney and Seyfarth, 1990; Valone and Templeton, 2002).”

“Some have argued that the expression of other-regarding preferences in humans reflects the evolution of mechanisms that promote cooperative reproduction, but the evidence for other-regarding behaviors in cooperatively breeding animals remains controversial (Burkart et al., 2007; de Waal et al., 2008; Lakshminarayanan…”

“Dorsolateral and ventromedial prefrontal cortices in humans have been implicated in observing an action and observing reward outcome of others, respectively (Burke et al., 2010).”

“Observing rewarding events of others has been shown to systematically and effectively modulate neural activity in classic reward areas in humans, including ventral striatum, ventromedial prefrontal cortex, and anterior cingulate cortex (Mobbs et al., 2009; Lombardo et al., 2010).”

“The “warm glow” hypothesis (Andreoni, 1990) suggests that vicarious reward and punishment motivates individuals to prefer either Edited by: Daeyeol Lee, Yale University School of Medicine, USA Reviewed by: Mehrdad Jazayeri, University of Washington, USA James Rilling, Emory University, USA *Correspondence: Steve W. C.…”

“We hypothesize that vicarious experiences are processed as rewarding signals in the brain, and are mediated by neurons in homologous circuits governing social perception and reward learning in non-human primates and humans (Bandura and Rosenthal, 1966; Fehr and Camerer, 2007; Lohrenz et al., 2007; Lee, 2008; Hayden et…”

“For example, fictive, or counterfactual, learning describes sensitivity to reward outcomes for options that were not chosen, were merely observed, or were even imagined (Byrne, 2002; Lohrenz et al., 2007; Epstude and Roese, 2008).”

“, 2007), and may depend on overlapping neural circuitry (Lohrenz et al., 2007; Hayden et al., 2009; Mobbs et al., 2009).”

“We hypothesize that vicarious experiences are processed as rewarding signals in the brain, and are mediated by neurons in homologous circuits governing social perception and reward learning in non-human primates and humans (Bandura and Rosenthal, 1966; Fehr and Camerer, 2007; Lohrenz et al., 2007; Lee, 2008; Hayden et…”

“, 2007), and may depend on overlapping neural circuitry (Lohrenz et al., 2007; Hayden et al., 2009; Mobbs et al., 2009).”

“…1990), display sensitivity to fictive outcomes in non-social settings (Hayden et al., 2009), show rudimentary understanding of the intentions of others (Flombaum and Santos, 2005), care for kin (Maestripieri, 1994), and may give up foods to alleviate pain in conspecifics (Masserman et al., 1964).”

“…attention in the absence of direct reward to self, suggesting vicarious reinforcement may be mediated by social attention circuits in the brain (Klein et al., 2009). saccade reactIon tIMes reveal the Internal delIBeratIve Process The pattern of saccade RTs on choice trials further corroborates…”

“The “warm glow” hypothesis (Andreoni, 1990) suggests that vicarious reward and punishment motivates individuals to prefer either Edited by: Daeyeol Lee, Yale University School of Medicine, USA Reviewed by: Mehrdad Jazayeri, University of Washington, USA James Rilling, Emory University, USA *Correspondence: Steve W. C.…”

“In some species, learning to perform a task is facilitated by watching others learn the same task (Zentall and Levine, 1972; Zentall et al., 1996; Drea and Wallen, 1999; Subiaul et al., 2004; Whiten et al., 2009).”

“In some species, learning to perform a task is facilitated by watching others learn the same task (Zentall and Levine, 1972; Zentall et al., 1996; Drea and Wallen, 1999; Subiaul et al., 2004; Whiten et al., 2009).”

“First, we examined social status, which is known to influence social behaviors in both young children and non-human animals (Hawley, 1999), and observational learning has been implicated in how monkeys acquire social hierarchical information (Cheney and Seyfarth, 1990).”

“In some species, learning to perform a task is facilitated by watching others learn the same task (Zentall and Levine, 1972; Zentall et al., 1996; Drea and Wallen, 1999; Subiaul et al., 2004; Whiten et al., 2009).”

“In some species, learning to perform a task is facilitated by watching others learn the same task (Zentall and Levine, 1972; Zentall et al., 1996; Drea and Wallen, 1999; Subiaul et al., 2004; Whiten et al., 2009).”

“…chimpanzees and bonobos, possess the derived features of human biology and cognition, in particular “theory of mind” (Call and Tomasello, 2008), express other-regarding preferences, but again the evidence for such behavior in apes remains inconclusive (Tomasello et al., 2003; Silk et al., 2005).”

“Frontiers in Neuroscience | Decision Neuroscience March 2011 | Volume 5 | Article 27 | 8 (Anderson and Mason, 1978).”

“This is consistent with observations that rhesus macaques are much more motivated when making voluntary choices compared to making simple operant responses (Suzuki, 1999).”

“These findings demonstrate context-dependent, vicarious reinforcement guides decision-making with respect to others in rhesus macaques.”

“A 1987), and their observational learning seems to be contingent on the associative strength of observed action and outcome (Crawford and Spence, 1921).”

“Such choice behavior depends not only on the stimulus but also on the impact a choice has on another animal observer (Chang et al., 2011).”

“For example, primates are more likely to imitate and learn behaviours demonstrated by a dominant group member 140,141 , and greater perceived similarity with a demonstrator produces stronger threat learning in both mice 142 and humans 104,143 .”

“neither, compared with when self reward is involved (34).”

“In Chang et al.’s (2011) study of vicarious reward in rhesus macaques described above, prosocial responses were more often directed from dominant towards subordinate individuals.”

“For example, primates are more likely to imitate behaviors modeled by a dominant and knowledgeable group member [46, 48].”

“Reaction times, the time from the onset of decisions to movement onset, were computed using a 20 °/s velocity threshold criterion (7, 33, 34).”

“This preference was especially true if the partner was familiar (39).”

“In this social reward-allocation task, monkeys tend to prefer to reward someone over no one, and this prosocial preference is magnified by familiarity and dominance status (96) and significantly modulated by neuropeptide OT (97).”

“…(Apps et al., 2016), but also incorporates rank information, particularly within one’s own social group, dovetailing with behavioral evidence concerning the influence of social dominance and familiarity on cooperative behavior (Chang et al., 2011; Molenberghs, 2013; Seyfarth and Cheney, 2012).”

“This evidence is in line with the idea that social cognition involves the integration of flexible and contextdependent information (Chang et al., 2011; Ibanez and Manes, 2012).”

“A behavioral experiment related to our own, but using a direct preference response, also found that monkeys more often choose a reward to self-only than a reward to both self and another monkey (27).”

“Masaki Isoda, Kansai Medical University, Japan Steve W. C. Chang, Duke University, USA”

“Vicarious reward signals have been shown in the brain of human and non-human primates [44–46].”

“…revealed that the insula, supplementary motor area, dorsolateral prefrontal cortex (DLPFC), and temporal parietal junction are involved (L.J. Chang et al., 2011), while studies of inequity aversion (Sanfey et al., 2003; Hsu Received Sept. 18, 2014; revised Jan. 1, 2015; accepted Jan. 6,…”

“Examples include Chang et al. (2011), who used eye tracking, and de Waal et al. (2008), using token exchange.”

“Subjects were significantly more likely to choose the reward than the withhold option, and this prosocial tendency was enhanced by familiarity [105].”

“This is similar to the dose previously found to affect socially relevant behaviors in monkeys (Chang et al., 2011) and humans (Kirsch et al., 2005; Guastella et al., 2008b; Rimmele et al., 2009; Evans et al., 2010).”

“1 E and F), confirming previous observations during social interaction of nonhuman primates (3, 8, 19).”

“These include social imitation (Ferrari et al., 2006; Subiaul, 2004), prosocial behaviors (Chang et al., 2011; Masserman et al., 1964), as well as understanding others' perceptions (Flombaum and Santos, 2005; Santos et al., 2006).”

“Notably, studies using real-life paradigms have revealed distinct social behaviors as well as specific neural activity that are not observed when the social partners are absent or display non-contingent behaviors (Chang et al., 2011; Azzi et al., 2012; Hosokawa andWatanabe, 2012; Haroush andWilliams, 2015; Noritake et…”

“distribution have produced mixed results [14-29].”

“Another task evaluates the prosocial tendencies of macaques by measuring the amount of reward obtained by conspecifics [221].”

“, 2008), OldWorld monkeys (OWM) (Chang et al., 2011; Massen et al., 2010), and New-World monkeys (NWM) (Burkart et al.”

“Are Social Stimuli Vicarious Reinforcers? The quest for proximal mechanisms of prosociality has led authors to hypothesize that prosocial behavior is shaped by the reinforcing value of certain social events (33).”

“…(e.g., strict dominance hierarchy, low reconciliation tendencies), like chimpanzees (Pan troglodytes) and rhesus macaques (Macaca mulatta), fail to do so, even when such donations pose no cost to the actor (Silk et al., 2005; Hare et al., 2007; Vonk et al., 2008; Chang et al., 2011; but see below).”

“Learning-by-observation (LeO) plays a crucial role in many adaptive behaviours such as foraging and predator avoidance [4] and it has been observed in several animal species including rats [5], dogs [6], pigeons [7] and monkeys [8–11].”

“It emerges from these studies that monkeys are able to monitor other agents’ choices and their outcome and to use this knowledge to guide their own decisions (Chang et al. 2011; Falcone, Brunamonti, Ferraina et al. 2012; Falcone, Brunamonti, and Genovesio 2012; Bevacqua et al. 2013; Falcone et al. 2013).”

“…longtailed macaques (Macaca fuscicularis): Massen et al. 2010, 2011; ravens (Corvus corax): Di Lascio et al. 2013; rhesus macaques (Macaca mulatta): Chang et al. 2011], comparative psychologists have looked primarily to two primate species—chimpanzees and brown capuchin monkeys—to more carefully…”

“In non-human primates, oxytocin increases time spent looking at faces [35,36], improves socially reinforced learning [37], increases the frequency of gaze following behaviours [38] and promotes the donation of rewards to social partners [39,40].”

“Previous and present behavioral results suggest that rhesus monkeys may be sensitive to reward for other monkeys (Azzi et al. 2012; Chang et al. 2011).”

“Reward consumption may help simply by providing vicarious reinforcement, the processing of others’ gains known to influence decision in monkeys [15,65,66].”

“Previous evidence shows that monkeys and other primates will benefit from vicarious learning about the positive experiences of their fellow cagemates and from humans (Chang et al., 2011; Darby and Riopelle, 1959; Falcone et al., 2012a, 2012b; Fogassi et al., 2005; Meunier et al., 2007; Myers, 1970; Snowdon and Boe, 20…”

“, 2014), and show altruistic behaviors (Chang et al., 2011).”

“Several researchers have begun to study social adaptation [20], vicarious rewards [21] and behaviors regarding self vs.”

“as mother-offspring bonds, social play, reciprocal prosocial interactions, strong family alliances that preserve the status in the social hierarchy, as well as social awareness (15-18).”

“Among the foraging behaviours, ‘group foraging’, ‘hold food in mouth’ and ‘retrieve from cheek pouch’ were particularly effective at attracting gaze to the video, consistent with the phenomenon of vicarious reinforcement [34].”

“For example, when 2 monkeys are sitting next to each other’s and the actor’s choice can lead to a different contingency of reward delivery, the choice of the actor monkey is shaped by the future recipient of the reward (Chang et al. 2011; Noritake et al. 2018).”

“Nevertheless, M–M paradigm can be more adapted to study aspects such as the vicarious reinforcement [32] or the influence of social hierarchy on social interactions but new experimental designs should be developed and tested within the paradigm to investigate these topics.”

“To quantify social decision preferences, we constructed a choice preference index as contrast ratios 27,35,38,44,45 (Equation 1).”

“…behavioral studies indicating that non-human primates care about what the other subject obtained (Brosnan and De Waal, 2003; Brosnan et al., 2005; Chang et al., 2011; Azzi et al., 2012), monkeys in the present study appeared to be interested in whether or not the opponent obtained a reward.”

“In monkeys it was furthermore found that cues that are associated with reward delivery to another monkey were preferred over cues that were associated with juice delivery to a chair with no monkey in it and this preference was absent, in the non-social condition, when there was only a juice bottle present (Chang et al…”

“Gaze following can impact future decisions related to the information in the focus of shared attention 18,19 , and is a conserved tendency across primate species found in both adult and infant humans as well as monkeys 17,20 – 22 .”

“In fact, in many other ethically accepted social neuroscience studies, non-human primates has already proven their ability to express coherent socio-cognitive behaviors (Chang et al., 2011; Haroush and Williams, 2015; Yoshida et al., 2011) that were consistent with naturally occurring social behaviors (Ballesta and Du…”

“Indeed, the mere presence of conspecifics can affect primate decision-making (Chang et al., 2011, 2013), while simply watching conspecific social interactions as a third-party observer has been shown to change the response properties of face cells (McMahon et al., 2015).”

“96 Moreover, OT can improve socially reinforced learning, 97 modulate prosocial preferences, 98,99 and affect social hierarchy.”

“Employing prosocial reinforcement learning paradigms similar to the one used here, further studies have shown that monkey ACCg neurons selectively activate for pro-social rewards 77,78 and that ACC lesions cause animals to become unable to learn to benefit another monkey over no one.”

“Monkeys, for example, prefer cues that are associated with reward delivery to another 579 monkey over cues that were associated with juice delivery to a chair with no monkey in it and this preference was 580 absent, in the non-social condition, when there was only a juice bottle present (Chang et al., 2011).”

“Humans for example learn to respond faster to stimuli that become associated with positive social reinforcement [11] and monkeys choose stimuli that predicted a reward delivery to a conspecific more than the stimuli that predicted no reward delivery [12].”