Vicarious Reinforcement Learning Signals When Instructing Others
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Reinforcement learning (RL) theory posits that learning is driven by discrepancies between the predicted and actual outcomes of actions (prediction errors [PEs]). In social environments, learning is often guided by similar RL mechanisms. For example, teachers monitor the actions of students and provide feedback to them. This feedback evokes PEs in students that guide their learning. We report the first study that investigates the neural mechanisms that underpin RL signals in the brain of a teacher. Neurons in the anterior cingulate cortex (ACC) signal PEs when learning from the outcomes of one's own actions but also signal information when outcomes are received by others. Does a teacher's ACC signal PEs when monitoring a student's learning? Using fMRI, we studied brain activity in human subjects (teachers) as they taught a confederate (student) action–outcome associations by providing positive or negative feedback. We examined activity time-locked to the students' responses, when teachers infer student predictions and know actual outcomes. We fitted a RL-based computational model to the behavior of the student to characterize their learning, and examined whether a teacher's ACC signals when a student's predictions are wrong. In line with our hypothesis, activity in the teacher's ACC covaried with the PE values in the model. Additionally, activity in the teacher's insula and ventromedial prefrontal cortex covaried with the predicted value according to the student. Our findings highlight that the ACC signals PEs vicariously for others' erroneous predictions, when monitoring and instructing their learning. These results suggest that RL mechanisms, processed vicariously, may underpin and facilitate teaching behaviors.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Empathy for Pain Involves the Affective but not Sensory Components of Pain
Singer, T., Seymour, B., O'Doherty, J., Kaube, H., Dolan, R. J., & Frith, C. D. (2004) · Science · doi:10.1126/science.1093535 · Cited by 3,929
The expected value of control: an integrative theory of anterior cingulate cortex function
Shenhav, A., Botvinick, M. M., & Cohen, J. D. (2013) · Neuron · doi:10.1016/j.neuron.2013.07.007 · Cited by 2,321
The Neural Basis of Mentalizing
Frith, C. D., & Frith, U. (2006) · Neuron · doi:10.1016/j.neuron.2006.05.001 · Cited by 1,855
Behavioral Theories and the Neurophysiology of Reward
Schultz, W. (2006) · Annual Review of Psychology · doi:10.1146/annurev.psych.56.091103.070229 · Cited by 1,637 · cited 1×
“The behavior of the student was modeled using a simple Rescorla-Wagner (R-W)-based reinforcement learning algorithm (Rescorla and Wagner, 1972), which has been extensively used to examine the behavioral and neural basis of arbitrary visuomotor associations (Dayan and Balleine, 2002; Schultz, 2006; Brovelli et al., 200…”
MethodToward a modern theory of adaptive networks: Expectation and prediction.
Sutton, R. S., & Barto, A. G. (1981) · Psychological Review · doi:10.1037/0033-295x.88.2.135 · Cited by 1,450 · cited 1×
“This method is a standard approach used in reinforcement learning theory (Sutton and Barto, 1981).”
MethodNeuronal Coding of Prediction Errors
Schultz, W., & Dickinson, A. (2000) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.23.1.473 · Cited by 1,427
Insula of the old world monkey. III: Efferent cortical output and comments on function
Mesulam, M. ‐., & Mufson, E. J. (1982) · Journal of Comparative Neurology · doi:10.1002/cne.902120104 · Cited by 1,013 · cited 1×
“In addition, the ACCg has mono- synaptic connections to the portions of the insula and the VmPFC that were found to covary with the student’s prediction in this study (Mesulam and Mufson, 1982; Mufson and Mesulam, 1982; Morecraft et al., 1992; Cavada et al., 2000).”
ResultMedial prefrontal cortex as an action-outcome predictor
Alexander, W. H., & Brown, J. W. (2011) · Nature Neuroscience · doi:10.1038/nn.2921 · Cited by 1,000
Associative learning of social value
Behrens, T. E. J., Hunt, L. T., Woolrich, M. W., & Rushworth, M. F. S. (2008) · Nature · doi:10.1038/nature07538 · Cited by 925
The Anatomical Connections of the Macaque Monkey Orbitofrontal Cortex. A Review
Cavada, C. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.3.220 · Cited by 917
Reward, Motivation, and Reinforcement Learning
Dayan, P., & Balleine, B. W. (2002) · Neuron · doi:10.1016/s0896-6273(02)00963-7 · Cited by 855
Cingulate cortex of the rhesus monkey: II. Cortical afferents
Vogt, B. A., & Pandya, D. N. (1987) · Journal of Comparative Neurology · doi:10.1002/cne.902620208 · Cited by 854 · cited 1×
“…midcingulate cortex) has strong connections to the posterior portions of the superior temporal sulcus, the temporal poles (Markowitsch et al., 1985; Seltzer and Pandya, 1989; Barbas et al., 1999), and the paracingulate cortex (Pandya et al., 1981; Vogt and Pandya, 1987; Petrides and Pandya, 2007).”
BackgroundOptimized EPI for fMRI studies of the orbitofrontal cortex
Deichmann, R., Gottfried, J. A., Hutton, C., & Turner, R. (2003) · NeuroImage · doi:10.1016/s1053-8119(03)00073-9 · Cited by 821 · cited 1×
“A total of 1016 EPI scans were acquired from each participant; 38 slices were acquired in an ascending manner, at an oblique angle ( 30°) to the AC-PC line to decrease the impact of susceptibility artifact in subgenual cortex (Deichmann et al., 2003).”
MethodHuman cingulate cortex: Surface features, flat maps, and cytoarchitecture
Vogt, B. A., Nimchinsky, E. A., Vogt, L. J., & Hof, P. R. (1995) · Journal of Comparative Neurology · doi:10.1002/cne.903590310 · Cited by 745
The neurobiology of rewards and values in social decision making
Ruff, C. C., & Fehr, E. (2014) · Nature reviews. Neuroscience · doi:10.1038/nrn3776 · Cited by 716 · cited 2×
“However, to date, there is no understanding of the neural or computational mechanisms that underpin teaching behaviors (Stanley and Adolphs, 2013; Gariépy et al., 2014; Ruff and Fehr, 2014).”
BackgroundReinforcement Learning Signal Predicts Social Conformity
Klucharev, V., Hytönen, K., Rijpkema, M., Smidts, A., & Fernández, G. (2009) · Neuron · doi:10.1016/j.neuron.2008.11.027 · Cited by 594
Decision theory, reinforcement learning, and the brain
Dayan, P., & Daw, N. D. (2008) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.8.4.429 · Cited by 580
Insula of the old world monkey. II: Afferent cortical input and comments on the claustrum
Mufson, E. J., & Mesulam, M. ‐. (1982) · Journal of Comparative Neurology · doi:10.1002/cne.902120103 · Cited by 559
Cytoarchitecture and neural afferents of orbitofrontal cortex in the brain of the monkey
Morecraft, R. J., Geula, C., & Mesulam, M. ‐. (1992) · Journal of Comparative Neurology · doi:10.1002/cne.903230304 · Cited by 539
Anterior cingulate cortex responds differentially to expectancy violation and social rejection
Somerville, L. H., Heatherton, T. F., & Kelley, W. M. (2006) · Nature Neuroscience · doi:10.1038/nn1728 · Cited by 460 · cited 1×
“, 2004; Yeung and Nieuwenhuis, 2009), including the processing of errors that are elicited by the actions of others (Somerville et al., 2006; Shane et al., 2008; Yoshida et al., 2012).”
BackgroundHayden, B. Y., Heilbronner, S. R., Pearson, J. M., & Platt, M. L. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.4652-10.2011 · Cited by 379
Neuronal reference frames for social decisions in primate frontal cortex
Chang, S. W. C., Gariépy, J.-F., & Platt, M. L. (2013) · Nature Neuroscience · doi:10.1038/nn.3287 · Cited by 336
The role of the striatum in social behavior
Báez-Mendoza, R., & Schultz, W. (2013) · Frontiers in Neuroscience · doi:10.3389/fnins.2013.00233 · Cited by 312 · cited 4×
“Interestingly, this supports recent claims that the ACCg (areas 24a / 24b ) may indeed act as a nexus between these two frames of reference (Hunt and Behrens, 2011; Apps et al., 2013a).”
BackgroundInfluentialA Role for the Macaque Anterior Cingulate Gyrus in Social Valuation
Rudebeck, P. H., Buckley, M. J., Walton, M. E., & Rushworth, M. F. S. (2006) · Science · doi:10.1126/science.1128197 · Cited by 312
The effect of cingulate lesions on social behaviour and emotion
Hadland, K. A., Rushworth, M. F. S., Gaffan, D., & Passingham, R. E. (2003) · Neuropsychologia · doi:10.1016/s0028-3932(02)00325-1 · Cited by 225 · cited 1×
“Lesions to the ACC disrupt the processing of social stimuli (Hadland et al., 2003; Rudebeck et al., 2006), neurons in the ACC are sensitive to rewarding stimuli that others will receive (Chang et al.”
BackgroundBehavioral and Neural Properties of Social Reinforcement Learning
Jones, R. M., Somerville, L. H., Li, J., Ruberry, E. J., Libby, V., Glover, G., Voss, H. U., Ballon, D. J., & Casey, B. J. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.2972-11.2011 · Cited by 191 · cited 1×
“, 2013), and neuroimaging studies have shown that the ACC processes predictions about the value of others’ actions (Behrens et al., 2008; Jones et al., 2011; Zhu et al., 2012; Apps et al., 2013b; Boorman et al., 2013; Apps and Ramnani, 2014).”
BackgroundSocial error monitoring in macaque frontal cortex
Yoshida, K., Saito, N., Iriki, A., & Isoda, M. (2012) · Nature Neuroscience · doi:10.1038/nn.3180 · Cited by 150 · cited 1×
“, 2004; Yeung and Nieuwenhuis, 2009), including the processing of errors that are elicited by the actions of others (Somerville et al., 2006; Shane et al., 2008; Yoshida et al., 2012).”
BackgroundZhu, L., Mathewson, K. E., & Hsu, M. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1116783109 · Cited by 125 · cited 1×
“, 2013), and neuroimaging studies have shown that the ACC processes predictions about the value of others’ actions (Behrens et al., 2008; Jones et al., 2011; Zhu et al., 2012; Apps et al., 2013b; Boorman et al., 2013; Apps and Ramnani, 2014).”
BackgroundThe Anterior Cingulate Gyrus Signals the Net Value of Others' Rewards
Apps, M. A. J., & Ramnani, N. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.2701-13.2014 · Cited by 109
Self-citationCooper, J. C., Dunne, S., Furey, T., & O'Doherty, J. P. (2014) · Cerebral Cortex · doi:10.1093/cercor/bht102 · Cited by 39
Cited by
The Anterior Cingulate Gyrus and Social Cognition: Tracking the Motivation of Others
Apps, M. A. J., Rushworth, M. F. S., & Chang, S. W. C. (2016) · Neuron · doi:10.1016/j.neuron.2016.04.018 · Cited by 559
“Furthermore, it makes specific, testable predictions about a key mechanism that may underpin variability in socio-cognitive abilities in health and disease.”
Self-citationResolving uncertainty in a social world
FeldmanHall, O., & Shenhav, A. (2019) · Nature Human Behaviour · doi:10.1038/s41562-019-0590-x · Cited by 274
Disrupted prediction errors index social deficits in autism spectrum disorder
Balsters, J. H., Apps, M. A. J., Bolis, D., Lehner, R., Gallagher, L., & Wenderoth, N. (2017) · Brain · doi:10.1093/brain/aww287 · Cited by 85
“…receive a reward (Lockwood et al., 2015), the net-value (costbenefit) of another individual’s decision (Apps and Ramnani, 2014), and the discrepancy between another person’s expectations and the actual outcomes, i.e. social prediction errors (Behrens et al., 2008; Apps et al., 2012, 2013a, 2015).”
BackgroundSelf-citationBalsters, J. H., Mantini, D., Apps, M. A. J., Eickhoff, S. B., & Wenderoth, N. (2016) · NeuroImage: Clinical · doi:10.1016/j.nicl.2016.03.016 · Cited by 50
Self-citation
Metadata
- Authors
- Apps, M. A. J., Lesage, E., & Ramnani, N.
- Venue
- The Journal of Neuroscience
- Publisher
- Society for Neuroscience
- Published
- 2015-02-18
- Volume
- 35
- Issue
- 7
- Pages
- 2904-2913
- ISSN
- 0270-6474 (ISSN-L)
- 1529-2401
- OpenAlex
- W2110359145
- Cited by
- 72
- References
- 30
- FWCI
- 5.16
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.