Pain: A Precision Signal for Reinforcement Learning and Control
Open accessThe DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Abstract
Since noxious stimulation usually leads to the perception of pain, pain has traditionally been considered sensory nociception. But its variability and sensitivity to a broad array of cognitive and motivational factors have meant it is commonly viewed as inherently imprecise and intangibly subjective. However, the core function of pain is motivational-to direct both short- and long-term behavior away from harm. Here, we illustrate that a reinforcement learning model of pain offers a mechanistic understanding of how the brain supports this, illustrating the underlying computational architecture of the pain system. Importantly, it explains why pain is tuned by multiple factors and necessarily supported by a distributed network of brain regions, recasting pain as a precise and objectifiable control signal.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Bellman, R. E. (2010) · doi:10.1515/9781400835386 · Cited by 14,588
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · cited 1×
“…Framework propose that sensation, motivation, and action are intrinsically related by their drive to understand the causes of unexpected stimuli (Friston, 2010), and the notion of ‘‘active inference’’ describes how actions can be conceived to ultimately reflect minimization of future…”
BackgroundHow do you feel — now? The anterior insula and human awareness
(Bud) Craig, A. D. (2009) · Nature Reviews Neuroscience · doi:10.1038/nrn2555 · Cited by 6,657 · cited 1×
“…is primarily descending, there should still be a restricted cortical response that primarily reflects pain aversive intensity after modulation (for example, in posterior insula or mid-anterior cingulate cortex [Segerdahl et al., 2015; Kragel et al., 2018; Craig and Craig, 2009]).”
Cognitive maps in rats and men
Tolman, E. C. (1948) · Psychological Review · doi:10.1037/h0061626 · Cited by 6,641
How do you feel? Interoception: The sense of the physiological condition of the body
Craig, A. D. (2002) · Nature Reviews Neuroscience · doi:10.1038/nrn894 · Cited by 6,006 · cited 1×
“…Craig subsequently proposed the homeostatic model, in which sensory and motivational dimensions are inherently integrated as a single system, involving pain-specific lateral thalamocortical projections to Neuron 101, March 20, 2019 ª 2019 Elsevier Inc. 1029 the insula cortex (Craig, 2002, 2003).”
BackgroundFear-avoidance and its consequences in chronic musculoskeletal pain: a state of the art.
Vlaeyen, J. W. S., & Linton, S. J. (2000) · Pain · doi:10.1016/s0304-3959(99)00242-0 · Cited by 4,931
Endogenous Pain Control Systems: Brainstem Spinal Pathways and Endorphin Circuitry
Basbaum, A. I., & Fields, H. L. (1984) · Annual Review of Neuroscience · doi:10.1146/annurev.ne.07.030184.001521 · Cited by 2,687
The Bayesian brain: The role of uncertainty in neural coding and computation
Knill, D. C., & Pouget, A. (2004) · Trends in Neurosciences · doi:10.1016/j.tins.2004.10.007 · Cited by 2,588
Daw, N. D., Niv, Y., & Dayan, P. (2005) · Nature Neuroscience · doi:10.1038/nn1560 · Cited by 2,533
Pain and emotion interactions in subregions of the cingulate gyrus
Vogt, B. A. (2005) · Nature Reviews Neuroscience · doi:10.1038/nrn1704 · Cited by 1,810 · cited 1×
“…is highly opioid rich and sits within an anatomical network with connections to PAG and other subcortical regions involved in pain and learning, including amygdala, VMPFC, hippocampus, Neuron 101, March 20, 2019 1035 lateral orbitofrontal cortex (OFC), and PFC (Margulies et al., 2007; Vogt, 2005).”
BackgroundUncertainty, Neuromodulation, and Attention
Yu, A. J., & Dayan, P. (2005) · Neuron · doi:10.1016/j.neuron.2005.04.026 · Cited by 1,774
An fMRI-Based Neurologic Signature of Physical Pain
Wager, T. D., Atlas, L. Y., Lindquist, M. A., Roy, M., Woo, C.-W., & Kross, E. (2013) · New England Journal of Medicine · doi:10.1056/nejmoa1204471 · Cited by 1,597
Pavlovian conditioning: it's not what you think it is
Rescorla, R. A. (1988) · American Psychologist · doi:10.1037/0003-066x.43.3.151 · Cited by 1,542 · cited 1×
“That is, cognitive processes are likely to be routinely involved in simple Pavlovian and instrumental tasks, allowing a more sophisticated representation of task structure than is possible than using simpler ‘‘model-free’’ learning algorithms (Rescorla, 1988).”
When More Pain Is Preferred to Less: Adding a Better End
Kahneman, D., Fredrickson, B. L., Schreiber, C. A., & Redelmeier, D. A. (1993) · Psychological Science · doi:10.1111/j.1467-9280.1993.tb00589.x · Cited by 1,502
Toward a modern theory of adaptive networks: Expectation and prediction.
Sutton, R. S., & Barto, A. G. (1981) · Psychological Review · doi:10.1037/0033-295x.88.2.135 · Cited by 1,450 · cited 1×
“This information is passed to an ‘‘agent’’ that decides what responses or actions to emit based on the current stored state and action values and then updates these values based on the next state (Sutton and Barto, 1981).”
BackgroundThe cortical representation of pain
Treede, R.-D., Kenshalo, D. R., Gracely, R. H., & Jones, A. K. P. (1999) · Pain · doi:10.1016/s0304-3959(98)00184-5 · Cited by 1,077
Corticostriatal functional connectivity predicts transition to chronic back pain
Baliki, M. N., Petre, B., Torbey, S., Herrmann, K. M., Huang, L., Schnitzer, T. J., Fields, H. L., & Apkarian, A. V. (2012) · Nature Neuroscience · doi:10.1038/nn.3153 · Cited by 1,050
Neurocognitive endophenotypes of impulsivity and compulsivity: Towards dimensional psychiatry
Robbins, T. W., Gillan, C. M., Smith, D. G., de Wit, S., & Ersche, K. D. (2012) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2011.11.009 · Cited by 1,003
Neural organization of the defensive behavior system responsible for fear
Fanselow, M. S. (1994) · Psychonomic Bulletin & Review · doi:10.3758/bf03210947 · Cited by 1,000
Statistically optimal perception and learning: from behavior to neural representations
Fiser, J., Berkes, P., Orbán, G., & Lengyel, M. (2010) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2010.01.003 · Cited by 672
Observational fear learning involves affective pain system and Cav1.2 Ca2+ channels in ACC
Jeon, D., Kim, S., Chetana, M., Jo, D., Ruley, H. E., Lin, S.-Y., Rabah, D., Kinet, J.-P., & Shin, H.-S. (2010) · Nature Neuroscience · doi:10.1038/nn.2504 · Cited by 651
Active Inference, homeostatic regulation and adaptive behavioural control
Pezzulo, G., Rigoli, F., & Friston, K. (2015) · Progress in Neurobiology · doi:10.1016/j.pneurobio.2015.09.001 · Cited by 650 · cited 2×
“But understanding how the brain actually achieves this is much more complex (Pezzulo et al., 2015), and none of the existing models fully capture how the pain system successfully balances stimulus identification, information seeking, harm minimisation, and perhaps most critically, speed.”
Olsson, A., & Phelps, E. A. (2007) · Nature Neuroscience · doi:10.1038/nn1968 · Cited by 622
Temporal difference models describe higher-order learning in humans
Seymour, B., O'Doherty, J. P., Dayan, P., Koltzenburg, M., Jones, A. K., Dolan, R. J., Friston, K. J., & Frackowiak, R. S. (2004) · Nature · doi:10.1038/nature02581 · Cited by 605
Self-citationKucyi, A., & Davis, K. D. (2015) · Trends in Neurosciences · doi:10.1016/j.tins.2014.11.006 · Cited by 587
Decision theory, reinforcement learning, and the brain
Dayan, P., & Daw, N. D. (2008) · Cognitive, Affective, & Behavioral Neuroscience · doi:10.3758/cabn.8.4.429 · Cited by 580
Humans use directed and random exploration to solve the explore–exploit dilemma.
Wilson, R. C., Geana, A., White, J. M., Ludvig, E. A., & Cohen, J. D. (2014) · Journal of Experimental Psychology: General · doi:10.1037/a0038199 · Cited by 580
Getting the pain you expect: mechanisms of placebo, nocebo and reappraisal effects in humans
Tracey, I. (2010) · Nature Medicine · doi:10.1038/nm.2229 · Cited by 553
Placebo Analgesia: A Predictive Coding Perspective
Büchel, C., Geuter, S., Sprenger, C., & Eippert, F. (2014) · Neuron · doi:10.1016/j.neuron.2014.02.042 · Cited by 505
Encoding of fear learning and memory in distributed neuronal circuits
Herry, C., & Johansen, J. P. (2014) · Nature Neuroscience · doi:10.1038/nn.3869 · Cited by 474
How prior experience shapes placebo analgesia
Colloca, L., & Benedetti, F. (2006) · Pain · doi:10.1016/j.pain.2006.04.005 · Cited by 443
Opponent appetitive-aversive neural processes underlie predictive learning of pain relief
Seymour, B., O'Doherty, J. P., Koltzenburg, M., Wiech, K., Frackowiak, R., Friston, K., & Dolan, R. (2005) · Nature Neuroscience · doi:10.1038/nn1527 · Cited by 439
Self-citationWiech, K., Kalisch, R., Weiskopf, N., Pleger, B., Stephan, K. E., & Dolan, R. J. (2006) · The Journal of Neuroscience · doi:10.1523/jneurosci.2568-06.2006 · Cited by 323
Navratilova, E., Xie, J. Y., Okun, A., Qu, C., Eyde, N., Ci, S., Ossipov, M. H., King, T., Fields, H. L., & Porreca, F. (2012) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1214605109 · Cited by 313
Habituation to painful stimulation involves the antinociceptive system
Bingel, U., Schoell, E., Herken, W., Büchel, C., & May, A. (2007) · Pain · doi:10.1016/j.pain.2006.12.005 · Cited by 298
Striatal activity underlies novelty-based choice in humans
Wittmann, B. C., Daw, N. D., Seymour, B., & Dolan, R. J. (2008) · Neuron · doi:10.1016/j.neuron.2008.04.027 · Cited by 277
Self-citationKragel, P. A., Kano, M., Van Oudenhove, L., Ly, H. G., Dupont, P., Rubio, A., Delon-Martin, C., Bonaz, B. L., Manuck, S. B., Gianaros, P. J., et al. (2018) · Nature Neuroscience · doi:10.1038/s41593-017-0051-7 · Cited by 272 · cited 1×
“That this implies that subjective pain will be best estimated from responses in multiple regions is consistent with general network and connectivity (Kucyi and Davis, 2015) and multivariate ‘‘signatures’’ of pain (Wager et al., 2013; Woo et al., 2015; The RL model involves a large set of parameters…”
ResultDistinct Brain Systems Mediate the Effects of Nociceptive Input and Self-Regulation on Pain
Woo, C.-W., Roy, M., Buhle, J. T., & Wager, T. D. (2015) · PLoS Biology · doi:10.1371/journal.pbio.1002036 · Cited by 263
Serotonin Selectively Modulates Reward Value in Human Decision-Making
Seymour, B., Daw, N. D., Roiser, J. P., Dayan, P., & Dolan, R. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.0053-12.2012 · Cited by 252
Self-citationRepresentation of aversive prediction errors in the human periaqueductal gray
Roy, M., Shohamy, D., Daw, N., Jepma, M., Wimmer, G. E., & Wager, T. D. (2014) · Nature Neuroscience · doi:10.1038/nn.3832 · Cited by 246
Prospective representation of navigational goals in the human hippocampus
Brown, T. I., Carr, V. A., LaRocque, K. F., Favila, S. E., Gordon, A. M., Bowles, B., Bailenson, J. N., & Wagner, A. D. (2016) · Science · doi:10.1126/science.aaf0784 · Cited by 233
Brown, C. A., Seymour, B., Boyle, Y., El-Deredy, W., & Jones, A. K. P. (2008) · Pain · doi:10.1016/j.pain.2007.05.022 · Cited by 218
Self-citationHomeostatic reinforcement learning for integrating reward collection and physiological stability
Keramati, M., & Gutkin, B. (2014) · eLife · doi:10.7554/elife.04811 · Cited by 218
Salomons, T. V., Johnstone, T., Backonja, M.-M., Shackman, A. J., & Davidson, R. J. (2007) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2007.19.6.993 · Cited by 195
The algorithmic anatomy of model-based evaluation
Daw, N. D., & Dayan, P. (2014) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2013.0478 · Cited by 195
Functional dissociation of stimulus intensity encoding and predictive coding of pain in the insula
Geuter, S., Boll, S., Eippert, F., & Büchel, C. (2017) · eLife · doi:10.7554/elife.24770 · Cited by 184
How expectations influence pain
Fields, H. L. (2018) · Pain · doi:10.1097/j.pain.0000000000001272 · Cited by 175 · cited 4×
“The idea that, as a motivational system, pain was directly modulated as a decision by the system itself was most clearly articulated in Fields’ motivation-decision model (Fields, 2006, 2018).”
BackgroundInfluentialYoshida, W., Seymour, B., Koltzenburg, M., & Dolan, R. J. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.4984-12.2013 · Cited by 165
Self-citationBayes in the Brain—On Bayesian Modelling in Neuroscience
Colombo, M., & Seriès, P. (2012) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/axr043 · Cited by 164
Behavioural and neural evidence for self-reinforcing expectancy effects on pain
Jepma, M., Koban, L., van Doorn, J., Jones, M., & Wager, T. D. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0455-8 · Cited by 155
Discovering latent causes in reinforcement learning
Gershman, S. J., Norman, K. A., & Niv, Y. (2015) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2015.07.007 · Cited by 151
Dorsal tegmental dopamine neurons gate associative learning of fear
Groessl, F., Munsch, T., Meis, S., Griessner, J., Kaczanowska, J., Pliota, P., Kargl, D., Badurek, S., Kraitsy, K., Rassoulpour, A., et al. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0174-5 · Cited by 118
Different brain circuitries mediating controllable and uncontrollable pain
Bräscher, A.-K., Becker, S., Hoeppli, M.-E., & Schweinhardt, P. (2016) · The Journal of Neuroscience · doi:10.1523/jneurosci.1954-15.2016 · Cited by 118
Tabor, A., Thacker, M. A., Moseley, G. L., & Körding, K. P. (2017) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1005142 · Cited by 111
A Bayesian Perspective on Sensory and Cognitive Integration in Pain Perception and Placebo Analgesia
Anchisi, D., & Zanon, M. (2015) · PLOS ONE · doi:10.1371/journal.pone.0117270 · Cited by 84 · cited 2×
“…some sort of Bayesian inference (Colombo and Seriès, 2012; Knill and Pouget, 2004) and can in principle explain why pain perception is routinely biased toward prior knowledge (Colloca and Benedetti, 2006; Anchisi and Zanon, 2015; Seymour and Dolan, 2013; Wiech, 2016; Atlas et al., 2010).”
BackgroundStriatal structure and function predict individual biases in learning to avoid pain
Eldar, E., Hauser, T. U., Dayan, P., & Dolan, R. J. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1519829113 · Cited by 79
Boundary effects of expectation in human pain perception
Hird, E. J., Charalambous, C., El-Deredy, W., Jones, A. K. P., & Talmi, D. (2019) · Scientific Reports · doi:10.1038/s41598-019-45811-x · Cited by 68
Bayesian Learning Models of Pain: A Call to Action
Tabor, A., & Burr, C. (2019) · Current Opinion in Behavioral Sciences · doi:10.1016/j.cobeha.2018.10.006 · Cited by 57
Neural Emotion Regulation Circuitry Underlying Anxiolytic Effects of Perceived Control over Pain
Salomons, T. V., Nusslock, R., Detloff, A., Johnstone, T., & Davidson, R. J. (2015) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_00702 · Cited by 54
Cited by
Ashar, Y. K., Gordon, A., Schubiner, H., Uipi, C., Knight, K., Anderson, Z., Carlisle, J., Polisky, L., Geuter, S., Flood, T. F., et al. (2022) · JAMA Psychiatry · doi:10.1001/jamapsychiatry.2021.2669 · Cited by 331
The complex syndrome of functional neurological disorder
Forejtová, Z., Serranová, T., Sieger, T., Slovák, M., Nováková, L., Věchetová, G., Růžička, E., & Edwards, M. J. (2023) · Psychological Medicine · doi:10.1017/s0033291721005225 · Cited by 44
Cognitive fatigue weakens the effects of distraction on pain
Vogel, T. A., Desjardins, S., Chabot, B., Otto, A. R., Falk, C. F., & Roy, M. (2023) · Pain · doi:10.1097/j.pain.0000000000002985 · Cited by 15
Mindfulness-Based Interventions for Psychiatry
Ivkovic, A., Evans, S., Mirsky, P., Ashar, Y. K., & Muskin, P. R. (2024) · Tasman’s Psychiatry · doi:10.1007/978-3-030-51366-5_159 · Cited by 1
Metadata
- Authors
- Seymour, B.
- Venue
- Neuron
- Publisher
- Elsevier BV
- Published
- 2019-03
- Volume
- 101
- Issue
- 6
- Pages
- 1029-1041
- ISSN
- 0896-6273 (ISSN-L)
- OpenAlex
- W2923628873
- Cited by
- 167
- References
- 59
- FWCI
- 9.57
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.