The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Frontal Theta Reflects Uncertainty and Unexpectedness during Exploration and Exploitation
Open accessAbstract
In order to understand the exploitation/exploration trade-off in reinforcement learning, previous theoretical and empirical accounts have suggested that increased uncertainty may precede the decision to explore an alternative option. To date, the neural mechanisms that support the strategic application of uncertainty-driven exploration remain underspecified. In this study, electroencephalography (EEG) was used to assess trial-to-trial dynamics relevant to exploration and exploitation. Theta-band activities over middle and lateral frontal areas have previously been implicated in EEG studies of reinforcement learning and strategic control. It was hypothesized that these areas may interact during top-down strategic behavioral control involved in exploratory choices. Here, we used a dynamic reward-learning task and an associated mathematical model that predicted individual response times. This reinforcement-learning model generated value-based prediction errors and trial-by-trial estimates of exploration as a function of uncertainty. Mid-frontal theta power correlated with unsigned prediction error, although negative prediction errors had greater power overall. Trial-to-trial variations in response-locked frontal theta were linearly related to relative uncertainty and were larger in individuals who used uncertainty to guide exploration. This finding suggests that theta-band activities reflect prefrontal-directed strategic control during exploratory choices.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Delorme, A., & Makeig, S. (2004) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2003.10.009 · Cited by 24,371
Holroyd, C. B., & Coles, M. G. H. (2002) · Psychological Review · doi:10.1037/0033-295x.109.4.679 · Cited by 4,027
Getting formal with dopamine and reward
Schultz, W. (2002) · Neuron · doi:10.1016/s0896-6273(02)00967-4 · Cited by 2,574
Cortical substrates for exploratory decisions in humans
Daw, N. D., O'Doherty, J. P., Dayan, P., Seymour, B., & Dolan, R. J. (2006) · Nature · doi:10.1038/nature04766 · Cited by 2,224
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053
Miltner, W. H. R., Braun, C. H., & Coles, M. G. H. (1997) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.1997.9.6.788 · Cited by 1,530
An information theoretical approach to prefrontal executive function
Koechlin, E., & Summerfield, C. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.04.005 · Cited by 1,183
Cohen, J. D., McClure, S. M., & Yu, A. J. (2007) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2007.2098 · Cited by 1,121
Dorsal anterior cingulate cortex: A role in reward-based decision making
Bush, G., Vogt, B. A., Holmes, J., Dale, A. M., Greve, D., Jenike, M. A., & Rosen, B. R. (2002) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.012470999 · Cited by 1,096
Debener, S., Ullsperger, M., Siegel, M., Fiehler, K., von Cramon, D. Y., & Engel, A. K. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.3286-05.2005 · Cited by 1,062
Cognitive control, hierarchy, and the rostro–caudal organization of the frontal lobes
Badre, D. (2008) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2008.02.004 · Cited by 980
Optimal decision making and the anterior cingulate cortex
Kennerley, S. W., Walton, M. E., Behrens, T. E. J., Buckley, M. J., & Rushworth, M. F. S. (2006) · Nature Neuroscience · doi:10.1038/nn1724 · Cited by 948
Children's Brain Development Benefits from Longer Gestation
Davis, E. P., Buss, C., Muftuler, L. T., Head, K., Hasso, A., Wing, D. A., Hobel, C., & Sandman, C. A. (2011) · Frontiers in Psychology · doi:10.3389/fpsyg.2011.00001 · Cited by 858
The feedback-related negativity reflects the binary evaluation of good versus bad outcomes
Hajcak, G., Moser, J. S., Holroyd, C. B., & Simons, R. F. (2006) · Biological Psychology · doi:10.1016/j.biopsycho.2005.04.001 · Cited by 699
Cavanagh, J. F., Cohen, M. X., & Allen, J. J. B. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.4137-08.2009 · Cited by 669
Self-citationTheta lingua franca: A common mid‐frontal substrate for action monitoring processes
Cavanagh, J. F., Zambrano‐Vazquez, L., & Allen, J. J. B. (2012) · Psychophysiology · doi:10.1111/j.1469-8986.2011.01293.x · Cited by 651
Self-citationRole for Cingulate Motor Area Cells in Voluntary Movement Selection Based on Reward
Shima, K., & Tanji, J. (1998) · Science · doi:10.1126/science.282.5392.1335 · Cited by 650
Holroyd, C. B., Pakzad‐Vaezi, K. L., & Krigolson, O. E. (2008) · Psychophysiology · doi:10.1111/j.1469-8986.2008.00668.x · Cited by 582
Luu, P., Tucker, D. M., & Makeig, S. (2004) · Clinical Neurophysiology · doi:10.1016/j.clinph.2004.03.031 · Cited by 545
The Electrophysiological Dynamics of Interference during the Stroop Task
Hanslmayr, S., Pastötter, B., Bäuml, K.-H., Gruber, S., Wimber, M., & Klimesch, W. (2008) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2008.20020 · Cited by 510
Electrophysiological Responses to Errors and Feedback in the Process of Action Regulation
Luu, P., Tucker, D. M., Derryberry, D., Reed, M., & Poulsen, C. (2003) · Psychological Science · doi:10.1111/1467-9280.01417 · Cited by 502
Scheffers, M. K., & Coles, M. G. H. (2000) · Journal of Experimental Psychology: Human Perception and Performance · doi:10.1037/0096-1523.26.1.141 · Cited by 493
Frank, M. J., Doll, B. B., Oas-Terpstra, J., & Moreno, F. (2009) · Nature Neuroscience · doi:10.1038/nn.2342 · Cited by 471
Self-citationFrontal theta links prediction errors to behavioral adaptation in reinforcement learning
Cavanagh, J. F., Frank, M. J., Klein, T. J., & Allen, J. J. B. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.11.080 · Cited by 469
Self-citationJepma, M., & Nieuwenhuis, S. (2011) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2010.21548 · Cited by 439
Oliveira, F. T. P., McDonald, J. J., & Goodman, D. (2007) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2007.19.12.1994 · Cited by 387
Hayden, B. Y., Heilbronner, S. R., Pearson, J. M., & Platt, M. L. (2011) · The Journal of Neuroscience · doi:10.1523/jneurosci.4652-10.2011 · Cited by 379 · cited 3×
“These midfrontal EEG findings in the theta-band TF-ROI support prior interpretations of a binary sign difference and a continuous magnitude effect in dorsal cingulate coding of reinforcement feedback (Hayden et al. 2011).”
BackgroundSingle-Trial Regression Elucidates the Role of Prefrontal Theta Oscillations in Response Conflict
Cohen, M. X., & Cavanagh, J. F. (2011) · Frontiers in Psychology · doi:10.3389/fpsyg.2011.00030 · Cited by 343
Self-citationTheta EEG dynamics of the error-related negativity
Trujillo, L. T., & Allen, J. J. B. (2007) · Clinical Neurophysiology · doi:10.1016/j.clinph.2006.11.009 · Cited by 314 · cited 1×
“…these aforementioned responselocked ERPs have also been shown to reflect phase-locked mid-frontal theta-band activities (Luu and Tucker 2001; Luu et al. 2004; Trujillo and Allen 2007), we are unaware of any investigations of the EEG correlates of response uncertainty during reinforcement learning.”
Wang, C., Ulbert, I., Schomer, D. L., Marinkovic, K., & Halgren, E. (2005) · The Journal of Neuroscience · doi:10.1523/jneurosci.4151-04.2005 · Cited by 295
Risk, unexpected uncertainty, and estimation uncertainty: Bayesian learning in unstable settings
Payzan-LeNestour, E., & Bossaerts, P. (2011) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1001048 · Cited by 267
Regulating action: alternating activation of midline frontal and motor cortical networks
Luu, P., & Tucker, D. M. (2001) · Clinical Neurophysiology · doi:10.1016/s1388-2457(01)00559-4 · Cited by 255
Testing the Reward Prediction Error Hypothesis with an Axiomatic Model
Rutledge, R. B., Dean, M., Caplin, A., & Glimcher, P. W. (2010) · The Journal of Neuroscience · doi:10.1523/jneurosci.1747-10.2010 · Cited by 244
Strauss, G. P., Frank, M. J., Waltz, J. A., Kasanova, Z., Herbener, E. S., & Gold, J. M. (2011) · Biological Psychiatry · doi:10.1016/j.biopsych.2010.10.015 · Cited by 233
Self-citationChase, H. W., Swainson, R., Durham, L., Benham, L., & Cools, R. (2011) · Journal of Cognitive Neuroscience · doi:10.1162/jocn.2010.21456 · Cited by 223
The effects of uncertainty in error monitoring on associated ERPs
Pailing, P. E., & Segalowitz, S. J. (2004) · Brain and Cognition · doi:10.1016/j.bandc.2004.06.005 · Cited by 205
Baker, T. E., & Holroyd, C. B. (2011) · Biological Psychology · doi:10.1016/j.biopsycho.2011.01.010 · Cited by 203
Resolution of Uncertainty in Prefrontal Cortex
Yoshida, W., & Ishii, S. (2006) · Neuron · doi:10.1016/j.neuron.2006.05.006 · Cited by 197
Cohen, M. X., Ridderinkhof, K. R., Haupt, S., Elger, C. E., & Fell, J. (2008) · Brain Research · doi:10.1016/j.brainres.2008.07.114 · Cited by 187
Self-citationPearson, J. M., Hayden, B. Y., Raghavachari, S., & Platt, M. L. (2009) · Current Biology · doi:10.1016/j.cub.2009.07.048 · Cited by 184
Pfabigan, D. M., Alexopoulos, J., Bauer, H., & Sailer, U. (2011) · Psychophysiology · doi:10.1111/j.1469-8986.2010.01136.x · Cited by 177
A Role for Dopamine in Temporal Decision Making and Reward Maximization in Parkinsonism
Moustafa, A. A., Cohen, M. X., Sherman, S. J., & Frank, M. J. (2008) · Journal of Neuroscience · doi:10.1523/jneurosci.3116-08.2008 · Cited by 169
Self-citationTemporal dynamics of prediction error processing during reward-based decision making
Philiastides, M. G., Biele, G., Vavatzanidis, N., Kazzer, P., & Heekeren, H. R. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2010.05.052 · Cited by 132
Exploration Bonuses and Dual Control
Dayan, P., & B, T. J. S. A. (1996) · Machine Learning · doi:10.1023/a:1018357105171 · Cited by 120
Noonan, C. J. (2010) · Routledge eBooks · doi:10.4324/9780080938646-22 · Cited by 117
Axiomatic methods, dopamine and reward prediction error
Caplin, A., & Dean, M. (2008) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2008.07.007 · Cited by 88
Potts, G. F., Martin, L. E., Kamp, S., & Donchin, E. (2010) · Psychophysiology · doi:10.1111/j.1469-8986.2010.01049.x · Cited by 74
Ichikawa, N., Siegle, G. J., Dombrovski, A. Y., & Ohira, H. (2010) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2010.09.001 · Cited by 36
A single case-study of diagonistic dyspraxia
Barbeau, E. J., Joubert, S., & Poncet, M. (2004) · Brain and Cognition · doi:10.1016/j.bandc.2004.02.006 · Cited by 22
Cited by
Frontal theta as a mechanism for cognitive control
Cavanagh, J. F., & Frank, M. J. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.04.012 · Cited by 2,726
Self-citationThe expected value of control: an integrative theory of anterior cingulate cortex function
Shenhav, A., Botvinick, M. M., & Cohen, J. D. (2013) · Neuron · doi:10.1016/j.neuron.2013.07.007 · Cited by 2,321
Frontal midline theta reflects anxiety and cognitive control: Meta-analytic evidence
Cavanagh, J. F., & Shackman, A. J. (2015) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2014.04.003 · Cited by 623
Self-citationFunctionally dissociable influences on learning rate in a dynamic environment
McGuire, J. T., Nassar, M. R., Gold, J. I., & Kable, J. W. (2014) · Neuron · doi:10.1016/j.neuron.2014.10.013 · Cited by 299
Frontosubthalamic Circuits for Control of Action and Cognition
Aron, A. R., Herz, D. M., Brown, P., Forstmann, B. U., & Zaghloul, K. (2016) · The Journal of Neuroscience · doi:10.1523/jneurosci.2348-16.2016 · Cited by 291
Metadata
- Authors
- Cavanagh, J. F., Figueroa, C. M., Cohen, M. X., & Frank, M. J.
- Venue
- Cerebral Cortex
- Publisher
- Oxford University Press (OUP)
- Published
- 2011-11-25
- Volume
- 22
- Issue
- 11
- Pages
- 2575-2586
- ISSN
- 1047-3211 (ISSN-L)
- 1460-2199
- OpenAlex
- W2170961478
- Cited by
- 291
- References
- 49
- FWCI
- 5.13
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.