Reinforcement Learning and Counterfactual Reasoning Explain Adaptive Behavior in a Changing Environment
Abstract
Animals routinely adapt to changes in the environment in order to survive. Though reinforcement learning may play a role in such adaptation, it is not clear that it is the only mechanism involved, as it is not well suited to producing rapid, relatively immediate changes in strategies in response to environmental changes. This research proposes that counterfactual reasoning might be an additional mechanism that facilitates change detection. An experiment is conducted in which a task state changes over time and the participants had to detect the changes in order to perform well and gain monetary rewards. A cognitive model is constructed that incorporates reinforcement learning with counterfactual reasoning to help quickly adjust the utility of task strategies in response to changes. The results show that the model can accurately explain human data and that counterfactual reasoning is key to reproducing the various effects observed in this change detection paradigm.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Contrastive reasoning forms the basis of abductive inference.
“In addition, contrastive reasoning forms the basis of abductive inference [19], i.e., the process of inferring certain facts that render some observation plausible [20].”
From A Survey of Contrastive and Counterfactual Explanation Generation Methods for Explainable Artificial Intelligence (2021) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Advances in prospect theory: Cumulative representation of uncertainty
Tversky, A., & Kahneman, D. (1992) · Journal of Risk and Uncertainty · doi:10.1007/bf00122574 · Cited by 15,467 · see citing works
Holroyd, C. B., & Coles, M. G. H. (2002) · Psychological Review · doi:10.1037/0033-295x.109.4.679 · Cited by 4,027 · see citing works
An Integrated Theory of the Mind.
Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C. J., & Qin, Y. (2004) · Psychological Review · doi:10.1037/0033-295x.111.4.1036 · Cited by 3,051 · see citing works
Learning the value of information in an uncertain world
Behrens, T. E. J., Woolrich, M. W., Walton, M. E., & Rushworth, M. F. S. (2007) · Nature Neuroscience · doi:10.1038/nn1954 · Cited by 2,053 · see citing works
Variance and the Value of Information
Stephens, D. W. (1989) · The American Naturalist · doi:10.1086/284969 · Cited by 1,129 · see citing works
Regret and its avoidance: a neuroimaging study of choice behavior
Coricelli, G., Critchley, H., Joffily, M., O’Doherty, J. P., Angela Sirigu, B., & Dolan, R. J. (2005) · Nature Neuroscience · doi:10.1038/nn1514 · Cited by 673 · see citing works
Nassar, M. R., Wilson, R. C., Heasly, B. S., & Gold, J. I. (2010) · Journal of Neuroscience · doi:10.1523/jneurosci.0822-10.2010 · Cited by 564 · see citing works
Posterior cingulate cortex: adapting behavior to a changing world
Pearson, J. M., Heilbronner, S. R., Barack, D. L., Hayden, B. Y., & Platt, M. L. (2011) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2011.02.002 · Cited by 482 · see citing works
Measuring an Individual's Tendency to Take Risks: The Risk Propensity Scale1
Meertens, R. M., & Lion, R. (2008) · Journal of Applied Social Psychology · doi:10.1111/j.1559-1816.2008.00357.x · Cited by 344 · see citing works
Memory and the efficient use of information
McNamara, J. M., & Houston, A. I. (1987) · Journal of Theoretical Biology · doi:10.1016/s0022-5193(87)80209-6 · Cited by 247 · see citing works
Gallistel, C. R., Mark, T. A., King, A. P., & Latham, P. E. (2001) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037//0097-7403.27.4.354 · Cited by 225 · see citing works
On economically tracking a variable environment
Stephens, D. W. (1987) · Theoretical Population Biology · doi:10.1016/0040-5809(87)90036-0 · Cited by 182 · see citing works
Gallistel, C. R., Mark, T. A., King, A. P., et al. (2001) · Journal of Experimental Psychology Animal Behavior Processes · doi:10.1037/0097-7403.27.4.354 · Cited by 174 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Meertens, R. & Lion, R. (2008). Measuring an individuals tendency to take risks: the risk propensity scale. Journal of Applied Social Psychology, 38 (6), 1506-1520.
Cited by
Sorted by citations
Stepin, I., Alonso, J. M., Catala, A., & Pereira-Farina, M. (2021) · IEEE Access · doi:10.1109/access.2021.3051315 · Cited by 376 · see citing works
The functional theory of counterfactual thinking : New evidence, new challenges, new insights
Roese, N. J., & Epstude, K. (2017) · Advances in experimental social psychology · doi:10.1016/bs.aesp.2017.02.001 · Cited by 218 · see citing works
Kentner, A. C., Lima, E., Migliore, M. M., Shin, J., & Scalia, S. (2018) · Neuroscience · doi:10.1016/j.neuroscience.2017.11.035 · Cited by 23 · see citing works
Counterfactual curiosity: motivated thinking about what might have been
FitzGibbon, L., & Murayama, K. (2022) · Philosophical Transactions of the Royal Society B Biological Sciences · doi:10.1098/rstb.2021.0340 · Cited by 22 · see citing works
Counterfactual Evolutionary Reasoning for Virtual Driver Reinforcement Learning in Safe Driving
Ye, P., Qi, H., Zhu, F., & Lv, Y. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3322694 · Cited by 19 · see citing works
Five Breakthroughs: A First Approximation of Brain Evolution From Early Bilaterians to Humans
Bennett, M. S. (2021) · Frontiers in Neuroanatomy · doi:10.3389/fnana.2021.693346 · Cited by 15 · see citing works
Bennett, M. S. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.685853 · Cited by 13 · see citing works
Urquhart, C. J., Lam, L. M. C., Cheuk, B., & Dervin, B. L. (2016) · doi:10.1093/obo/9780199756841-0112 · Cited by 13 · see citing works
Personalized Causal Graph Reasoning for LLMs: An Implementation for Dietary Recommendations
Yang, Z., & Rahmani, A. M. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2025.3624144 · Cited by 5 · see citing works
Counterfactual reasoning as a key for explaining adaptive behavior in a changing environment
Paik, J., Zhang, Y., & Pirolli, P. L. T. (2014) · Biologically Inspired Cognitive Architectures · doi:10.1016/j.bica.2014.11.004 · Cited by 5 · see citing works
Self-citationTaber, P. A., Weir, C. R., Zickmund, S. L., Rutter, E. D., Butler, J. M., & Jones, B. E. (2024) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-024-02805-8 · Cited by 4 · see citing works
Gibbs, R., Landi, P., & Hui, C. (2024) · Ecological Modelling · doi:10.1016/j.ecolmodel.2024.110693 · Cited by 3 · see citing works
Transfer learning with causal counterfactual reasoning in Decision TransformersPreprint
Boustati, A., Chockler, H., & McNamee, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.14355 · Cited by 2 · see citing works
When Remembering and Planning Are Worth It: On Navigation Strategies Under Change
Madani, O., Burns, J. B., Eghbali, R., & Dean, T. (2026) · Studies in computational intelligence · doi:10.1007/978-3-032-13592-6_6 · Cited by 1 · see citing works
Boutyour, Y., & Idrissi, A. (2024) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2024.3485482 · Cited by 1 · see citing works
Robot Delivery of Actionable Counterfactual Neural Network Explanations: Results in Group Perception
Narcomey, A., Claure, H., & Vázquez, M. (2025) · IEEE RO-MAN · doi:10.1109/ro-man63969.2025.11217869
Aria, H. P., & Xu, Z. (2025) · IEEE Control Systems Letters · doi:10.1109/lcsys.2025.3645286
Emmanuel, D. E., & Fernando, A. (2026) · Research Square · doi:10.21203/rs.3.rs-9542617/v1
- Authors
- Zhang, Y., Paik, J., & Pirolli, P.
- Affiliations
- University of Oregon (US)
- Palo Alto Research Center (US)
- Corresponding author
- Zhang, Yunfeng
- Paik, Jaehyon
- Venue
- Topics in Cognitive Science
- Publisher
- Wiley
- Published
- 2015-04
- Volume
- 7
- Issue
- 2
- Pages
- 368-381
- ISSN
- 1756-8757 (ISSN-L)
- 1756-8765
- OpenAlex
- W2113992026
- Cited by
- 18
- References
- 12
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 3 · 2023: 1 · 2022: 1 · 2021: 4 · 2017: 2 · 2016: 1 · 2014: 1
- FWCI
- 0.76
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Topics in Cognitive Science (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neural and Behavioral Psychology Studies
- Physical Sciences › Computer Science › Artificial Intelligence › Reinforcement Learning in Robotics
- Keywords
- reinforcement learning, counterfactual reasoning, change detection, adaptive behavior, environmental change, cognitive modeling, strategy adjustment, human decision-making
- MeSH terms
- Adaptation, Psychological, Adult, Environment, Female, Humans, Male, Middle Aged, Models, Theoretical, Reinforcement, Psychology, Reward, Thinking, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
