The Importance of Falsification in Computational Cognitive Modeling
Published version of a preprint
Abstract
In the past decade the field of cognitive sciences has seen an exponential growth in the number of computational modeling studies. Previous work has indicated why and how candidate models of cognition should be compared by trading off their ability to predict the observed data as a function of their complexity. However, the importance of falsifying candidate models in light of the observed data has been largely underestimated, leading to important drawbacks and unjustified conclusions. We argue here that the simulation of candidate models is necessary to falsify models and therefore support the specific claims about cognitive function made by the vast majority of model-based studies. We propose practical guidelines for future research that combine model comparison and falsification.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The study used overlapping categories, withholding feedback on testing trials, and varying task, stimulus type, and stimulus reliability as a strong test of human confidence reports.
“We carried out a strong test of human confidence reports, using overlapping categories [52], withholding feedback on testing trials, and varying experimental components such as task, stimulus type, and stimulus reliability [32].”
From Comparing Bayesian and non-Bayesian accounts of human confidence reports (2018) · cites this paper as [52]
The cited paper’s discussion of task design (Current common practice and associated problems, p. 3) concerns tests of computational models, while its illustrated task uses simulated reinforcement-learning data (Box 3, p. 16). Searches for confidence reports, overlapping categories, withheld feedback, and stimulus reliability found no passage describing the claimed test of human confidence reports. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The free-energy principle: a unified brain theory?
Friston, K. (2010) · Nature Reviews Neuroscience · doi:10.1038/nrn2787 · Cited by 8,120 · cited 1× · see citing works
“At the ‘computational’ level, model comparison informs about the actual task that subjects realize, whereas at the ‘algorithmic’ level model comparison informs about the way subjects perform this task [30].”
Reasoning, Learning, and Creativity: Frontal Lobe Function and Human Decision-Making
Collins, A., & Koechlin, E. (2012) · PLoS Biology · doi:10.1371/journal.pbio.1001293 · Cited by 558 · cited 2× · see citing works
“These guidelines combine both relative model comparison criteria and model simulations ( [240_TD$DIFF]see [16,19,20] as examples of studies that include [241_TD$DIFF]all these steps).”
Self-citationVBA: A Probabilistic Treatment of Nonlinear Models for Neurobiological and Behavioural Data
Daunizeau, J., Adam, V., & Rigoux, L. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003441 · Cited by 385 · see citing works
Flagged first, then by citations
Research works printed but not linked: 10
- [6]6. Marr, D. Vision: A Computational Investigation of Visual Representation in Man . (Henry Holt and Co, 1982). doi:10.1007/s11097-009-9141-7 · DOI
- [7]7. O'Reilly, R. C. & Munakata, Y. Computational Explorations in Cognitive Neuroscience. J. Math. Psychol. 46, 504 (2000).
- [9]9. Corrado, G. S., Sugrue, L. P., Brown, J. R. & Newsome, W. T. in Neuroeconomics: Decision Making and the Brain (eds. Glimcher, P. W., Fehr, E., Camerer, C. F. & Poldrack, R. a) 463-480 (Academic Press, London, UK, 2009).
- [12]12. Dienes, Z. Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference . (Palgrave Macmillan, 2008). at <https://books.google.com/books?hl=en&lr=&id=qCQdBQAAQBAJ&pgis=1>
- [14]14. Dienes, Z. Bayesian Versus Orthodox Statistics: Which Side Are You On? Perspect. Psychol. Sci. 6, 274290 (2011).
- [15]15. Meyniel, F., Sergent, C., Rigoux, L., Daunizeau, J. & Pessiglione, M. Neurocomputational account of how the human brain decides when to have a break. Proc. Natl. Acad. Sci. U. S. A. 110, 2641-6 (2013).
- [16]16. Lebreton, M. et al. A Critical Role for the Hippocampus in the Valuation of Imagined Outcomes. PLoS Biol. 11, e1001684 (2013).
- [18]18. Palminteri, S., Khamassi, M., Joffily, M. & Coricelli, G. Contextual modulation of value signals in reward and punishment learning. Nat. Commun. 6, 8096 (2015).
- [19]19. Viejo, G., Khamassi, M., Brovelli, A. & Girard, B. Modeling choice and reaction time during arbitrary visuomotor learning through the coordination of adaptive working memory and reinforcement learning. Front. Behav. Neurosci. 9, 225 (2015).
- [26]26. Barto, A. G. & Sutton, R. S. Reinforcement Learning: An Introduction . (MIT Press, 1998). doi:10.1109/TNN.1998.712192 · DOI · Paper page
Cited by
Sorted by citations
Lack of theory building and testing impedes progress in the factor and network literature
Fried, E. I. (2020) · Psychological Inquiry · doi:10.1080/1047840x.2020.1853461 · Cited by 317 · see citing works
A computational reward learning account of social media engagement
Lindström, B., Bellander, M., Schultner, D. T., Chang, A., Tobler, P. N., & Amodio, D. M. (2021) · Nature Communications · doi:10.1038/s41467-020-19607-x · Cited by 185 · see citing works
Comparing Bayesian and non-Bayesian accounts of human confidence reports
Adler, W. T., & Ma, W. J. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006572 · Cited by 118 · see citing works
Forming global estimates of self-performance from local confidence
Rouault, M., Dayan, P., & Fleming, S. M. (2019) · Nature Communications · doi:10.1038/s41467-019-09075-3 · Cited by 134 · see citing works
Visual attention modulates the integration of goal-relevant evidence and not value
Sepulveda, P., Usher, M., Davies, N., Benson, A. A., Ortoleva, P., & De Martino, B. (2020) · eLife · doi:10.7554/elife.60705 · Cited by 121 · see citing works
A foundation model to predict and capture human cognition
Binz, M., Akata, E., Bethge, M., Brändle, F., Callaway, F., Coda-Forno, J., Dayan, P., Demircan, C., Eckstein, M. K., Éltető, N., et al. (2025) · Nature · doi:10.1038/s41586-025-09215-4 · Cited by 119 · see citing works
Neural computations underpinning the strategic management of influence in advice giving
Hertz, U., Palminteri, S., Brunetti, S., Olesen, C., Frith, C. D., & Bahrami, B. (2017) · Nature Communications · doi:10.1038/s41467-017-02314-5 · Cited by 77 · see citing works
Self-citationDepressive symptoms are associated with blunted reward learning in social contexts
Safra, L., Chevallier, C., & Palminteri, S. (2019) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1007224 · Cited by 40 · see citing works
Self-citationEckstein, M. K., Summerfield, C., Daw, N. D., & Miller, K. J. (2023) · doi:10.1101/2023.05.17.541226 · Cited by 18 · see citing works
Colas, J. T., O’Doherty, J. P., & Grafton, S. T. (2024) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1011950 · Cited by 6 · see citing works
When Can AI Models Explain Learning? Validity Criteria for AI as Cognitive Models in Education
Wang, P., Viberg, O., Law, E. L.-C., & Greiff, S. (2026) · Educational Psychology Review · doi:10.1007/s10648-026-10202-y · Cited by 3 · see citing works
Blaming luck, claiming skill: Self-attribution bias in error assignment
Okamoto, N., Taylor, M., Kubo, T., Ishii, S., De Martino, B., & Cortese, A. (2025) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1013787 · Cited by 2 · see citing works
- Authors
- Palminteri, S., Wyart, V., & Koechlin, E.
- Affiliations
- Inserm (FR)
- Corresponding author
- Palminteri, Stefano
- Wyart, Valentin
- Venue
- Trends in Cognitive Sciences
- Publisher
- Elsevier BV
- Published
- 2017-06
- Volume
- 21
- Issue
- 6
- Pages
- 425-433
- ISSN
- 1364-6613 (ISSN-L)
- OpenAlex
- W2610253745
- Cited by
- 448
- References
- 3
- Citations by year
- 2026: 45 · 2025: 48 · 2024: 54 · 2023: 50 · 2022: 47 · 2021: 49 · 2020: 53 · 2019: 52 · 2018: 37 · 2017: 13
- FWCI
- 4.75
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Trends in Cognitive Sciences (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Agence Nationale de la Recherche (ANR-10-IDEX-0001, ANR-14-CE13-0028, ANR-10-LABX-0087, ANR-16-NEUC-0004)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Cognitive Science and Education Research
- Social Sciences › Decision Sciences › General Decision Sciences › Decision-Making and Behavioral Economics
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Philosophy and History of Science
- Keywords
- falsification, computational cognitive modeling, model comparison, model complexity, model simulation, research guidelines
- MeSH terms
- Cognition, Humans, Models, Neurological, Research, Computational Biology, Cognitive Science
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
