Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks
Open accessAbstract
The impressive performance of recent language models across a wide range of tasks suggests that they possess a degree of abstract reasoning skills. Are these skills general and transferable, or specialized to specific tasks seen during pretraining? To disentangle these effects, we propose an evaluation framework based on "counterfactual" task variants that deviate from the default assumptions underlying standard tasks. Across a suite of 11 tasks, we observe nontrivial performance on the counterfactual variants, but nevertheless find that performance substantially and consistently degrades compared to the default conditions. This suggests that while current LMs may possess abstract task-solving skills to an extent, they often also rely on narrow, non-transferable procedures for task-solving. These results motivate a more careful interpretation of language model performance that teases apart these aspects of behavior.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Reasoning tests that break common co-occurrence patterns in the input or require multi-step operations lead to model failure.
“Reasoning tests that break common co-occurrence patterns in the input or require multi-step operations also lead to model failure [144, 145].”
From Dissociating language and thought in large language models (2024) · cites this paper as [145]
The most commonly cited reason for LLM formal reasoning failures is the failure of artificial neural nets to generalize to patterns outside their training distribution.
“The most commonly cited reason for these failures is the failure of artificial neural nets to generalize to patterns outside their training distribution [145, 146].”
From Dissociating language and thought in large language models (2024) · cites this paper as [145]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Still being checked.
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
“[215] showed that the current LLMs have certain limitations in abstract reasoning ability.”
BackgroundDissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345 · see citing works
Simple techniques to bypass GenAI text detectors: implications for inclusive education
Perkins, M., Roe, J., Vu, B. H., Postma, D., Hickerson, D., McGaughran, J., & Khuat, H. Q. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00487-w · Cited by 117 · see citing works
“Despite these achievements, Claude 2 lags in scientific writing and quantitative accuracy compared to other models (Chang et al., 2023; Lozić & Štular, 2023; Z. Wu et al., 2023), primarily because of the ability of GPT-4 and Bard to access the internet for information gathering.”
MethodExploring the role of large language models in the scientific method: from hypothesis to discovery
Zhang, Y., Khan, S. A., Mahmud, A., Yang, H., Lavin, A., Levin, M. G., Frey, J. G., Dunnmon, J., Evans, J., Bundy, A., et al. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00019-5 · Cited by 109 · see citing works
Pan, L., Saxon, M., Xu, W., Nathani, D., Wang, X., & Wang, W. Y. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00660 · Cited by 86 · see citing works
Limitations of large language models in clinical problem-solving arising from inflexible reasoning
Kim, J., Podlasek, A., Shidara, K., Liu, F., Alaa, A., & Bernardo, D. (2025) · Scientific Reports · doi:10.1038/s41598-025-22940-0 · Cited by 78 · see citing works
Evaluating the Utilities of Foundation Models in Single-cell Data Analysis
Liu, T., Li, K., Wang, Y., Li, H., & Zhao, H. (2023) · doi:10.1101/2023.09.08.555192 · Cited by 62 · see citing works
Schwieger, A., Angst, K., de Bardeci, M., Burrer, A., Cathomas, F., Ferrea, S., Grätz, F., Knorr, M. S., Kronenberg, G., Spiller, T. R., et al. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105654 · Cited by 45 · see citing works
Correctness Comparison of ChatGPT ‐4, Gemini, Claude‐3, and Copilot for Spatial Tasks
Hochmair, H. H., Juhász, L., & Kemp, T. (2024) · Transactions in GIS · doi:10.1111/tgis.13233 · Cited by 44 · see citing works
Nielbo, K. L., Karsdorp, F., Wevers, M., Lassche, A., Baglini, R., Kestemont, M., & Tahmasebi, N. (2024) · Nature Reviews Methods Primers · doi:10.1038/s43586-024-00302-w · Cited by 38 · see citing works
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · doi:10.31219/osf.io/ac4sp · Cited by 51 · see citing works
What is it like to be a bot? The world according to GPT-4
Lloyd, D. E. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1292675 · Cited by 12 · see citing works
Are Emergent Abilities in Large Language Models just In-Context Learning?
Lu, S., Bigoulaeva, I., Sachdeva, R., Madabushi, H. T., & Gurevych, I. (2023) · arXiv · doi:10.48550/arxiv.2309.01809 · Cited by 12 · see citing works
Lewis, M., & Mitchell, M. (2024) · arXiv · doi:10.48550/arxiv.2402.08955 · Cited by 10 · see citing works
What Algorithms can Transformers Learn? A Study in Length Generalization
Zhou, H., Bradley, A., Littwin, E., Razin, N., Saremi, O., Susskind, J., Bengio, S., & Nakkiran, P. (2023) · arXiv · doi:10.48550/arxiv.2310.16028 · Cited by 10 · see citing works
“However, when evaluated in controlled studies, Transformers often struggle with out-of-distribution generalization (Nogueira et al., 2021; Ontañón et al., 2022; Dziri et al., 2023; Wu et al., 2023; Saparov et al., 2023).”
BackgroundFletcher, A., & Benveniste, M. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009614801 · Cited by 11 · see citing works
Perils and Opportunities in Using Large Language Models in Psychological ResearchPreprint
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J. P., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2023) · doi:10.31219/osf.io/tg79n · Cited by 8 · see citing works
Perils and Opportunities in Using Large Language Models in Psychological ResearchPreprint
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J. P., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2023) · doi:10.31234/osf.io/d695y · Cited by 8 · see citing works
Evaluating the Utilities of Large Language Models in Single-cell Data AnalysisPreprint
Zhao, H., Liu, T., Li, K., Wang, Y., & Li, H. (2023) · Research Square · doi:10.21203/rs.3.rs-3376641/v1 · Cited by 7 · see citing works
Finding Blind Spots in Evaluator LLMs with Interpretable Checklists
Doddapaneni, S., Khan, M. S. U. R., Verma, S., & Khapra, M. M. (2024) · Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2024.emnlp-main.911 · Cited by 9 · see citing works
Reliable Reasoning Beyond Natural Language
Borazjanizadeh, N., & Piantadosi, S. T. (2024) · arXiv · doi:10.48550/arxiv.2407.11373 · Cited by 3 · see citing works
“However, despite their linguistic competence, LLMs often demonstrate significant limitations in their capacity to reason reliably and flexibly [20, 13, 40].”
LLMs can Design Sustainable Concrete – a Systematic BenchmarkPreprint
Völker, C., Rug, T., Jablonka, K. M., & Kruschwitz, S. (2024) · Research Square · doi:10.21203/rs.3.rs-3913272/v1 · Cited by 4 · see citing works
Krawiec, K., & Nowinowski, A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71167-1_13 · Cited by 2 · see citing works
Large Language Models' Understanding of Math: Source Criticism and ExtrapolationPreprint
Yousefzadeh, R., & Cao, X. (2023) · Research Square · doi:10.21203/rs.3.rs-3598853/v1 · Cited by 1 · see citing works
Looped Transformers for Length Generalization
Fan, Y., Du, Y., Ramchandran, K., & Lee, K. (2024) · arXiv · doi:10.48550/arxiv.2409.15647
Zhao, L. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5251016
Do Language Models Know Who Did What to Whom?
Denning, J. M., Guo, X., Snefjella, B., & Blank, I. (2026) · Open Mind · doi:10.1162/opmi.a.365 · Cited by 2 · see citing works
Li, Y., Gu, H., de Weghe, N. V., & Huang, H. (2026) · Journal of Geovisualization and Spatial Analysis · doi:10.1007/s41651-026-00275-2
- Authors
- Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., & Kim, Y.
- Affiliations
- Boston University (US)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2307.02477
- OpenAlex
- W4383473944
- Cited by
- 19
- References
- 0
- Citations by year
- 2026: 2 · 2025: 3 · 2024: 9 · 2023: 5
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (2238240, 2212310)
- Harvard University
- Institute for Catastrophic Loss Reduction
- Quest for Intelligence, Massachusetts Institute of Technology
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- abstract thinking, language models, transferability, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
