Easy Problems that LLMs Get Wrong
The DOI matches its Crossref record; 9 of the 10 references it cites were found in a registry or an index, and 1 are not checked yet.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For example, several LLMs still cannot correctly answer questions like whether one can seat John, Matthew, and Taylor at a small round table in such a way that John does not sit beside Matthew.”
From There is no such thing as conscious artificial intelligence (2025) · cites this paper as Huckle and Williams, 2025
In the citing paper
“Huckle and Williams (2025) identified several categories of LLM failings: (1) overfitting (resulting in, e.g., classifying prompts similar to well-known puzzles as equal to these puzzles despite significant changes); (2) lack of logic or common sense; (3) lack of spatial intelligence; (4) incorrect mathematical reasoning; (5) poor linguistic understanding; (6) popular science; (7) relational mis”
From There is no such thing as conscious artificial intelligence (2025) · cites this paper as Huckle and Williams, 2025
In the citing paper
“Second, they struggle with many aspects of reasoning (Shwartz, 2024;Zhou et al., 2024;Huckle and Williams, 2025).”
From There is no such thing as conscious artificial intelligence (2025) · cites this paper as Huckle and Williams, 2025
In the citing paper
“Remarkably, in the benchmark created by Huckle and Williams (2025), the average human performance (accuracy of approximately 85%) is better than the performance of even the most advanced models supposedly created for advanced reasoning (such as Gemini 2.5 Pro or OpenAI o3, accuracy of approximately 75%), let alone more widely available models such as GPT-4o (accuracy of approximately 40%) or Gem”
From There is no such thing as conscious artificial intelligence (2025) · cites this paper as Huckle, 2025
In the citing paper
“Remarkably, in the benchmark created by Huckle and Williams (2025), the average human performance (accuracy of approximately 85%) is better than the performance of even the most advanced models supposedly created for advanced reasoning (such as Gemini 2.5 Pro or OpenAI o3, accuracy of approximately 75%), let alone more widely available models such as GPT-4o (accuracy of approximately 40%) or Gem”
From There is no such thing as conscious artificial intelligence (2025) · cites this paper as Huckle and Williams, 2025
Citations
Cites
Datasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 303
The effect of sampling temperature on problem solving in large language models
Renze, M. (2024) · Findings of the Association for Computational Linguistics: EMNLP 2024 · doi:10.18653/v1/2024.findings-emnlp.432 · Cited by 212
Mathematics, word problems, common sense, and artificial intelligence
Davis, E. (2024) · Bulletin of the American Mathematical Society · doi:10.1090/bull/1828 · Cited by 125
Puzzle Solving using Reasoning of Large Language Models: A Survey
Giadikiaroglou, P., Lymperaiou, M., Filandrianos, G., & Stamou, G. B. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.646 · Cited by 109
Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligence
McIntosh, T. R., Susnjak, T., Arachchilage, N., Liu, T., Watters, P., & Halgamuge, M. N. (2024) · arXiv · doi:10.48550/arxiv.2402.09880 · Cited by 104
LLMs for Relational Reasoning: How Far are We?
Li, Z., Cao, Y., Xu, X., Jiang, J., Liu, X., Teo, Y. S., Lin, S., & Liu, Y. (2024) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1145/3643795.3648387 · Cited by 81
Datasets for Large Language Models: A Comprehensive SurveyPreprint
Liu, Y., Cao, J., Liu, C., et al. (2024) · Research Square · doi:10.21203/rs.3.rs-3996137/v1 · Cited by 57
Embedded or embodied? a review of Hubert Dreyfus' What Computers Still Can't Do
Collins, H. M. (1996) · Artificial Intelligence · doi:10.1016/0004-3702(96)00083-6 · Cited by 47
Enhancing Contextual Understanding of Mistral LLM with External Knowledge BasesPreprint
Sasaki, M., Watanabe, N., & Komanaka, T. (2024) · Research Square · doi:10.21203/rs.3.rs-4215447/v1 · Cited by 28
Do large language models have a legal duty to tell the truth?Preprint
Wachter, S., Mittelstadt, B., & Russell, C. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4771884 · Cited by 10
Research works printed but not linked: 7
- Asher, N., et al.: Limits for Learning with Language Models (2023). arXiv preprint arXiv:2306.12213 · arXiv
- Wu, W., et al.: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models. arXiv preprint arXiv:2404.03622 (2024) · arXiv
- Ahn, J., et al.: Large Language Models for Mathematical Reasoning: Progresses and Challenges. arXiv preprint arXiv:2402.00157 (2024) · arXiv
- Bordt, S., et al.: Elephants never forget: memorization and learning of tabular data in large language models. arXiv preprint arXiv:2404.06209 (2024) · arXiv
- Zhou, K., et al.: Don’t Make Your LLM an Evaluation Benchmark Cheater (2023). arXiv preprint arXiv:2311.01964 · arXiv
- Delétang, G., Ruoss, A., Duquenne, P.-A., et al.: Language Modeling Is Compression. arXiv preprint arXiv:2309.10668 (2024) · arXiv
- Ouyang, S., Zhang, J.M., Harman, M., Wang, M.: LLM is Like a Box of Chocolates: the Non-determinism of ChatGPT in Code Generation (2023). arXiv preprint arXiv:2308.02828 · arXiv
Cited by
Human–AI collectives most accurately diagnose clinical vignettes
Zöller, N., Berger, J., Lin, I., Fu, N., Komarneni, J., Barabucci, G., Laskowski, K., Shia, V., Harack, B., Chu, E. A., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2426153122 · Cited by 98
Take caution in using LLMs as human surrogates
Gao, Y., Lee, D., Burtch, G., & Fazelpour, S. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2501660122 · Cited by 52
There is no such thing as conscious artificial intelligence
Porębski, A., & Figura, J. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-05868-8 · Cited by 20
PDE-Controller: LLMs for Autoformalization and Reasoning of PDEsPreprint
Soroco, M., Song, J., Xia, M., Emond, K., Sun, W., & Chen, W. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2502.00963 · Cited by 27
Weber, M. T., Noll, R., Marchl, A., Facchinello, C., Grünewaldt, A., Hügel, C., Musleh, K., Wagner, T. O. F., Storf, H., & Schaaf, J. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf034 · Cited by 14
Metadata
- Authors
- Huckle, J., & Williams, S.
- Corresponding author
- Huckle, James
- Venue
- Lecture Notes in Networks and Systems
- Publisher
- Springer Nature Switzerland
- Published
- 2025
- Pages
- 313-332
- ISSN
- 2367-3370 (ISSN-L)
- 2367-3389
- OpenAlex
- W4408104541
- Cited by
- 18
- References
- 10
- Citations by year
- 2026: 8 · 2025: 10
- FWCI
- 17.07
- Type
- conference paper
- Language
- en
- Open access
- closed
- Licence
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Lecture notes in networks and systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- large language models, logical reasoning, spatial ability, language comprehension, prompt engineering, commonsense reasoning, human-in-the-loop, LLM limitations, training methods
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Easy Problems that LLMs Get Wrong. https://app.cochat.ai/papers/10.1007/978-3-031-84457-7_19
- The paper
- Huckle, J., & Williams, S. (2025). Easy Problems that LLMs Get Wrong. Lecture Notes in Networks and Systems. https://doi.org/10.1007/978-3-031-84457-7_19
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.