Judgments of learning distinguish humans from large language models in predicting memory
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Large language models (LLMs) increasingly mimic human cognition in various language-based tasks. However, their capacity for metacognition—particularly in predicting memory performance—remains unexplored. Here, we introduce a cross-agent prediction model to assess whether ChatGPT-based LLMs align with human judgments of learning (JOL), a metacognitive measure where individuals predict their own future memory performance. We tested humans and LLMs on pairs of sentences, one of which was a garden-path sentence—a sentence that initially misleads the reader toward an incorrect interpretation before requiring reanalysis. By manipulating contextual fit (fitting vs. unfitting sentences), we probed how intrinsic cues (i.e., relatedness) affect both LLM and human JOL. Our results revealed that while human JOL reliably predicted actual memory performance, none of the tested LLMs (GPT-3.5-turbo, GPT-4-turbo, and GPT-4o) demonstrated comparable predictive accuracy. This discrepancy emerged regardless of whether sentences appeared in fitting or unfitting contexts. These findings indicate that, despite LLMs’ demonstrated capacity to model human cognition at the object-level, they struggle at the meta-level, failing to capture the variability in individual memory predictions. By identifying this shortcoming, our study underscores the need for further refinements in LLMs’ self-monitoring abilities, which could enhance their utility in educational settings, personalized learning, and human–AI interactions. Strengthening LLMs’ metacognitive performance may reduce the reliance on human oversight, paving the way for more autonomous and seamless integration of AI into tasks requiring deeper cognitive awareness.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“In contrast, the task-based cognitive approaches discussed earlier in this paper36,38,40–43 do not commit this error. They use established experimental paradigms to empirically test whether human and LLM cognition converge or diverge on specific dimensions, without presupposing cognitive equivalence as a starting point.”
From Understanding large language models demands distinguishing human projection from machine cognition (2026) · cites this paper as [42]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry
Flavell, J. H. (1979) · American Psychologist · doi:10.1037/0003-066x.34.10.906 · Cited by 11,282
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Artificial Intelligence in Education: A Review
Chen, L., Chen, P., & Lin, Z. (2020) · IEEE Access · doi:10.1109/access.2020.2988510 · Cited by 4,133 · cited 3×
“This lack of metacognitive ability of LLMs is not confined to the classroom, it also hinders the basic interactions between humans and chatbots.”
Metamemory: A Theoretical Framework and New Findings
Nelson, T. O. (1990) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60053-5 · Cited by 2,583 · cited 2×
“This extension of the Nelson and Narens 15 framework is depicted in Fig.”
Monitoring one’s own knowledge during study: A cue-utilization approach to judgments of learning
Koriat, A. (1997) · Journal of Experimental Psychology: General · doi:10.1037/0096-3445.126.4.349 · Cited by 1,363 · cited 1×
“…for both LLMs and humans 25 as a basis for their predictive processing 26 , and the role of relatedness as an intrinsic cue in shaping human judgments during predicting 16,18 , we demonstrated the inability of LLMs to predict memory performance at the meta-level in the judgments of learning task.”
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264 · cited 1×
“These findings underscore the important roles of both JOL and context (fitting, unfitting) in predicting correct responses.”
ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · arXiv · doi:10.48550/arxiv.2303.15056 · Cited by 1,207 · cited 1×
“Beyond their success in NLP, LLMs also reveal notable”
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197 · cited 1×
“Beyond their success in NLP, LLMs also reveal notable”
Artificial Intelligence trends in education: a narrative overview
Chassignol, M., Khoroshavin, A., Klimova, A., & Bilyatdinova, A. (2018) · Procedia Computer Science · doi:10.1016/j.procs.2018.08.233 · Cited by 956 · cited 1×
“However, our study demonstrated that AI’s predictive capability for recognition memory, an essential element of learning, is insufficient and requires improvement for AI-based applications to be considered reliable tools in education 32,33 .”
Out of One, Many: Using Language Models to Simulate Human Samples
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023) · Political Analysis · doi:10.1017/pan.2023.2 · Cited by 783 · cited 2×
“LLMs, trained on massive sets of human-generated data, show advancements across various facets of natural language processing (NLP) (Manning et al., 6 ).”
Research works printed but not linked: 3
- Huff, M., & Ulakçı, E. Towards a Psychology of Machines: Large Language Models Predict Human Memory (No. arXiv:2403.05152). arXiv. http://arxiv.org/abs/2403.05152 (2024). · arXiv
- Peirce, J., Hirst, R., & MacAskill, M. Building experiments in PsychoPy (2nd edition). SAGE (2022).
- Fox, J. & Weisberg, S. An R companion to applied regression (Sage, 2010).
Cited by
Metacognition in LLMs: Foundations, Progress, and OpportunitiesPreprint
Liu, G., Gani, A., Lu, J., Thomas, J., Steyvers, M., & Cohan, A. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.11881 · Cited by 9
“Other psychology-based methodologies include the use of judgments of learning (i.e., self-predictions of future memory performance) [117] or vectorized dimensional measurements of metacognitive capabilities of LLMs [244].”
InfluentialDo AI know what they know? Exploring metacognition in LLMs
Iqbal, S. (2026) · Intelligent Data Analysis · doi:10.1177/1088467x261436903 · Cited by 3
Alomar, E. H. (2026) · Sustainability · doi:10.3390/su18105059 · Cited by 2
HUMAN METACOGNITION IN AI-ASSISTED DECISION- MAKING
Schneider, L. (2025) · AIS Electronic Library (AISeL) · Cited by 1
Lemieux, A., Kutcher, S., Tietcheu, B. R. G., Seitz, G., Trickovic, J., Archibald, D., Grosjean, S., Hogg, W. E., & Johnston, S. (2026) · BMJ Open · doi:10.1136/bmjopen-2025-115673 · Cited by 1
Metadata
- Authors
- Huff, M., & Ulakci, E.
- Affiliations
- Leibniz-Institut für Wissensmedien (DE)
- Corresponding author
- Huff, Markus
- ORCID
- Huff, Markus
- Venue
- Scientific Reports
- Published
- 2025-10-07
- Volume
- 15
- Issue
- 1
- Pages
- 35030-35030
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC12504528
- OpenAlex
- W4414907014
- Cited by
- 9
- References
- 35
- Citations by year
- 2026: 8 · 2025: 1
- FWCI
- 5.13
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Scientific Reports (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- Leibniz-Institut für Wissensmedien
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Life Sciences › Neuroscience › Cognitive Neuroscience › Neurobiology of Language and Bilingualism
- Keywords
- judgments of learning, metacognition, large language models, memory prediction, garden-path sentences, contextual fit, self-monitoring, human-AI comparison, ChatGPT, personalized learning
- MeSH terms
- Metacognition, Large Language Models, Adult, Cognition, Female, Humans, Judgment, Language, Learning, Male, Memory, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Judgments of learning distinguish humans from large language models in predicting memory. https://app.cochat.ai/papers/10.1038/s41598-025-22290-x
- The paper
- Huff, M., & Ulakci, E. (2025). Judgments of learning distinguish humans from large language models in predicting memory. Scientific Reports. https://doi.org/10.1038/s41598-025-22290-x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.