CoChat

Judgments of learning distinguish humans from large language models in predicting memory

Open access
Huff, Markus, Ulakci, Elanur · 2025 · Scientific Reports

The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.

Abstract

Large language models (LLMs) increasingly mimic human cognition in various language-based tasks. However, their capacity for metacognition—particularly in predicting memory performance—remains unexplored. Here, we introduce a cross-agent prediction model to assess whether ChatGPT-based LLMs align with human judgments of learning (JOL), a metacognitive measure where individuals predict their own future memory performance. We tested humans and LLMs on pairs of sentences, one of which was a garden-path sentence—a sentence that initially misleads the reader toward an incorrect interpretation before requiring reanalysis. By manipulating contextual fit (fitting vs. unfitting sentences), we probed how intrinsic cues (i.e., relatedness) affect both LLM and human JOL. Our results revealed that while human JOL reliably predicted actual memory performance, none of the tested LLMs (GPT-3.5-turbo, GPT-4-turbo, and GPT-4o) demonstrated comparable predictive accuracy. This discrepancy emerged regardless of whether sentences appeared in fitting or unfitting contexts. These findings indicate that, despite LLMs’ demonstrated capacity to model human cognition at the object-level, they struggle at the meta-level, failing to capture the variability in individual memory predictions. By identifying this shortcoming, our study underscores the need for further refinements in LLMs’ self-monitoring abilities, which could enhance their utility in educational settings, personalized learning, and human–AI interactions. Strengthening LLMs’ metacognitive performance may reduce the reliance on human oversight, paving the way for more autonomous and seamless integration of AI into tasks requiring deeper cognitive awareness.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41598-025-22290-x

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Extended metacognition framework illustrating intra-agent and cross-agent monitoring. The human side (blue) depicts the classic metacognitive loop: information flows upward from the object-level to the meta-level via monitoring, and downward via control. On the machine side (red), a parallel structure is shown, with an analogous distinction between meta- and object-level processing. The horizontal arrow depicts the cross-agent JOL process, while the dashed arrows show the analytic overlay testing how human and machine JOLs relate to human recognition outcomes. Finally, the arrows between the object-levels indicate that both agents received the same stimuli and contextual information. Together, the figure highlights (i) the classical monitoring–control cycle within humans, (ii) the extension to machines, and (iii) the novel cross-agent monitoring framework evaluated in this study.
Fig. 1 Extended metacognition framework illustrating intra-agent and cross-agent monitoring. The human side (blue) depicts the classic metacognitive loop: information flows upward from the object-level to the meta-level via monitoring, and downward via control. On the machine side (red), a parallel structure is shown, with an analogous distinction between meta- and object-level processing. The horizontal arrow depicts the cross-agent JOL process, while the dashed arrows show the analytic overlay testing how human and machine JOLs relate to human recognition outcomes. Finally, the arrows between the object-levels indicate that both agents received the same stimuli and contextual information. Together, the figure highlights (i) the classical monitoring–control cycle within humans, (ii) the extension to machines, and (iii) the novel cross-agent monitoring framework evaluated in this study.CC BY · PubMed Central
Fig. 2 Human memory performance as a function of JOL and context (fitting, unfitting).
Fig. 2 Human memory performance as a function of JOL and context (fitting, unfitting).CC BY · PubMed Central
Fig. 3 Density plots representing the results of the bootstrapping analysis of the slopes (human memory ~ JOL) as a function of context (fitting, unfitting) and model (human, gpt-3.5-turbo-0125, gpt-4-turbo-2024-04-09, gpt-4o-2024-05-13).
Fig. 3 Density plots representing the results of the bootstrapping analysis of the slopes (human memory ~ JOL) as a function of context (fitting, unfitting) and model (human, gpt-3.5-turbo-0125, gpt-4-turbo-2024-04-09, gpt-4o-2024-05-13).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 40 references

Research works printed but not linked: 3

  1. Huff, M., & Ulakçı, E. Towards a Psychology of Machines: Large Language Models Predict Human Memory (No. arXiv:2403.05152). arXiv. http://arxiv.org/abs/2403.05152 (2024). · arXiv
  2. Peirce, J., Hirst, R., & MacAskill, M. Building experiments in PsychoPy (2nd edition). SAGE (2022).
  3. Fox, J. & Weisberg, S. An R companion to applied regression (Sage, 2010).

Cited by

See all 13

Metadata

Authors
Huff, M., & Ulakci, E.
Affiliations
Leibniz-Institut für Wissensmedien (DE)
Corresponding author
Huff, Markus
Venue
Scientific Reports
Published
2025-10-07
Volume
15
Issue
1
Pages
35030-35030
ISSN
2045-2322 (ISSN-L)
OpenAlex
W4414907014
Cited by
9
References
35
Citations by year
2026: 8 · 2025: 1
FWCI
5.13
Type
journal article
Language
en
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
Leibniz-Institut für Wissensmedien
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Memory Processes and Influences
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neurobiology of Language and Bilingualism
Keywords
judgments of learning, metacognition, large language models, memory prediction, garden-path sentences, contextual fit, self-monitoring, human-AI comparison, ChatGPT, personalized learning
MeSH terms
Metacognition, Large Language Models, Adult, Cognition, Female, Humans, Judgment, Language, Learning, Male, Memory, Young Adult

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Judgments of learning distinguish humans from large language models in predicting memory. https://app.cochat.ai/papers/10.1038/s41598-025-22290-x
The paper
Huff, M., & Ulakci, E. (2025). Judgments of learning distinguish humans from large language models in predicting memory. Scientific Reports. https://doi.org/10.1038/s41598-025-22290-x

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.