Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 5 of the 20 sources its claims cite were checked for support.

Counterfeit judgments in large language models

Open access
Perc, Matjaž·2025·Proceedings of the National Academy of Sciences

Overview

CoChat Summary

TL;DR

This PNAS commentary by Perc discusses Loru et al.'s study showing that large language models can mimic human evaluative judgments of news credibility while relying on fundamentally different mechanisms—lexical associations, statistical priors, and surface-level cues rather than deliberative reasoning. Perc frames this divergence as "counterfeit judgments" and introduces "epistemia," the illusion of knowledge when linguistic plausibility substitutes for verification.

Key findings

  • Loru et al. benchmarked six LLMs against expert evaluations from NewsGuard and Media Bias/Fact Check and against human participants; at the surface level LLMs often aligned with expert classifications, especially in flagging unreliable or low-credibility outlets, but this masked differences in how judgments were produced.
  • Humans drew on contextual understanding, content comprehension, and pragmatic reasoning, whereas LLMs relied on lexical associations, statistical priors, and surface-level linguistic cues—pattern-based approximation rather than deliberative or normative reasoning.
  • Systematic asymmetries emerged: right-leaning outlets were more likely to be classified as unreliable, and models exhibited a "skepticism bias"—a tendency to overreject accurate information.
  • Both LLMs and humans rated "factual accuracy" as the most important criterion, but interpreted it differently: humans assessed accuracy through comprehension and contextual reasoning, while LLMs inferred it statistically. LLMs emphasized "ownership transparency" and institutional markers; humans focused on tone, clarity, and emotional neutrality.

Methods

  • This is a commentary on Loru et al. (ref. 4); it reports no new primary data of its own.
  • The reviewed study used a structured agentic framework in which both models and human subjects followed the same stepwise evaluation process: selecting assessment criteria, retrieving relevant information, and providing written justifications.
  • The focus was not on news content itself but on how evaluative reasoning unfolds within and across agents; six LLMs were compared against expert ratings and human participants.

Limitations

  • The commentary does not state formal limitations of its own argument.
  • Evident from the text: as a commentary, it synthesizes and interprets Loru et al.'s findings rather than presenting independent evidence, so its claims about LLM mechanisms rest on that single study.
  • The text notes that the mechanisms by which AI arrives at judgments "remain poorly understood," and that LLM judgments are "underpinned by mechanisms opaque to human scrutiny"—an acknowledged epistemic opacity.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1073/pnas.2528527122

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1. Parallel information pathways in human and artificial judgment. On the Right hand side, we have a typical information pathway of a human to reach a certain judgment. On the Left hand side, we have a generic large language model pathway with steps that very roughly correspond to human action. Research done by Loru et al. (4) finds consistent differences in judgments arrived at one way or the other. In fact, they conclude that lexical associations and statistical priors influence evaluations in ways that differ from human reasoning and normative criteria. This divergence is termed epistemia—the illusion of knowledge that emerges when surface plausibility replaces verification. More to the point, these findings are indicative not only of systematic performance asymmetries, but they reveal also a deep shift in the heuristics that underlie evaluative processes and raise fundamental questions about delegating judgment to LLMs.
Fig. 1. Parallel information pathways in human and artificial judgment. On the Right hand side, we have a typical information pathway of a human to reach a certain judgment. On the Left hand side, we have a generic large language model pathway with steps that very roughly correspond to human action. Research done by Loru et al. (4) finds consistent differences in judgments arrived at one way or the other. In fact, they conclude that lexical associations and statistical priors influence evaluations in ways that differ from human reasoning and normative criteria. This divergence is termed epistemia—the illusion of knowledge that emerges when surface plausibility replaces verification. More to the point, these findings are indicative not only of systematic performance asymmetries, but they reveal also a deep shift in the heuristics that underlie evaluative processes and raise fundamental questions about delegating judgment to LLMs.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

5 Valid · 0 Not valid · 13 Not checked · 15 Own results and interpretations

16 more claims

Create a free account to see all 33 claims and to verify the 14 references still open.

See all claims — free account
  • Probably not a substantial claim. 10
  • It may state two separate claims as one. 4

Create a free account to see these claims.

Citations

Cites

Show all 18 references

Research works printed but not linked: 1

  1. D. C. Dennett, The Problem with Counterfeit People (The Atlantic, 2023).

Cited by

See all 16

Metadata

Authors
Perc, M.
Affiliations
Korea University (KR)
Corresponding author
Perc, Matjaž
Venue
Proceedings of the National Academy of Sciences
Published
2025-11-24
Volume
122
Issue
48
Pages
e2528527122-e2528527122
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
OpenAlex
W4416584882
Cited by
16
References
18
Citations by year
2026: 16
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Law › Artificial Intelligence in Law
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
Keywords
large language models, hallucinations, factual accuracy, trustworthiness

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Counterfeit judgments in large language models. https://app.cochat.ai/papers/10.1073/pnas.2528527122
The paper
Perc, M. (2025). Counterfeit judgments in large language models. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2528527122

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.