The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index; 5 of the 20 sources its claims cite were checked for support.
Counterfeit judgments in large language models
Open accessOverview
CoChat Summary
TL;DR
This PNAS commentary by Perc discusses Loru et al.'s study showing that large language models can mimic human evaluative judgments of news credibility while relying on fundamentally different mechanisms—lexical associations, statistical priors, and surface-level cues rather than deliberative reasoning. Perc frames this divergence as "counterfeit judgments" and introduces "epistemia," the illusion of knowledge when linguistic plausibility substitutes for verification.
Key findings
- Loru et al. benchmarked six LLMs against expert evaluations from NewsGuard and Media Bias/Fact Check and against human participants; at the surface level LLMs often aligned with expert classifications, especially in flagging unreliable or low-credibility outlets, but this masked differences in how judgments were produced.
- Humans drew on contextual understanding, content comprehension, and pragmatic reasoning, whereas LLMs relied on lexical associations, statistical priors, and surface-level linguistic cues—pattern-based approximation rather than deliberative or normative reasoning.
- Systematic asymmetries emerged: right-leaning outlets were more likely to be classified as unreliable, and models exhibited a "skepticism bias"—a tendency to overreject accurate information.
- Both LLMs and humans rated "factual accuracy" as the most important criterion, but interpreted it differently: humans assessed accuracy through comprehension and contextual reasoning, while LLMs inferred it statistically. LLMs emphasized "ownership transparency" and institutional markers; humans focused on tone, clarity, and emotional neutrality.
Methods
- This is a commentary on Loru et al. (ref. 4); it reports no new primary data of its own.
- The reviewed study used a structured agentic framework in which both models and human subjects followed the same stepwise evaluation process: selecting assessment criteria, retrieving relevant information, and providing written justifications.
- The focus was not on news content itself but on how evaluative reasoning unfolds within and across agents; six LLMs were compared against expert ratings and human participants.
Limitations
- The commentary does not state formal limitations of its own argument.
- Evident from the text: as a commentary, it synthesizes and interprets Loru et al.'s findings rather than presenting independent evidence, so its claims about LLM mechanisms rest on that single study.
- The text notes that the mechanisms by which AI arrives at judgments "remain poorly understood," and that LLM judgments are "underpinned by mechanisms opaque to human scrutiny"—an acknowledged epistemic opacity.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures

Verification
Claims the paper makes
5 Valid · 0 Not valid · 13 Not checked · 15 Own results and interpretations
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, their interpretations of accuracy differed profoundly.”
What the sources say
- Supportedcites [4] E. Loru, The simulation of judgment in LLMs. Proc. Natl. Acad. Sci. U.S.A. 122, e2518443122 (2025). 41082665 PMC12557803 source
The Results section 'Human and LLM Credibility Assessment' (p. ~Results/Fig. 4C discussion) states that both groups prioritize 'Factual Accuracy' but that 'this convergence masks deeper differences in reasoning processes' — humans via deliberative content comprehension, LLMs via lexical associations and training patterns. The Conclusions repeat it: 'Both groups prioritize "Factual Accuracy" ... but the underlying mechanisms differ,' with humans interpreting accuracy through content comprehension and pragmatic reasoning and LLMs deriving it from statistical regularities. The paraphrase 'differed profoundly' matches the source's framing of a deeper/structural divergence. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“models tended to exhibit what the authors describe as a “skepticism bias”—a tendency to overreject accurate information”
What the sources say
- Supportedcites [4] E. Loru, The simulation of judgment in LLMs. Proc. Natl. Acad. Sci. U.S.A. 122, e2518443122 (2025). 41082665 PMC12557803 source
The Conclusions state that 'Models display a pattern reminiscent of the skepticism bias observed in humans—an overrejection of accurate information,' which matches both the label and the gloss in the citing sentence. The Results and Discussion (Human and LLM Credibility Assessment) section reinforces this, noting that 'LLMs reproduce some behavioral regularities observed in humans (e.g., skepticism bias)' and that the bias 'persists even under structured and controlled evaluation,' with Fig. 4B showing 77% of sources rated reliable by humans being classified unreliable by the LLM agents. The paper's own hedge ('reminiscent of', 'tendency') is at or below the hedging of the citing sentence ('tended to exhibit'), so no overstatement arises. (LLM, conf=0.90)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“humans focused on rhetorical features such as tone, clarity, and emotional neutrality.”
What the sources say
- Supportedcites [4] E. Loru, The simulation of judgment in LLMs. Proc. Natl. Acad. Sci. U.S.A. 122, e2518443122 (2025). 41082665 PMC12557803 source
The Conclusions passage states that, in the secondary criteria, 'humans give more weight to stylistic and rhetorical features such as tone and writing fluency,' and immediately adds that 'These preferences mirror known heuristics like processing fluency, whereby clarity and emotional neutrality enhance perceived truth' — covering all three features the claim names (tone, clarity, emotional neutrality). The Results and Discussion section (Human and LLM Credibility Assessment) corroborates this: humans 'tend to prioritize rhetorical and stylistic cues, such as "Language Manipulation" and "Writing Professionalism"', with emotional tone and fluency driving their judgments, in contrast to LLMs. The claim is a faithful paraphrase of the source's characterization of the human group's evaluative focus. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“However, this apparent convergence masked fundamental differences in how the judgments were produced.”
What the sources say
- Supportedcites [4] E. Loru, The simulation of judgment in LLMs. Proc. Natl. Acad. Sci. U.S.A. 122, e2518443122 (2025). 41082665 PMC12557803 source
The Conclusions section (p. of our copy) states exactly this: 'While model outputs often align with expert classifications—especially when flagging unreliable sources—this apparent agreement conceals a deeper divergence in the evaluative mechanisms themselves.' The same section adds that LLMs 'partially replicate behavioral regularities identified in psychology while relying on fundamentally different evaluative mechanisms,' and the Abstract frames it as 'the similarity concerns results, not the process.' The paraphrase therefore matches the source's own claim about convergence masking differences in how judgments are produced. (LLM, conf=0.92)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Right-leaning outlets were more likely to be classified as unreliable”
What the sources say
- Supportedcites [4] E. Loru, The simulation of judgment in LLMs. Proc. Natl. Acad. Sci. U.S.A. 122, e2518443122 (2025). 41082665 PMC12557803 source
The Results section 'LLMs vs. Expert-Driven Assessments' (p. Fig. 1B discussion) states that among domains rated Reliable by NewsGuard, 'Right-leaning outlets are consistently misclassified as Unreliable more often than Center or Left-leaning ones,' and the Fig. 1 caption confirms right-leaning outlets are consistently misclassified as unreliable while Center and Left-leaning ones are misclassified as reliable. The Conclusions restate it as a systematic asymmetry: 'Right-leaning outlets are disproportionately classified as unreliable.' The keyword analysis (Fig. 2) corroborates the direction, with right-leaning terms appearing almost exclusively in descriptions of Unreliable domains. (LLM, conf=0.92)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
16 more claims
Create a free account to see all 33 claims and to verify the 14 references still open.
- Probably not a substantial claim. 10
- It may state two separate claims as one. 4
Citations
Cites
The spreading of misinformation online
Del Vicario, M., Bessi, A., Zollo, F., Petroni, F., Scala, A., Caldarelli, G., Stanley, H. E., & Quattrociocchi, W. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1517441113 · Cited by 2,407
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090
Pennycook, G., & Rand, D. G. (2019) · Cognition · doi:10.1016/j.cognition.2018.06.011 · Cited by 2,022
The dark side of information: overload, anxiety and other paradoxes and pathologies
Bawden, D., & Robinson, L. (2009) · Journal of Information Science · doi:10.1177/0165551508095781 · Cited by 1,400
Computing Machinery and Intelligence
Turing, A. M. (2009) · Parsing the Turing Test · doi:10.1007/978-1-4020-6710-5_3 · Cited by 1,367
Cascella, M., Montomoli, J., Bellini, V., & Bignami, E. (2023) · Journal of Medical Systems · doi:10.1007/s10916-023-01925-4 · Cited by 1,120
AI and the transformation of social science research
Grossmann, I., Feinberg, M., Parker, D. C., Christakis, N. A., Tetlock, P. E., & Cunningham, W. A. (2023) · Science · doi:10.1126/science.adi1778 · Cited by 377
Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus?
Horton, J., Filippas, A., & Manning, B. (2023) · National Bureau of Economic Research · doi:10.3386/w31122 · Cited by 329
Mathematical foundations of moral preferences
Capraro, V., & Perc, M. (2021) · Journal of The Royal Society Interface · doi:10.1098/rsif.2020.0880 · Cited by 310
Self-citation
Research works printed but not linked: 1
- D. C. Dennett, The Problem with Counterfeit People (The Atlantic, 2023).
Cited by
On the optimal integration of intelligent agents into network systems to steer cooperation
Fu, F., Chen, X., & Christakis, N. A. (2026) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2537939123 · Cited by 4
Wolny, R., Hoffmann-Burdzińska, K., Jaciow, M., Sączewska-Piotrowska, A., Stolecka-Makowska, A., & Szojda, G. (2026) · Sustainability · doi:10.3390/su18020619 · Cited by 3
Complexities, gaps, and risks in simulating AI empathy
Carlo, G., Ahn, J., & Cervera, K. (2026) · AI Brain and Child · doi:10.1007/s44436-026-00037-3 · Cited by 3
A closer look at how large language models ‘trust' humans: patterns and biases
Lerman, V., & Dover, Y. (2026) · Proceedings of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rspa.2025.1113 · Cited by 2
Haim, E., van den Bergh, L., Siew, C. S. Q., Kenett, Y. N., Marinazzo, D., & Stella, M. (2026) · Journal of Complex Networks · doi:10.1093/comnet/cnag004 · Cited by 1
Metadata
- Authors
- Perc, M.
- Affiliations
- Korea University (KR)
- Corresponding author
- Perc, Matjaž
- ORCID
- Perc, Matjaž
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2025-11-24
- Volume
- 122
- Issue
- 48
- Pages
- e2528527122-e2528527122
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC12685031
- OpenAlex
- W4416584882
- Cited by
- 16
- References
- 18
- Citations by year
- 2026: 16
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Law › Artificial Intelligence in Law
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Keywords
- large language models, hallucinations, factual accuracy, trustworthiness
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-14, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-14 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Counterfeit judgments in large language models. https://app.cochat.ai/papers/10.1073/pnas.2528527122
- The paper
- Perc, M. (2025). Counterfeit judgments in large language models. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2528527122
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.