CoChat
Sign Up
Cited by 181Misinformation and Its ImpactsTopic ModelingAdversarial Robustness in Machine Learning

The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.

The Internal State of an LLM Knows When It’s Lying

Open access
Azaria, Amos, Mitchell, Tom·2023·Findings of the Association for Computational Linguistics: EMNLP 2023
DOI 10.18653/v1/2023.findings-emnlp.68
Open this paper in CoChat — free accountOpen PDF

Abstract

While Large Language Models (LLMs) have shown exceptional performance in various tasks, one of their most prominent drawbacks is generating inaccurate or false information with a confident tone.In this paper, we provide evidence that the LLM's internal state can be used to reveal the truthfulness of statements.This includes both statements provided to the LLM, and statements that the LLM itself generates.Our approach is to train a classifier that outputs the probability that a statement is truthful, based on the hidden layer activations of the LLM as it reads or generates the statement.Experiments demonstrate that given a set of test sentences, of which half are true and half false, our trained classifier achieves an average of 71% to 83% accuracy labeling which sentences are true versus false, depending on the LLM base model.Furthermore, we explore the relationship between our classifier's performance and approaches based on the probability assigned to the sentence by the LLM.We show that while LLM-assigned sentence probability is related to sentence truthfulness, this probability is also dependent on sentence length and the frequencies of words in the sentence, resulting in our trained classifier providing a more reliable approach to detecting truthfulness, highlighting its potential to enhance the reliability of LLM-generated content and its practical applicability in real-world scenarios.

Overview

Integrity signals

Registry

Crossref · 10.18653/v1/2023.findings-emnlp.68

Cites

18 of 18 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs tend to generate responses that are factually incorrect or completely fabricated (hallucinations).

    “At the same time, LLMs also tend to generate responses that are factually incorrect or completely fabricated ( hallucinations ) (Manakul et al., 2023; Peng et al., 2023; Azaria and Mitchell, 2023).”

    From NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails (2023) · cites this paper as Azaria and Mitchell, 2023

  • Azaria and Mitchell (2023) trained a multi-layer perception classifier using an LLM's hidden representations to predict truthfulness of a sentence, but this is a white-box approach that requires internal states and labelled data.

    “Recently, Azaria and Mitchell (2023) trained a multi-layer perception classifier where an LLM's hidden representations are used as inputs to predict the truthfulness of a sentence. However, this approach is a white-box approach that uses the internal states of the LLM, which may not be available through API calls, and requires labelled data for supervised training.”

    From SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (2023) · cites this paper as Azaria and Mitchell, 2023

  • Several studies have demonstrated that LLM's activations encapsulate an internal belief related to the truthfulness of its generated statements.

    “Several studies [13, 31] have demonstrated that LLM's activations encapsulate an internal belief related to the truthfulness of its generated statements.”

    From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [13]

  • Current lie-detector methods for LLMs are not yet fully reliable, and the probing methods proposed by Burns et al. and Azaria and Mitchell do not adequately generalize.

    “Their empirical results suggest that current lie-detector methods for LLMs are not yet fully reliable, and the probing methods proposed by Burns et al. [31] and Azaria and Mitchell [13] do not adequately generalize.”

    From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [13]

  • LLMs contain latent structures within their activation space that relate to beliefs about truthfulness.

    “Furthermore, a line of work [13, 31, 172, 220] has revealed that LLMs contain latent structures within their activation space that relate to beliefs about truthfulness.”

    From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [13]

Citations

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Related

Metadata

Authors
Azaria, A., & Mitchell, T.
Affiliations
  • Ariel University (IL)
  • Carnegie Mellon University (US)
ORCID
Venue
Findings of the Association for Computational Linguistics: EMNLP 2023
Publisher
Association for Computational Linguistics
Published
2023
Pages
967-976
DOI
10.18653/v1/2023.findings-emnlp.68
OpenAlex
W4389519449
Cited by
175
References
18
Citations by year
2026: 65 · 2025: 59 · 2024: 34 · 2023: 17
FWCI
162.52
Type
conference paper
Language
en
Open access
Licence
cc-by
Locations
  • Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Funders
Ministry of Science and Technology, Israel
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
large language models, LLM-generated text, hidden layer activations, confidence calibration, factual accuracy, word frequency effects

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review