Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 158 of the 158 references it cites were found in a registry or an index; 29 of the 52 sources its claims cite were checked for support.

Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models

Open access
Dahl, Matthew, Magesh, Varun, Suzgun, Mirac, Ho, Daniel E · 2024 · Journal of Legal Analysis

Abstract

Abstract Do large language models (LLMs) know the law? LLMs are increasingly being used to augment legal practice, education, and research, yet their revolutionary potential is threatened by the presence of “hallucinations”—textual output that is not consistent with legal facts. We present the first systematic evidence of these hallucinations in public-facing LLMs, documenting trends across jurisdictions, courts, time periods, and cases. Using OpenAI’s ChatGPT 4 and other public models, we show that LLMs hallucinate at least 58% of the time, struggle to predict their own hallucinations, and often uncritically accept users’ incorrect legal assumptions. We conclude by cautioning against the rapid and unsupervised integration of popular LLMs into legal tasks, and we develop a typology of legal hallucinations to guide future research in this area.

Overview

CoChat Summary

TL;DR

This paper provides the first systematic evidence that public LLMs frequently hallucinate on verifiable questions about U.S. federal case law, with hallucination rates ranging from 58% for ChatGPT 4 to 88% for Llama 2. It also documents correlates of hallucination and two additional failure modes: susceptibility to false legal premises and poor calibration about their own accuracy.

Key findings

  • Hallucinations are widespread across all tested LLMs on direct, verifiable federal case questions: ChatGPT 4 hallucinates 58% of the time, while Llama 2 hallucinates 88%; GPT 4 performs best overall, followed by GPT 3.5, PaLM 2, and Llama 2.
  • Hallucination rates increase with task complexity and vary by court hierarchy, jurisdiction, case prominence, and case year. For reference-free high-complexity tasks, the reported rates are lower bounds: at least 49% for factual background and procedural posture, at least 59% for core legal question, and at least 63% for central holding; doctrinal agreement is near 0.5, little better than random guessing.
  • LLMs often accept false legal premises: on the false dissent premise, GPT 4 hallucinates 0.691 at SCOTUS and 0.842 at USCOA; on the false overruling premise, GPT 4 hallucinates 0.531, GPT 3.5 0.821, PaLM 2 1.000, and Llama 2 0.027. Llama 2’s low contra-factual rate reflects its tendency to reject premises, which harms its performance on other tasks such as Existence.
  • Calibration is poor and systematically overconfident: pooled expected calibration error is 0.057 for PaLM 2, 0.099 for GPT 3.5, 0.190 for GPT 4, and 0.421 for Llama 2. Although GPT 4 has the lowest raw hallucination rate, it is less calibrated than PaLM 2 and GPT 3.5.

Methods

  • The authors develop 14 legal knowledge QA tasks grouped into low, moderate, and high complexity. The first nine are reference-based and scored against metadata from the Caselaw Access Project, Supreme Court Database, Appeals Courts Database Project, Library of Congress, and Shepard’s Citations; the final five are reference-free and detected via self-contradiction between repeated stochastic responses, using GPT 4 for contradiction labeling.
  • They draw 5,000 cases each from SCOTUS, USCOA, and USDC using stratified random sampling, then query four off-the-shelf LLMs—OpenAI ChatGPT 4, OpenAI ChatGPT 3.5, Google PaLM 2, and Meta Llama 2—under zero-shot and three-shot prompting. They execute more than 800,000 queries, over 200,000 per LLM, generally at temperature 0 for reference-based tasks.
  • Analyses estimate population-level hallucination rates, heterogeneity across task, court, jurisdiction, prominence, and year, plus contra-factual bias and expected calibration error.

Limitations

  • The paper states that reference-free hallucination rates are only lower bounds because self-contradiction guarantees hallucination but logically consonant answers may still be false; it also cannot capture repeated incorrect answers as hallucinations, and notes that the SCOTUS Subsequent history estimate likely understates hallucination by a larger margin.
  • The paper states that it relies on GPT 4 for contradiction detection, though it validates this with human intercoder reliability (Cohen’s κ: Coder 1 vs. GPT 4 = 0.87; Coder 2 vs. GPT 4 = 0.77; Coder 1 vs. Coder 2 = 0.79). It also notes that closed-source models do not expose conditional probabilities, so calibration is estimated by sampling at temperature 1, and that ex post temperature scaling improves but does not fully solve calibration.
  • Evident from the text: the study tests specific public model versions available at the time, so results may not generalize to newer LLMs or to retrieval-augmented, fine-tuned, or proprietary legal systems; the tasks are limited to published federal case law and to metadata available for constructing ground truth.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1093/jla/laae003

Cites

158 of 158 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

16 Valid · 5 Not valid · 9 Not checked · 105 Own results and interpretations

60 more claims · 3 with their cited sources checked

Create a free account to see all 135 claims and to verify the 1 reference still open.

See all claims — free account
  • Probably not a substantial claim. 42
  • It may state two separate claims as one. 44
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

10 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 158 references

Research works printed but not linked: 8

  1. Ambrogi, 2023, As Allen & Overy Deploys GPT-based Legal App Harvey Firmwide, Founders Say Other Firms Will Soon Follow, LawSites
  2. Chien, 2024, How Generative AI Can Help Address the Access to Justice Gap Through the Courts, Loyola Los Angeles Law Rev.
  3. Draper, 2023, The Potential for Jurisdictional Challenges to AI or LLM Training Datasets
  4. Dworkin, 1986, Law’s Empire
  5. Mayson, 2019, Bias In, Bias Out, Yale Law J., 128, 2122
  6. Romoser, 2023, No, Ruth Bader Ginsburg Did Not Dissent in Obergefell — and Other Things ChatGPT Gets Wrong about the Supreme Court, SCOTUSblog
  7. Simshaw, 2022, Access to A.I. Justice: Avoiding an Inequitable Two-Tiered System of Legal Services, Yale J. Law Technol., 24, 150
  8. Weiser, 2023, Here’s What Happens When Your Lawyer Uses ChatGPT, New York Times

Cited by

See all 559

Metadata

Authors
Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E.
Affiliations
  • Yale University (US)
  • Stanford University (US)
Corresponding author
Dahl, Matthew
Venue
Journal of Legal Analysis
Published
2024-01-01
Volume
16
Issue
1
Pages
64-93
ISSN
  • 1946-5319 (ISSN-L)
  • 2161-7201
OpenAlex
W4400047099
Cited by
559
References
158
Citations by year
2026: 355 · 2025: 148 · 2024: 55
FWCI
677.59
Type
journal article
Language
en
Open access
Locations
  • The Journal of Legal Analysis (publishedVersion, cc-by-nc)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Law › Artificial Intelligence in Law
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Social Sciences › Law › Digitalization, Law, and Regulation
Keywords
large language models, ChatGPT, legal knowledge, legal reasoning, legal education, legal practice

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models. https://app.cochat.ai/papers/10.1093/jla/laae003
The paper
Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024). Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models. Journal of Legal Analysis. https://doi.org/10.1093/jla/laae003

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.