CoChat
Sign Up
Cited by 420Artificial Intelligence in LawLegal Education and Practice InnovationsTopic Modeling

The DOI matches its Crossref record; 75 of the 77 references it cites were found in a registry or an index, and 2 are not checked yet; none of the 18 sources its claims cite is checked for support yet.

GPT-4 passes the bar exam

Open access
Katz, Daniel Martin, Bommarito, Michael James, Gao, Shang, Arredondo, Pablo·2024·Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
DOI 10.1098/rsta.2023.0254PMC PMC10894685
Open this paper in CoChat — free account

Abstract

In this paper, we experimentally evaluate the zero-shot performance of GPT-4 against prior generations of GPT on the entire uniform bar examination (UBE), including not only the multiple-choice multistate bar examination (MBE), but also the open-ended multistate essay exam (MEE) and multistate performance test (MPT) components. On the MBE, GPT-4 significantly outperforms both human test-takers and prior models, demonstrating a 26% increase over ChatGPT and beating humans in five of seven subject areas. On the MEE and MPT, which have not previously been evaluated by scholars, GPT-4 scores an average of 4.2/6.0 when compared with much lower scores for ChatGPT. Graded across the UBE components, in the manner in which a human test-taker would be, GPT-4 scores approximately 297 points, significantly in excess of the passing threshold for all UBE jurisdictions. These findings document not just the rapid and remarkable advance of large language model performance generally, but also the potential for such models to support the delivery of legal services in society. This article is part of the theme issue ‘A complexity science approach to law and governance’.

Checked 2026-10-09 · 2 references not checked yet

Overview

AI snapshot

Written from the paper’s full text.

TL;DR

GPT-4 achieved zero-shot passing performance on the entire Uniform Bar Exam, including the MBE, MEE, and MPT, with an overall UBE score of approximately 297 points. The authors argue this demonstrates rapid LLM progress and potential to support legal services delivery.

Key findings

  • On the MBE, GPT-4 scored 75.7% accuracy, a 26.5% increase over ChatGPT, beating the average human test-taker by more than 7% and outperforming humans in five of seven subject areas.
  • On the MEE, GPT-4 averaged 4.2/6.0 versus ChatGPT’s 3.0/6.0; on the MPT, GPT-4 averaged 4.2/6.0 versus ChatGPT’s 2.8/6.0.
  • Combined UBE score was 297 points for GPT-4 and 213 points for ChatGPT, exceeding passing thresholds ranging from 260 to 273 points across UBE jurisdictions.
  • GPT-4 was Pareto dominant over prior GPT models on the MBE, with largest gains in Contracts and Evidence, though Civil Procedure was its weakest subject.

Methods

  • The authors evaluated zero-shot performance on the full UBE: official MBE practice questions and the most recently released July 2022 MEE and MPT questions.
  • MBE grading was automated, with three runs per prompt/parameter set and average scores reported; OpenAI ran a contamination check on the test questions.
  • MEE and MPT responses were graded by two authors—a tenured law professor and a multi-jurisdiction licensed attorney—using conservative scoring against state bar representative “good” answers, with additional peer review of selected samples.

Limitations

  • The authors state they could not replicate the NCBE’s full-scale grading process, and MEE/MPT scoring involved subjective qualitative assessment despite conservative standards and peer checks.
  • The paper states GPT-4 still made errors, including on a testamentary trust distribution, a civil procedure diversity jurisdiction question, and a real property future interest/Rule Against Perpetuities subquestion.
  • The text makes evident that MEE/MPT evaluation used only the July 2022 exam, the MBE questions could not be released, and results reflect minimal zero-shot prompting rather than more exhaustive prompt engineering or few-shot techniques.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1098/rsta.2023.0254

Cites

75 of 77 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 of 18 cited sources checked · 0 supported · 0 flagged

Show all 43 claims

  • It may state two separate claims as one. 24
  • Probably not a substantial claim. 24
43 of 43

  • The paper relies on this without support.

    Bar exam performance is a valid indicator of a model's potential to support the delivery of legal services in society.

    “but also the potential for such models to support the delivery of legal services in society.”
    “we sought an exemplary challenge to demonstrate this potential to both the legal domain and general scientific community.”

  • The paper relies on this without support.

    The NCBE BarNow average performance by topic is comparable to the performance of human test-takers on the specific MBE exam used in this study.

    “Table 3 offers both the NCBE Bar Now average accuracy by question category as well as the overall approximate national average MBE performance for recent test-takers.”
    “it outperforms the average NCBE BarNow test-taker in five out of seven categories (Civil Procedure, Contracts, Criminal Law and Procedure, Evidence and Real Property).”

  • The paper relies on this without support.

    The July 2022 MEE and MPT questions are representative of the UBE for the purpose of evaluating GPT-4's ability to pass the bar.

    “For the MEE and the MPT, we collected the most recently released questions from the July 2022 Bar Examination.”
    “GPT-4 can ‘pass the Bar’ in all UBE jurisdictions.”

  • The paper relies on this without support.

    Most jurisdictions use a six-point scale, and a score of four or higher is generally considered passing.

    “Most jurisdictions leverage the six-point scale, where a score of four or higher is generally considered passing.”

  • The paper relies on this without support.

    The Rule Against Perpetuities is considered by many to be among the most difficult issues in all of law.

    “In particular, the Rule Against Perpetuities is considered by many to be among the most difficult issues in all of law.”

  • The paper relies on this without support.

    Most real-life examinees who otherwise pass the Bar Exam are unable to complete an end-to-end MEE that is free from errors.

    “In addition, it should be mentioned that most real life examinees who otherwise pass the Bar Exam are unable to complete an end-to-end MEE that is free from errors.”

  • The paper relies on this without support.

    Prior models were unable to handle longer documents in zero-shot tasks due to their token limits.

    “As many have noted, prior models were unable to handle longer documents in zero-shot tasks due to their token limits.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Flagged first, then by citations

Show all 63 references

Research works printed but not linked: 14

  1. Tiersma PM. 1999 Legal language. Chicago, IL: University of Chicago Press.
  2. Rhode DL. 2004 Access to justice. Oxford, UK: Oxford University Press.
  3. Von Neumann J. 1958 The computer and the brain. New Haven, CT: Yale University Press.
  4. Vaswani A Shazeer N Parmar N Uszkoreit J Jones L Gomez AN Kaiser L Polosukhin I. 2017 Attention is all you need. In Advances in neural information processing systems vol. 30.
  5. Christiano PF Leike J Brown T Martic M Legg S Amodei D. 2017 Deep reinforcement learning from human preferences. Advances in neural information processing systems vol. 30.
  6. Devlin J Chang MW Lee K Toutanova K. 2018 BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019 pp. 4171–4186.
  7. Kenton JDMWC Toutanova LK. 2019 BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies pp. 4171–4186.
  8. Brown T Mann B Ryder N Subbiah M Kaplan JD Dhariwal P Neelakantan A Shyam P Sastry G Askell A Agarwal S. 2020 Language models are few-shot learners. In Advances in neural information processing systems pp. 1877–1901.
  9. Barrett D Hill F Santoro A Morcos A Lillicrap T. 2018 Measuring abstract reasoning in neural networks. In International conference on machine learning pp. 511–520. PMLR.
  10. Wang A Pruksachatkun Y Nangia N Singh A Michael J Hill F Levy O Bowman S. 2019 Superglue: a stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems 32.
  11. Mitchell M Krakauer DC. 2022 The Debate Over Understanding in AI’s Large Language Models. (http://arxiv.org/abs/2210.13966) · arXiv
  12. Fawei B Wyner A Pan J. 2016 Passing a USA national bar exam: a first corpus for experimentation. In LREC 2016 Tenth International Conference on Language Resources and Evaluation pp. 23–30.
  13. Wang Z Panda R Karlinsky L Feris R Sun H Kim Y. 2023 Multitask prompt tuning enables parameter-efficient transfer learning. In The Eleventh International Conference on Learning Representations.
  14. Wei J et al. 2022 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems 35 24 824-24 837.

Cited by

Sorted by citations

Items 101–106 of 106

Related

See all

Metadata

Authors
Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P.
Affiliations
  • Illinois Institute of Technology (US)
  • Casetext, Inc., USA
  • Stanford University (US)
Corresponding author
Katz, Daniel Martin
ORCID
Venue
Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
Publisher
The Royal Society
Published
2024-02-26
Volume
382
Issue
2270
Pages
20230254-20230254
ISSN
  • 1364-503X (ISSN-L)
  • 1471-2962
DOI
10.1098/rsta.2023.0254
PMC
PMC10894685
OpenAlex
W4392773006
Semantic Scholar
0e8297549b4ec852ce8fd55dee9ae21501805af2
Cited by
420
References
63
Citations by year
2026: 91 · 2025: 116 · 2024: 127
FWCI
532.03
Type
journal article
Language
en
Open access
Licence
Locations
  • Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences (publishedVersion, cc-by)
  • SSRN Electronic Journal (acceptedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Law › Artificial Intelligence in Law
  • Social Sciences › Law › Legal Education and Practice Innovations
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
GPT-4, bar examination, zero-shot performance, large language models, ChatGPT, legal education, legal services

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review