GPT-4 Passes the Bar Exam
Open accessAbstract
In this paper, we experimentally evaluate the zero-shot performance of a preliminary version of GPT-4 against prior generations of GPT on the entire Uniform Bar Examination (UBE), including not only the multiple-choice Multistate Bar Examination (MBE), but also the open-ended Multistate Essay Exam (MEE) and Multistate Performance Test (MPT) components. On the MBE, GPT-4 significantly outperforms both human test-takers and prior models, demonstrating a 26% increase over ChatGPT and beating humans in five of seven subject areas. On the MEE and MPT, which have not previously been evaluated by scholars, GPT-4 scores an average of 4.2/6.0 as compared to much lower scores for ChatGPT. Graded across the UBE components, in the manner in which a human tast-taker would be, GPT-4 scores approximately 297 points, significantly in excess of the passing threshold for all UBE jurisdictions. These findings document not just the rapid and remarkable advance of large language model performance generally, but also the potential for such models to support the delivery of legal services in society.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ChatGPT-4 has been shown to outperform the average student in a variety of verbally-oriented tests and exams.
“The latest version, ChatGPT-4, has been shown to outperform the average student in a variety of verbally-oriented tests and exams (De Winter, 2023; Katz et al., 2023; Nori et al., 2023; OpenAI, 2023a).”
From Can ChatGPT be used to predict citation counts, readership, and social media interaction? An exploration among 2222 scientific abstracts (2024) · cites this paper as Katz et al., 2023
When LLMs are confronted with tasks requiring domain-specific expertise, such as medical and legal questions, they may exhibit pronounced hallucinations, often manifesting as factual fabrication.
“This limitation becomes particularly evident when LLMs are confronted with tasks that require domain-specific expertise, such as medical [179, 279] and legal [149, 353] questions, these models may exhibit pronounced hallucinations, often manifesting as factual fabrication.”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [149]
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 62 · Cited by 422
- Authors
- Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P.
- Affiliations
- Illinois Institute of Technology (US)
- Stanford University (US)
- Corresponding author
- Katz, Daniel Martin
- Venue
- Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences
- Publisher
- Elsevier BV
- Published
- 2023
- Volume
- 382
- Issue
- 2270
- Pages
- 20230254-20230254
- ISSN
- 1364-503X (ISSN-L)
- PMC
- PMC10894685
- OpenAlex
- W4392773006
- Cited by
- 422
- References
- 62
- Citations by year
- 2026: 94 · 2025: 116 · 2024: 126 · 2023: 86
- FWCI
- 538.69
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences (publishedVersion, cc-by)
- SSRN Electronic Journal (acceptedVersion)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
- Social Sciences › Law › Legal Education and Practice Innovations
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- GPT-4, bar examination, zero-shot performance, large language models, ChatGPT, legal education, legal services
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
