CoChat
Sign Up
PreprintCited by 181Artificial Intelligence in LawTopic ModelingLegal Education and Practice Innovations

GPT Takes the Bar Exam

Open access
Bommarito, Michael James, Katz, Daniel Martin·2022·SSRN Electronic Journal
DOI 10.2139/ssrn.4314839
Open this paper in CoChat — free account

Abstract

Nearly all jurisdictions in the United States require a professional license exam, commonly referred to as “the Bar Exam,” as a precondition for law practice. To even sit for the exam, most jurisdictions require that an applicant completes at least seven years of post-secondary education, including three years at an accredited law school. In addition, most test-takers also undergo weeks to months of further, exam-specific preparation. Despite this significant investment of time and capital, approximately one in five test-takers still score under the rate required to pass the exam on their first try. In the face of a complex task that requires such depth of knowledge, what, then, should we expect of the state of the art in “AI?” In this research, we document our experimental evaluation of the performance of OpenAI’s text-davinci-003 model, often-referred to as GPT-3.5, on the multistate multiple choice (MBE) section of the exam. While we find no benefit in fine-tuning over GPT-3.5’s zero-shot performance at the scale of our training data, we do find that hyperparameter optimization and prompt engineering positively impacted GPT-3.5’s zero-shot performance. For best prompt and parameters, GPT-3.5 achieves a headline correct rate of 50.3% on a complete NCBE MBE practice exam, significantly in excess of the 25% baseline guessing rate, and performs at a passing rate for both Evidence and Torts. GPT-3.5’s ranking of responses is also highly correlated with correctness; its top two and top three choices are correct 71% and 88% of the time, respectively, indicating very strong non-entailment performance. While our ability to interpret these results is limited by nascent scientific understanding of LLMs and the proprietary nature of GPT, we believe that these results strongly suggest that an LLM will pass the MBE component of the Bar Exam in the near future.

Integrity signals

Registry

Crossref · 10.2139/ssrn.4314839

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The rate of performance increase from text-davinci-001 to text-davinci-003 strongly suggested that passing performance could occur within the next 0-18 months.

    “the rate of performance increase from text-davinci-001 to text-davinci-003 strongly suggested that passing performance could ‘occur within the next 0–18 months’ [8].”

    From GPT-4 passes the bar exam (2024) · cites this paper as [8]

  • In recent prior work, a subset of authors demonstrated near-passing zero-shot performance of text-davinci-003 on the MBE.

    “in recent prior work [8], a subset of the authors demonstrated the near-passing zero-shot performance of text-davinci-003 on the multiple choice component (MBE) of the uniform bar exam (UBE)”

    From GPT-4 passes the bar exam (2024) · cites this paper as [8]

Cited by

Sorted by citations

Related

Authors
Bommarito, M. J., & Katz, D. M.
Affiliations
  • Michigan State University (US)
  • Illinois Institute of Technology (US)
Corresponding author
Katz, Daniel Martin
ORCID
Venue
SSRN Electronic Journal
Publisher
Elsevier BV
Published
2022
ISSN
1556-5068 (ISSN-L)
DOI
10.2139/ssrn.4314839
OpenAlex
W4313384047
Cited by
181
References
0
Citations by year
2026: 15 · 2025: 32 · 2024: 44 · 2023: 90
Type
posted content
Language
en
Open access
Locations
  • SSRN Electronic Journal (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Topics
  • Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Law › Legal Education and Practice Innovations
Keywords
bar examination, GPT-3.5, large language models, zero-shot learning, prompt engineering, hyperparameter optimization, legal education, law school admissions, exam preparation, response ranking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review