The DOI matches its Crossref record; 78 of the 79 references it cites were found in a registry or an index, and 1 are not checked yet; none of the 18 sources its claims cite is checked for support yet.
GPT-4 passes the bar exam
Open accessAbstract
In this paper, we experimentally evaluate the zero-shot performance of GPT-4 against prior generations of GPT on the entire uniform bar examination (UBE), including not only the multiple-choice multistate bar examination (MBE), but also the open-ended multistate essay exam (MEE) and multistate performance test (MPT) components. On the MBE, GPT-4 significantly outperforms both human test-takers and prior models, demonstrating a 26% increase over ChatGPT and beating humans in five of seven subject areas. On the MEE and MPT, which have not previously been evaluated by scholars, GPT-4 scores an average of 4.2/6.0 when compared with much lower scores for ChatGPT. Graded across the UBE components, in the manner in which a human test-taker would be, GPT-4 scores approximately 297 points, significantly in excess of the passing threshold for all UBE jurisdictions. These findings document not just the rapid and remarkable advance of large language model performance generally, but also the potential for such models to support the delivery of legal services in society. This article is part of the theme issue ‘A complexity science approach to law and governance’.
Verifying this paper
8 of 9 steps done · this page updates itself — you can leave and come back
- Registry recordVerified
- Full textRead the full text
- Reference list63 references found
- Reference checkChecking each cited work — 78 of 79
- StatisticsRecomputed
- Claims86 claims anchored to a quote
- Cited sourcesChecked
- Self-checkChecked
- VerificationFindings verified
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
GPT-4 achieved zero-shot passing performance on the entire Uniform Bar Exam, including the MBE, MEE, and MPT, with an overall UBE score of approximately 297 points. The authors argue this demonstrates rapid LLM progress and potential to support legal services delivery.
Key findings
- On the MBE, GPT-4 scored 75.7% accuracy, a 26.5% increase over ChatGPT, beating the average human test-taker by more than 7% and outperforming humans in five of seven subject areas.
- On the MEE, GPT-4 averaged 4.2/6.0 versus ChatGPT’s 3.0/6.0; on the MPT, GPT-4 averaged 4.2/6.0 versus ChatGPT’s 2.8/6.0.
- Combined UBE score was 297 points for GPT-4 and 213 points for ChatGPT, exceeding passing thresholds ranging from 260 to 273 points across UBE jurisdictions.
- GPT-4 was Pareto dominant over prior GPT models on the MBE, with largest gains in Contracts and Evidence, though Civil Procedure was its weakest subject.
Methods
- The authors evaluated zero-shot performance on the full UBE: official MBE practice questions and the most recently released July 2022 MEE and MPT questions.
- MBE grading was automated, with three runs per prompt/parameter set and average scores reported; OpenAI ran a contamination check on the test questions.
- MEE and MPT responses were graded by two authors—a tenured law professor and a multi-jurisdiction licensed attorney—using conservative scoring against state bar representative “good” answers, with additional peer review of selected samples.
Limitations
- The authors state they could not replicate the NCBE’s full-scale grading process, and MEE/MPT scoring involved subjective qualitative assessment despite conservative standards and peer checks.
- The paper states GPT-4 still made errors, including on a testamentary trust distribution, a civil procedure diversity jurisdiction question, and a real property future interest/Rule Against Perpetuities subquestion.
- The text makes evident that MEE/MPT evaluation used only the July 2022 exam, the MBE questions could not be released, and results reflect minimal zero-shot prompting rather than more exhaustive prompt engineering or few-shot techniques.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
46 Own result · 12 Own interpretation · 10 Own method · 9 Not checked yet · 9 No basis given
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| GPT-4 | 75.7% |”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it meets or exceeds the approximate passing threshold in all seven subject matter areas.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plainn = 7- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Civil Procedure is both the worst subject for GPT-4, ChatGPT and human test-takers.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Contracts and Evidence are the topics with the largest overall improvement.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“First, it has difficulty calculating the distribution of assets from a testamentary trust which has been deemed to be invalid.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 scores an average of 4.2/6.0 when compared with much lower scores for ChatGPT.”
§ GPT-4 passes the bar exam > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“a more than 35% raw increase in Evidence.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 is now Pareto dominant over all prior versions of GPT”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the MBE, GPT-4 significantly outperforms both human test-takers and prior models”
§ GPT-4 passes the bar exam > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the MBE, GPT-4 significantly outperforms both human test-takers and prior models”
§ GPT-4 passes the bar exam > Abstract
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4’s MBE score is not only more than 15% above the minimum passing threshold”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“for Torts, in roughly 15% of instances, GPT-4 is unable to eliminate even a single wrong answer”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“GPT-4 does an overall better job of addressing the question presented by properly citing the relevant law, connecting the law to the facts and otherwise staying on topic.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 achieves a nearly 40% increase over ChatGPT in Contracts”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4’s performance on the ‘Top 2’ task is roughly in line with the ‘Top 3’ performance of prior models.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“demonstrating a 26% increase over ChatGPT”
- Also stated in: § Multistate bar exam results · plain
“GPT-4 delivers a 26.5% increase in the accuracy over ChatGPT”
§ GPT-4 passes the bar exam > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Multistate bar exam results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“for the Contracts-related problems, GPT-4 is able to identify the right answer within its ‘Top 2’ choices in nearly 97% of instances.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While GPT-4 performs well on many questions, its output is not completely free of errors.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This basic dynamic is replicated across not only the balance of the MEE Evidence question but also across much of the MEE model output.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Civil Procedure is a topic where GPT-4 was able to generate 26% raw increase over ChatGPT.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“both GPT-4 and ChatGPT delivered more detailed results when both were provided with a single MEE subquestion to consider.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“beating humans in five of seven subject areas”
- Also stated in: § Multistate bar exam results · plain
“it outperforms the average NCBE BarNow test-taker in five out of seven categories (Civil Procedure, Contracts, Criminal Law and Procedure, Evidence and Real Property).”
§ GPT-4 passes the bar exam > Abstract
plaindirectn = 5- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Multistate bar exam results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Graded across the UBE components, in the manner in which a human test-taker would be, GPT-4 scores approximately 297 points”
§ GPT-4 passes the bar exam > Abstract
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“outperforms the average human test-taker by more than 7%.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 scores an average of 4.2/6.0 when compared with much lower scores for ChatGPT.”
§ GPT-4 passes the bar exam > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the three sub-questions that we assign the lowest scores, GPT-4 produces several notable errors.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4’s second best answer is highly correlated with correctness for all subjects other than Criminal Law and Procedure”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The vast majority of UBE states impose a minimum UBE passing score threshold between 260 and 270.”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Beyond the numeric results displayed in table 6, our grading procedure revealed some clear and important distinctions in the quality of the output as between GPT-4 and ChatGPT.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Finally, GPT-4 provides improper analysis on a real property (real estate) subquestion regarding both the proper designation of a Future Interest and the application of the Rule Against Perpetuities.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Here, GPT-4 correctly distinguishes between the ability of the Franklin court to annul the marriage and its inability to dispose of the parties’ property.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, even in problems for which we assign a lower grade, GPT-4 is often able to deliver a partial answer, such as identifying some, but not all, relevant legal principles or providing reasonable discussion of some of the facts that are relevant to the legal question.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Most real-life examinees who otherwise pass the Bar Exam are unable to complete an end-to-end MEE that is free from errors.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we also solicited analysis from peers who were provided with selected samples of the model’s responses and reached assessments that met or exceeded our own.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These findings document not just the rapid and remarkable advance of large language model performance generally”
§ GPT-4 passes the bar exam > Abstract
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“significantly in excess of the passing threshold for all UBE jurisdictions.”
- Also stated in: § Introduction · plain
“GPT-4 can ‘pass the Bar’ in all UBE jurisdictions.”
§ GPT-4 passes the bar exam > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Introduction · plain
In the paper
“the questions we processed within our test set are not within the provenance of model [42].”
§ GPT-4 passes the bar exam > Data and methods > Methods
What the sources say
- Not checked yetcites [42] Open AI. 2023 GPT-4 Technical Report. See https://cdn.openai.com/papers/gpt-4.pdf.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“its overall performance in rank-ordering responses demonstrates state-of-the-art capabilities for information retrieval tasks.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“our reliance on the state bars’ representative ‘good’ answers, our internal debates and conservative standard as well as the use of other individuals reduces the likelihood that our assessment is incorrect enough to alter the ultimate conclusions set forth in this paper.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“in recent prior work [8], a subset of the authors demonstrated the near-passing zero-shot performance of text-davinci-003 on the multiple choice component (MBE) of the uniform bar exam (UBE)”
§ GPT-4 passes the bar exam > Introduction
What the sources say
- Not checked yetcites [8] Bommarito MJ, Katz DM. 2022 GPT Takes the Bar Exam. Available at SSRN 4314839. source
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“This longer context window was critical to our analysis.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“More fundamentally, despite partial success on the problem, ChatGPT ultimately draws the incorrect legal conclusion. Columbia law and not Franklin law will likely govern the question of annulment.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“we were unable to replicate a full scale grading process identical to that undertaken by the NCBE in the actual administration of the bar exam.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similar to differences in the MEE scores, the differences in quality manifest themselves within the deeper details of the problem.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 24
- Probably not a substantial claim. 24
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-3.0 can vaguely recite some of the relevant principles and rules but does not properly connect those principles to the facts and consistently reaches improper legal conclusions.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT fails to properly identify all four prongs of Rule 702 of the Federal Rules of Evidence (FRE). This results in a failure to discuss Rule 702(a).”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4, by contrast, correctly discusses both FRE Rule 403 and Rule 404(b) and does not devote attention to extraneous issues.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“consider the second subquestion on the MEE Evidence problem (figures 7 and 10) where ChatGPT fails to address a relevant rule (FRE 403) and instead focuses significant attention on non-relevant rules (FRE 701 and 702).”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the average student answers more than three in ten questions incorrectly.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These type of patterns repeat themselves over the remainder of both MPT-1 and MPT-2.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The July 2022 MPT featured two questions, each over the publicly available ChatGPT 4096 token limit—5297 tokens for MPT-1 and 5188 tokens for MPT-2.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Subquestion three of MPT-1, as displayed in figures 16 and 20, provides another example of the distinction in quality between GPT-4 and ChatGPT.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, in the first subquestion of MPT-1, both GPT-4 and ChatGPT produce a lengthy and otherwise reasonable looking answer.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| MPT 1 - Hixon Marriages | 4.2/6.0 | 3.0/6.0 |”
- Also stated in: § Multistate performance test results · plain
“| MPT 2 - In re Briotti | 4.1/6.0 | 2.5/6.0 |”
- Also stated in: § Multistate performance test results · plain
“| overall score | 4.2 | 2.8 |”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Multistate performance test results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 obtains an overall UBE score of 297 points4 while ChatGPT obtains a score of 213 points.5”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In figure 14, GPT-4 not only correctly identifies that Columbia Law is controlling but provides reasonable arguments as to why Columbia should be selected under the ‘significant relationship’ test.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In figure 18, ChatGPT correctly identifies that it should address the Restatement of the Conflicts of Law as this is core to the answer. However, it incorrectly cites §6 as the proper source when it should cite §283 as the source of the ‘significant relationship’ test.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT fails to distinguish between the circumstances and assigns authority to the Franklin Courts that they do not possess.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both of the models we considered produced long-form answers to the respective problems, but there are some fairly straightforward differences in the quality of output produced.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As displayed in table 8, different jurisdictions impose different UBE passing score thresholds ranging from 260 (for states such as Alabama and Minnesota) to 273 points (for Arizona).”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although GPT-4 obtains within-category passing-level scores for both the MEE and MPT”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Next, it fails to grasp the call of the question and provides an incorrect answer on a civil procedure question regarding diversity jurisdiction after the joinder of a necessary party.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both GPT-4 and, to a lesser extent, ChatGPT were able to largely avoid the trap of citing legal principles, cases, or other materials which would otherwise appear to be on point but would not be responsive to the requirements of the MPT.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, as presented in table 5, our final MEE grade is 4.2 out of 6 points for GPT-4 and 3 out of 6 points for ChatGPT.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The complexity of the law [12–15] imposes real consequences for many individuals and organizations [16,17].”
§ GPT-4 passes the bar exam > Introduction
What the sources say
- Not checked yetcites [12] Bourcier D, Mazzega P. 2007 Toward measures of complexity in legal systems. In Proceedings of the 11th international conference on Artificial intelligence and law, pp. 211–215. source
- Not checked yetcites [13] Ruhl J, Katz D, Bommarito M. 2017 Harnessing legal complexity. Science 355, 1377- 1378. (10.1126/science.aag3013) 28360284 source
- Not checked yetcites [14] Bommarito II M, Katz D. 2017 Measuring and modeling the US regulatory ecosystem. J. Stat. Phys. 168, 1125- 1135. (10.1007/s10955-017-1846-3) source
- Not checked yetcites [15] Katz DM, Coupette C, Beckedorf J, Hartung D. 2020 Complex societies and the growth of the law. Sci. Rep. 10, 1- 14. (10.1038/s41598-020-73623-x) 33127960 PMC7599339 source
- Not checked yetcites [16] Staudt RW. 2008 All the wild possibilities: technology that attacks barriers to access to justice. Loy. LAL Rev. 42, 1117.
- Not checked yetcites [17] Sandefur RL, Teufel J. 2020 Assessing America’s Access to Civil Justice Crisis. UC Irvine L. Rev. 11, 753.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“the rate of performance increase from text-davinci-001 to text-davinci-003 strongly suggested that passing performance could ‘occur within the next 0–18 months’ [8].”
§ GPT-4 passes the bar exam > Introduction
What the sources say
- Not checked yetcites [8] Bommarito MJ, Katz DM. 2022 GPT Takes the Bar Exam. Available at SSRN 4314839. source
boostedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Two of this study’s authors, a tenured law professor and an attorney licensed in multiple jurisdictions, reviewed the model output and collaboratively assigned scores to each of the MEE questions.”
- Also stated in: § Multistate essay examination results · plain
“we debated each score with the view that we should be somewhat conservative in the assignment of MEE scores.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Multistate essay examination results · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“All grading is automated.”
§ GPT-4 passes the bar exam > Data and methods > Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“the ability to interpret such complex documents often requires years of study.”
§ GPT-4 passes the bar exam > Introduction
What the sources say
- Not checked yetcites [6] Mertz E. 2007 The language of law school: learning to ‘think like a lawyer’. USA: Oxford University Press. source
- Not checked yetcites [7] Martínez E, Mollica F, Gibson E. 2022 Poor writing, not specialized concepts, drives processing difficulty in legal language. Cognition 224, 105070. (10.1016/j.cognition.2022.105070) 35257980 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“legal systems have struggled to assist with the quantity, quality, and accessibility of legal services demanded by society [17–19].”
§ GPT-4 passes the bar exam > Introduction
What the sources say
- Not checked yetcites [17] Sandefur RL, Teufel J. 2020 Assessing America’s Access to Civil Justice Crisis. UC Irvine L. Rev. 11, 753.
- Not checked yetcites [18] Rhode DL. 2004 Access to justice. Oxford, UK: Oxford University Press.
- Not checked yetcites [19] Susskind RE. 2019 Online courts and the future of justice. Oxford, UK: Oxford University Press. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“a combined score of 266 points is enough to pass in jurisdictions such as Illinois, New York and the District of Columbia, while a score of 270 points would pass in the vast majority of states which use the UBE.”
§ GPT-4 passes the bar exam > The Uniform Bar Exam > Description of the Uniform Bar Exam
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Experiments involved three runs for each set of prompts and parameters; we report the average score across all runs in our results.”
§ GPT-4 passes the bar exam > Data and methods > Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“but also the potential for such models to support the delivery of legal services in society.”
§ GPT-4 passes the bar exam > Abstract
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: Bar exam performance is a valid indicator of a model's potential to support the delivery of legal services in society.
None of the sources cited for this claim have been checked yet.
In the paper
“Legal language is notoriously complex [3–5]”
§ GPT-4 passes the bar exam > Introduction
What the sources say
- Not checked yetcites [3] Friedrich R. 2021 Complexity and entropy in legal language. Front. Phys. 9, 671882. (10.3389/fphy.2021.671882) source
- Not checked yetcites [4] Katz DM, Bommarito MJ. 2014 Measuring the complexity of the law: the United States Code. Artif. Intell. Law 22, 337- 374. (10.1007/s10506-014-9160-8) source
- Not checked yetcites [5] Ruhl JB. 2007 Law’s complexity: a primer. Ga. St. UL Rev. 24, 885. (10.58948/0738-6206.1052)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These representative good answers are made available by state bar associations and reflect actual MEE and MPT answers produced by real examinees.”
§ GPT-4 passes the bar exam > Data and methods > Data
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“While the exact sequence of questions administered is not identical to any actual exam as administered”
§ GPT-4 passes the bar exam > Data and methods > Data
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Franklin Court cannot, without personal jurisdiction over Tucker, take action against a property outside its borders.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The UBE has three components (MBE, MEE and MPT) which are typically, but not always, weighed using the approach highlighted in table 1.”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4, like prior models, may still hallucinate sources, incorrectly interpret facts or fail to follow ethical requirements”
§ GPT-4 passes the bar exam > Conclusion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“As the NCBE describes it, ‘the MPT is not a test of substantive law; the Library materials provide sufficient substantive information to complete the task.’”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
What the sources say
- Not checked yetcites NCBE
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Consequently, the MPT requires a suspension of knowledge, whereby the examinee must, for the period of the test, imagine themselves in a jurisdiction that may contradict their actual knowledge of real law.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As we noted in the grading of the MEE, we recognize that there is a subjective aspect to this sort of analysis”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“its high MBE percentile provided it with substantial latitude; GPT-4 would likely pass even with a much lower MEE or MPT score in some or all jurisdictions.”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, thanks to assistance from OpenAI, we were able to use an ‘8K’ version of ChatGPT that has a wider context window of 8193 tokens, and could thus accommodate the length of the overall materials.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Prior models were unable to handle longer documents in zero-shot tasks due to their token limits.
None of the sources cited for this claim have been checked yet.
In the paper
“Indeed, the instructions provided with the test remind the test-taker that even ‘if the cases appear familiar to you, do not assume that they are precisely the same as you have read before. Read them thoroughly, as if they all were new to you.’”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In total, our analysis highlights that GPT-4 has indeed passed the Bar and has done so by a significant margin.”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Most jurisdictions use a six-point scale, and a score of four or higher is generally considered passing.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It should be noted that several of these topics where GPT-4 struggles are also areas where law students and bar examinees would also likely struggle.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The Rule Against Perpetuities is considered by many to be among the most difficult issues in all of law.
The paper relies on this without support.
Bar exam performance is a valid indicator of a model's potential to support the delivery of legal services in society.
“but also the potential for such models to support the delivery of legal services in society.”
“we sought an exemplary challenge to demonstrate this potential to both the legal domain and general scientific community.”
The paper relies on this without support.
The NCBE BarNow average performance by topic is comparable to the performance of human test-takers on the specific MBE exam used in this study.
“Table 3 offers both the NCBE Bar Now average accuracy by question category as well as the overall approximate national average MBE performance for recent test-takers.”
“it outperforms the average NCBE BarNow test-taker in five out of seven categories (Civil Procedure, Contracts, Criminal Law and Procedure, Evidence and Real Property).”
The paper relies on this without support.
The July 2022 MEE and MPT questions are representative of the UBE for the purpose of evaluating GPT-4's ability to pass the bar.
“For the MEE and the MPT, we collected the most recently released questions from the July 2022 Bar Examination.”
“GPT-4 can ‘pass the Bar’ in all UBE jurisdictions.”
The paper relies on this without support.
Most jurisdictions use a six-point scale, and a score of four or higher is generally considered passing.
“Most jurisdictions leverage the six-point scale, where a score of four or higher is generally considered passing.”
The paper relies on this without support.
The Rule Against Perpetuities is considered by many to be among the most difficult issues in all of law.
“In particular, the Rule Against Perpetuities is considered by many to be among the most difficult issues in all of law.”
The paper relies on this without support.
Most real-life examinees who otherwise pass the Bar Exam are unable to complete an end-to-end MEE that is free from errors.
“In addition, it should be mentioned that most real life examinees who otherwise pass the Bar Exam are unable to complete an end-to-end MEE that is free from errors.”
The paper relies on this without support.
Prior models were unable to handle longer documents in zero-shot tasks due to their token limits.
“As many have noted, prior models were unable to handle longer documents in zero-shot tasks due to their token limits.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“GPT has been lauded for its ability to exhibit human-level performance on a variety of tasks (e.g., passing the Bar Exam or acing the SAT test), and better performance than existing LLMs (47, 48).”
From GPT is an effective tool for multilingual psychological text analysis (2024) · cites this paper as [47]
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Advances in neural information processing systems 7
Cios, K. J., & Shields, M. E. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,093
A logical calculus of the ideas immanent in nervous activity
McCulloch, W. S., & Pitts, W. (1943) · The Bulletin of Mathematical Biophysics · doi:10.1007/bf02478259 · Cited by 18,566
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
A Survey of the Usages of Deep Learning for Natural Language Processing
Otter, D. W., Medina, J. R., & Kalita, J. K. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2979670 · Cited by 1,836
Research works printed but not linked: 13
- Tiersma PM. 1999 Legal language. Chicago, IL: University of Chicago Press.
- Rhode DL. 2004 Access to justice. Oxford, UK: Oxford University Press.
- Von Neumann J. 1958 The computer and the brain. New Haven, CT: Yale University Press.
- Vaswani A Shazeer N Parmar N Uszkoreit J Jones L Gomez AN Kaiser L Polosukhin I. 2017 Attention is all you need. In Advances in neural information processing systems vol. 30.
- Christiano PF Leike J Brown T Martic M Legg S Amodei D. 2017 Deep reinforcement learning from human preferences. Advances in neural information processing systems vol. 30.
- Devlin J Chang MW Lee K Toutanova K. 2018 BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019 pp. 4171–4186.
- Kenton JDMWC Toutanova LK. 2019 BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies pp. 4171–4186.
- Brown T Mann B Ryder N Subbiah M Kaplan JD Dhariwal P Neelakantan A Shyam P Sastry G Askell A Agarwal S. 2020 Language models are few-shot learners. In Advances in neural information processing systems pp. 1877–1901.
- Barrett D Hill F Santoro A Morcos A Lillicrap T. 2018 Measuring abstract reasoning in neural networks. In International conference on machine learning pp. 511–520. PMLR.
- Wang A Pruksachatkun Y Nangia N Singh A Michael J Hill F Levy O Bowman S. 2019 Superglue: a stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems 32.
- Fawei B Wyner A Pan J. 2016 Passing a USA national bar exam: a first corpus for experimentation. In LREC 2016 Tenth International Conference on Language Resources and Evaluation pp. 23–30.
- Wang Z Panda R Karlinsky L Feris R Sun H Kim Y. 2023 Multitask prompt tuning enables parameter-efficient transfer learning. In The Eleventh International Conference on Learning Representations.
- Wei J et al. 2022 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems 35 24 824-24 837.
Cited by
Sorted by citations
Grassini, S. (2023) · Education Sciences · doi:10.3390/educsci13070692 · Cited by 1,048
Yenduri, G., Ramalingam, M., Selvi, G. C., Supriya, Y., Srivastava, G., Maddikunta, P. K. R., Raj, G. D., Jhaveri, R. H., Prabadevi, B., Wang, W., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3389497 · Cited by 916
The promise and challenges of generative AI in education
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2394886 · Cited by 543
Bengesi, S., El-Sayed, H., Sarker, M. K., Houkpati, Y., Irungu, J., & Oladunni, T. (2024) · IEEE Access · doi:10.1109/access.2024.3397775 · Cited by 470
Metadata
- Authors
- Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P.
- Affiliations
- Illinois Institute of Technology (US)
- Casetext, Inc., USA
- Stanford University (US)
- Corresponding author
- Katz, Daniel Martin
- Venue
- Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
- Publisher
- The Royal Society
- Published
- 2024-02-26
- Volume
- 382
- Issue
- 2270
- Pages
- 20230254-20230254
- ISSN
- 1364-503X (ISSN-L)
- 1471-2962
- PMC
- PMC10894685
- OpenAlex
- W4392773006
- Semantic Scholar
- 0e8297549b4ec852ce8fd55dee9ae21501805af2
- Cited by
- 420
- References
- 63
- Citations by year
- 2026: 91 · 2025: 116 · 2024: 127
- FWCI
- 532.03
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences (publishedVersion, cc-by)
- SSRN Electronic Journal (acceptedVersion)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Law › Artificial Intelligence in Law
- Social Sciences › Law › Legal Education and Practice Innovations
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- GPT-4, bar examination, zero-shot performance, large language models, ChatGPT, legal education, legal services
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
