The DOI matches its Crossref record; 75 of the 77 references it cites were found in a registry or an index, and 2 are not checked yet; none of the 18 sources its claims cite is checked for support yet.
GPT-4 passes the bar exam
Open accessAbstract
In this paper, we experimentally evaluate the zero-shot performance of GPT-4 against prior generations of GPT on the entire uniform bar examination (UBE), including not only the multiple-choice multistate bar examination (MBE), but also the open-ended multistate essay exam (MEE) and multistate performance test (MPT) components. On the MBE, GPT-4 significantly outperforms both human test-takers and prior models, demonstrating a 26% increase over ChatGPT and beating humans in five of seven subject areas. On the MEE and MPT, which have not previously been evaluated by scholars, GPT-4 scores an average of 4.2/6.0 when compared with much lower scores for ChatGPT. Graded across the UBE components, in the manner in which a human test-taker would be, GPT-4 scores approximately 297 points, significantly in excess of the passing threshold for all UBE jurisdictions. These findings document not just the rapid and remarkable advance of large language model performance generally, but also the potential for such models to support the delivery of legal services in society. This article is part of the theme issue ‘A complexity science approach to law and governance’.
Checked 2026-10-09 · 2 references not checked yet
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
GPT-4 achieved zero-shot passing performance on the entire Uniform Bar Exam, including the MBE, MEE, and MPT, with an overall UBE score of approximately 297 points. The authors argue this demonstrates rapid LLM progress and potential to support legal services delivery.
Key findings
- On the MBE, GPT-4 scored 75.7% accuracy, a 26.5% increase over ChatGPT, beating the average human test-taker by more than 7% and outperforming humans in five of seven subject areas.
- On the MEE, GPT-4 averaged 4.2/6.0 versus ChatGPT’s 3.0/6.0; on the MPT, GPT-4 averaged 4.2/6.0 versus ChatGPT’s 2.8/6.0.
- Combined UBE score was 297 points for GPT-4 and 213 points for ChatGPT, exceeding passing thresholds ranging from 260 to 273 points across UBE jurisdictions.
- GPT-4 was Pareto dominant over prior GPT models on the MBE, with largest gains in Contracts and Evidence, though Civil Procedure was its weakest subject.
Methods
- The authors evaluated zero-shot performance on the full UBE: official MBE practice questions and the most recently released July 2022 MEE and MPT questions.
- MBE grading was automated, with three runs per prompt/parameter set and average scores reported; OpenAI ran a contamination check on the test questions.
- MEE and MPT responses were graded by two authors—a tenured law professor and a multi-jurisdiction licensed attorney—using conservative scoring against state bar representative “good” answers, with additional peer review of selected samples.
Limitations
- The authors state they could not replicate the NCBE’s full-scale grading process, and MEE/MPT scoring involved subjective qualitative assessment despite conservative standards and peer checks.
- The paper states GPT-4 still made errors, including on a testamentary trust distribution, a civil procedure diversity jurisdiction question, and a real property future interest/Rule Against Perpetuities subquestion.
- The text makes evident that MEE/MPT evaluation used only the July 2022 exam, the MBE questions could not be released, and results reflect minimal zero-shot prompting rather than more exhaustive prompt engineering or few-shot techniques.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
46 Own result · 12 Own interpretation · 10 Own method · 9 Not checked yet · 9 No basis given
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| GPT-4 | 75.7% |”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it meets or exceeds the approximate passing threshold in all seven subject matter areas.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plainn = 7- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Civil Procedure is both the worst subject for GPT-4, ChatGPT and human test-takers.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Contracts and Evidence are the topics with the largest overall improvement.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“First, it has difficulty calculating the distribution of assets from a testamentary trust which has been deemed to be invalid.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 scores an average of 4.2/6.0 when compared with much lower scores for ChatGPT.”
§ GPT-4 passes the bar exam > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“a more than 35% raw increase in Evidence.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 is now Pareto dominant over all prior versions of GPT”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the MBE, GPT-4 significantly outperforms both human test-takers and prior models”
§ GPT-4 passes the bar exam > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the MBE, GPT-4 significantly outperforms both human test-takers and prior models”
§ GPT-4 passes the bar exam > Abstract
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 24
- Probably not a substantial claim. 24
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-3.0 can vaguely recite some of the relevant principles and rules but does not properly connect those principles to the facts and consistently reaches improper legal conclusions.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT fails to properly identify all four prongs of Rule 702 of the Federal Rules of Evidence (FRE). This results in a failure to discuss Rule 702(a).”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4, by contrast, correctly discusses both FRE Rule 403 and Rule 404(b) and does not devote attention to extraneous issues.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“consider the second subquestion on the MEE Evidence problem (figures 7 and 10) where ChatGPT fails to address a relevant rule (FRE 403) and instead focuses significant attention on non-relevant rules (FRE 701 and 702).”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the average student answers more than three in ten questions incorrectly.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These type of patterns repeat themselves over the remainder of both MPT-1 and MPT-2.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The July 2022 MPT featured two questions, each over the publicly available ChatGPT 4096 token limit—5297 tokens for MPT-1 and 5188 tokens for MPT-2.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Subquestion three of MPT-1, as displayed in figures 16 and 20, provides another example of the distinction in quality between GPT-4 and ChatGPT.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, in the first subquestion of MPT-1, both GPT-4 and ChatGPT produce a lengthy and otherwise reasonable looking answer.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| MPT 1 - Hixon Marriages | 4.2/6.0 | 3.0/6.0 |”
- Also stated in: § Multistate performance test results · plain
“| MPT 2 - In re Briotti | 4.1/6.0 | 2.5/6.0 |”
- Also stated in: § Multistate performance test results · plain
“| overall score | 4.2 | 2.8 |”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Multistate performance test results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 obtains an overall UBE score of 297 points4 while ChatGPT obtains a score of 213 points.5”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In figure 14, GPT-4 not only correctly identifies that Columbia Law is controlling but provides reasonable arguments as to why Columbia should be selected under the ‘significant relationship’ test.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In figure 18, ChatGPT correctly identifies that it should address the Restatement of the Conflicts of Law as this is core to the answer. However, it incorrectly cites §6 as the proper source when it should cite §283 as the source of the ‘significant relationship’ test.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT fails to distinguish between the circumstances and assigns authority to the Franklin Courts that they do not possess.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both of the models we considered produced long-form answers to the respective problems, but there are some fairly straightforward differences in the quality of output produced.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As displayed in table 8, different jurisdictions impose different UBE passing score thresholds ranging from 260 (for states such as Alabama and Minnesota) to 273 points (for Arizona).”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although GPT-4 obtains within-category passing-level scores for both the MEE and MPT”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Next, it fails to grasp the call of the question and provides an incorrect answer on a civil procedure question regarding diversity jurisdiction after the joinder of a necessary party.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both GPT-4 and, to a lesser extent, ChatGPT were able to largely avoid the trap of citing legal principles, cases, or other materials which would otherwise appear to be on point but would not be responsive to the requirements of the MPT.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, as presented in table 5, our final MEE grade is 4.2 out of 6 points for GPT-4 and 3 out of 6 points for ChatGPT.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The complexity of the law [12–15] imposes real consequences for many individuals and organizations [16,17].”
§ GPT-4 passes the bar exam > Introduction
What the sources say
- Not checked yetcites [12] Bourcier D, Mazzega P. 2007 Toward measures of complexity in legal systems. In Proceedings of the 11th international conference on Artificial intelligence and law, pp. 211–215. source
- Not checked yetcites [13] Ruhl J, Katz D, Bommarito M. 2017 Harnessing legal complexity. Science 355, 1377- 1378. (10.1126/science.aag3013) 28360284 source
- Not checked yetcites [14] Bommarito II M, Katz D. 2017 Measuring and modeling the US regulatory ecosystem. J. Stat. Phys. 168, 1125- 1135. (10.1007/s10955-017-1846-3) source
- Not checked yetcites [15] Katz DM, Coupette C, Beckedorf J, Hartung D. 2020 Complex societies and the growth of the law. Sci. Rep. 10, 1- 14. (10.1038/s41598-020-73623-x) 33127960 PMC7599339 source
- Not checked yetcites [16] Staudt RW. 2008 All the wild possibilities: technology that attacks barriers to access to justice. Loy. LAL Rev. 42, 1117.
- Not checked yetcites [17] Sandefur RL, Teufel J. 2020 Assessing America’s Access to Civil Justice Crisis. UC Irvine L. Rev. 11, 753.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“the rate of performance increase from text-davinci-001 to text-davinci-003 strongly suggested that passing performance could ‘occur within the next 0–18 months’ [8].”
§ GPT-4 passes the bar exam > Introduction
What the sources say
- Not checked yetcites [8] Bommarito MJ, Katz DM. 2022 GPT Takes the Bar Exam. Available at SSRN 4314839. source
boostedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Two of this study’s authors, a tenured law professor and an attorney licensed in multiple jurisdictions, reviewed the model output and collaboratively assigned scores to each of the MEE questions.”
- Also stated in: § Multistate essay examination results · plain
“we debated each score with the view that we should be somewhat conservative in the assignment of MEE scores.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Multistate essay examination results · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“All grading is automated.”
§ GPT-4 passes the bar exam > Data and methods > Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“the ability to interpret such complex documents often requires years of study.”
§ GPT-4 passes the bar exam > Introduction
What the sources say
- Not checked yetcites [6] Mertz E. 2007 The language of law school: learning to ‘think like a lawyer’. USA: Oxford University Press. source
- Not checked yetcites [7] Martínez E, Mollica F, Gibson E. 2022 Poor writing, not specialized concepts, drives processing difficulty in legal language. Cognition 224, 105070. (10.1016/j.cognition.2022.105070) 35257980 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“legal systems have struggled to assist with the quantity, quality, and accessibility of legal services demanded by society [17–19].”
§ GPT-4 passes the bar exam > Introduction
What the sources say
- Not checked yetcites [17] Sandefur RL, Teufel J. 2020 Assessing America’s Access to Civil Justice Crisis. UC Irvine L. Rev. 11, 753.
- Not checked yetcites [18] Rhode DL. 2004 Access to justice. Oxford, UK: Oxford University Press.
- Not checked yetcites [19] Susskind RE. 2019 Online courts and the future of justice. Oxford, UK: Oxford University Press. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“a combined score of 266 points is enough to pass in jurisdictions such as Illinois, New York and the District of Columbia, while a score of 270 points would pass in the vast majority of states which use the UBE.”
§ GPT-4 passes the bar exam > The Uniform Bar Exam > Description of the Uniform Bar Exam
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Experiments involved three runs for each set of prompts and parameters; we report the average score across all runs in our results.”
§ GPT-4 passes the bar exam > Data and methods > Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“but also the potential for such models to support the delivery of legal services in society.”
§ GPT-4 passes the bar exam > Abstract
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: Bar exam performance is a valid indicator of a model's potential to support the delivery of legal services in society.
None of the sources cited for this claim have been checked yet.
In the paper
“Legal language is notoriously complex [3–5]”
§ GPT-4 passes the bar exam > Introduction
What the sources say
- Not checked yetcites [3] Friedrich R. 2021 Complexity and entropy in legal language. Front. Phys. 9, 671882. (10.3389/fphy.2021.671882) source
- Not checked yetcites [4] Katz DM, Bommarito MJ. 2014 Measuring the complexity of the law: the United States Code. Artif. Intell. Law 22, 337- 374. (10.1007/s10506-014-9160-8) source
- Not checked yetcites [5] Ruhl JB. 2007 Law’s complexity: a primer. Ga. St. UL Rev. 24, 885. (10.58948/0738-6206.1052)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These representative good answers are made available by state bar associations and reflect actual MEE and MPT answers produced by real examinees.”
§ GPT-4 passes the bar exam > Data and methods > Data
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“While the exact sequence of questions administered is not identical to any actual exam as administered”
§ GPT-4 passes the bar exam > Data and methods > Data
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Franklin Court cannot, without personal jurisdiction over Tucker, take action against a property outside its borders.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The UBE has three components (MBE, MEE and MPT) which are typically, but not always, weighed using the approach highlighted in table 1.”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4, like prior models, may still hallucinate sources, incorrectly interpret facts or fail to follow ethical requirements”
§ GPT-4 passes the bar exam > Conclusion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“As the NCBE describes it, ‘the MPT is not a test of substantive law; the Library materials provide sufficient substantive information to complete the task.’”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
What the sources say
- Not checked yetcites NCBE
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Consequently, the MPT requires a suspension of knowledge, whereby the examinee must, for the period of the test, imagine themselves in a jurisdiction that may contradict their actual knowledge of real law.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As we noted in the grading of the MEE, we recognize that there is a subjective aspect to this sort of analysis”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“its high MBE percentile provided it with substantial latitude; GPT-4 would likely pass even with a much lower MEE or MPT score in some or all jurisdictions.”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, thanks to assistance from OpenAI, we were able to use an ‘8K’ version of ChatGPT that has a wider context window of 8193 tokens, and could thus accommodate the length of the overall materials.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Prior models were unable to handle longer documents in zero-shot tasks due to their token limits.
None of the sources cited for this claim have been checked yet.
In the paper
“Indeed, the instructions provided with the test remind the test-taker that even ‘if the cases appear familiar to you, do not assume that they are precisely the same as you have read before. Read them thoroughly, as if they all were new to you.’”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In total, our analysis highlights that GPT-4 has indeed passed the Bar and has done so by a significant margin.”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Most jurisdictions use a six-point scale, and a score of four or higher is generally considered passing.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It should be noted that several of these topics where GPT-4 struggles are also areas where law students and bar examinees would also likely struggle.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The Rule Against Perpetuities is considered by many to be among the most difficult issues in all of law.
The paper relies on this without support.
Bar exam performance is a valid indicator of a model's potential to support the delivery of legal services in society.
“but also the potential for such models to support the delivery of legal services in society.”
“we sought an exemplary challenge to demonstrate this potential to both the legal domain and general scientific community.”
The paper relies on this without support.
The NCBE BarNow average performance by topic is comparable to the performance of human test-takers on the specific MBE exam used in this study.
“Table 3 offers both the NCBE Bar Now average accuracy by question category as well as the overall approximate national average MBE performance for recent test-takers.”
“it outperforms the average NCBE BarNow test-taker in five out of seven categories (Civil Procedure, Contracts, Criminal Law and Procedure, Evidence and Real Property).”
The paper relies on this without support.
The July 2022 MEE and MPT questions are representative of the UBE for the purpose of evaluating GPT-4's ability to pass the bar.
“For the MEE and the MPT, we collected the most recently released questions from the July 2022 Bar Examination.”
“GPT-4 can ‘pass the Bar’ in all UBE jurisdictions.”
The paper relies on this without support.
Most jurisdictions use a six-point scale, and a score of four or higher is generally considered passing.
“Most jurisdictions leverage the six-point scale, where a score of four or higher is generally considered passing.”
The paper relies on this without support.
The Rule Against Perpetuities is considered by many to be among the most difficult issues in all of law.
“In particular, the Rule Against Perpetuities is considered by many to be among the most difficult issues in all of law.”
The paper relies on this without support.
Most real-life examinees who otherwise pass the Bar Exam are unable to complete an end-to-end MEE that is free from errors.
“In addition, it should be mentioned that most real life examinees who otherwise pass the Bar Exam are unable to complete an end-to-end MEE that is free from errors.”
The paper relies on this without support.
Prior models were unable to handle longer documents in zero-shot tasks due to their token limits.
“As many have noted, prior models were unable to handle longer documents in zero-shot tasks due to their token limits.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“GPT has been lauded for its ability to exhibit human-level performance on a variety of tasks (e.g., passing the Bar Exam or acing the SAT test), and better performance than existing LLMs (47, 48).”
From GPT is an effective tool for multilingual psychological text analysis (2024) · cites this paper as [47]
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Advances in neural information processing systems 7
Cios, K. J., & Shields, M. E. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,093
A logical calculus of the ideas immanent in nervous activity
McCulloch, W. S., & Pitts, W. (1943) · The Bulletin of Mathematical Biophysics · doi:10.1007/bf02478259 · Cited by 18,566
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
A Survey of the Usages of Deep Learning for Natural Language Processing
Otter, D. W., Medina, J. R., & Kalita, J. K. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2979670 · Cited by 1,836
LEGAL-BERT: The Muppets straight out of Law School
Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Αλέτρας, Ν., & Androutsopoulos, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.261 · Cited by 1,453
Big Bird: Transformers for Longer Sequences
Zaheer, M., Guruganesh, G., Dubey, A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2007.14062 · Cited by 1,349
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., & Schmidt, D. C. (2023) · arXiv · doi:10.48550/arxiv.2302.11382 · Cited by 838
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778
ChatGPT Goes to Law SchoolPreprint
Choi, J. H., Hickman, K. E., Monahan, A., & Schwarcz, D. B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4335905 · Cited by 565
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2018) · arXiv · doi:10.48550/arxiv.1804.07461 · Cited by 562
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English
Chalkidis, I., Jana, A., Hartung, D., Bommarito, M. J., Androutsopoulos, I., Katz, D. M., & Αλέτρας, Ν. (2021) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.297 · Cited by 511
Self-citationHow Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence
Zhong, H., Xiao, C., Tu, C., Zhang, T., Liu, Z., & Sun, M. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.466 · Cited by 485
In-Context Retrieval-Augmented Language Models
Ram, O., Levine, Y., Dalmedigos, I., Muhlgay, D., Shashua, A., Leyton-Brown, K., & Shoham, Y. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00605 · Cited by 449
Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2023) · doi:10.2139/ssrn.4389233 · Cited by 422
Self-citationOnline Courts and the Future of Justice
Susskind, R. (2019) · doi:10.1093/oso/9780198838364.001.0001 · Cited by 418
The Language of Law School: Learning to "Think Like a Lawyer"
Mertz, E. (2007) · doi:10.1093/acprof:oso/9780195183108.001.0001 · Cited by 403
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345
Progress in Neural NLP: Modeling, Learning, and Reasoning
Zhou, M., Duan, N., Liu, S., & Shum, H. (2020) · Engineering · doi:10.1016/j.eng.2019.12.014 · Cited by 325
Lawformer: A pre-trained language model for Chinese legal long documents
Xiao, C., Hu, X., Liu, Z., Tu, C., & Sun, M. (2021) · AI Open · doi:10.1016/j.aiopen.2021.06.003 · Cited by 324
Dynabench: Rethinking Benchmarking in NLP
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.324 · Cited by 254
PromptChainer: Chaining Large Language Model Prompts through Visual Programming
Wu, T., Jiang, E., Donsbach, A., Gray, J., Molina, A., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519729 · Cited by 198
Bommarito, M. J., & Katz, D. M. (2022) · doi:10.2139/ssrn.4314839 · Cited by 181
Self-citationWhat do RNN Language Models Learn about Filler–Gap Dependencies?
Wilcox, E., Lévy, R., Morita, T., & Futrell, R. (2018) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/w18-5423 · Cited by 173
Large Language Models and the Reverse Turing Test
Sejnowski, T. J. (2023) · Neural Computation · doi:10.1162/neco_a_01563 · Cited by 169 · cited 1×
“Details of these prompts and parameters are available in the online repository.”
BackgroundNatural Language Processing in the Legal DomainPreprint
Hartung, D., Katz, D. M., Bommarito, M. J., Gerlach, L., Jana, A., & Soh, J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4336224 · Cited by 162
Self-citationLegalbench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
Guha, N., Nyarko, J., Ho, D. E., Ré, C., Chilton, A., Narayana, A., Chohlas-Wood, A., Peters, A., Waldon, B., Rockmore, D., et al. (2023) · doi:10.2139/ssrn.4583531 · Cited by 154
Self-citation(2021) · doi:10.1145/3462757.3466088 · Cited by 151
Recognizing textual entailment: Rational, evaluation and approaches – Erratum
Dagan, I., Dolan, W., Magnini, B., & Roth, D. (2010) · Natural Language Engineering · doi:10.1017/s1351324909990234 · Cited by 134
Word meaning in minds and machines.
Lake, B. M., & Murphy, G. L. (2023) · Psychological Review · doi:10.1037/rev0000297 · Cited by 131
Measuring the complexity of the law: the United States Code
Katz, D. M., & Bommarito, M. J. (2014) · Artificial Intelligence and Law · doi:10.1007/s10506-014-9160-8 · Cited by 117
Self-citationLexGLUE: A Benchmark Dataset for Legal Language Understanding in English
Chalkidis, I., Jana, A., Hartung, D., Bommarito, M. J., Androutsopoulos, I., Katz, D. M., & Aletras, N. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3936759 · Cited by 111
Self-citationComprehension of legal contracts by non‐experts: Effectiveness of plain language redrafting
Masson, M. E. J., & Waldron, M. A. (1994) · Applied Cognitive Psychology · doi:10.1002/acp.2350080107 · Cited by 110
Chalkidis, I., Fergadiotis, M., & Androutsopoulos, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.559 · Cited by 104
Law and Word Order: NLP in Legal Tech
Dale, R. (2018) · Natural Language Engineering · doi:10.1017/s1351324918000475 · Cited by 96
Re-evaluating GPT-4’s bar exam performance
Martínez, E. (2024) · Artificial Intelligence and Law · doi:10.1007/s10506-024-09396-9 · Cited by 96
Ruhl, J. B., Katz, D. M., & Bommarito, M. J. (2017) · Science · doi:10.1126/science.aag3013 · Cited by 94
Self-citationDissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2023) · arXiv · doi:10.48550/arxiv.2301.06627 · Cited by 80
Gpt as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA CapabilitiesPreprint
Bommarito, J., Bommarito, M. J., Katz, J. A. M., & Katz, D. M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4322372 · Cited by 68
Self-citationSyntaxGym: An Online Platform for Targeted Evaluation of Language Models
Gauthier, J., Hu, J., Wilcox, E., Qian, P., & Levy, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2020.acl-demos.10 · Cited by 61
Complex societies and the growth of the law
Katz, D. M., Coupette, C., Beckedorf, J., & Hartung, D. (2020) · Scientific Reports · doi:10.1038/s41598-020-73623-x · Cited by 55
Self-citationCoupette, C., Beckedorf, J., Hartung, D., Bommarito, M. J., & Katz, D. M. (2021) · Frontiers in Physics · doi:10.3389/fphy.2021.658463 · Cited by 51
Self-citationLEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain
Niklaus, J., Matoshi, V., Rani, P., Galassi, A., Stürmer, M., & Chalkidis, I. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.200 · Cited by 50
Ruhl, J. B. (2008) · bepress Legal Repository · Cited by 48
Improving Access to Justice in State Courts with Platform Technology
Prescott, J. J. (2017) · eYLS (Yale Law School) · Cited by 44
Poor writing, not specialized concepts, drives processing difficulty in legal language
Martínez, E., Mollica, F., & Gibson, E. (2022) · Cognition · doi:10.1016/j.cognition.2022.105070 · Cited by 39
Toward measures of complexity in legal systems
Bourcier, D., & Mazzega, P. (2007) · Proceedings of the 11th international conference on Artificial intelligence and law · doi:10.1145/1276318.1276359 · Cited by 32
Large Language Models as Corporate LobbyistsPreprint
Nay, J. (2023) · Social Science Research Network · doi:10.48550/arxiv.2301.01181 · Cited by 26
Talking Nets: An Oral History of Neural Networks
(2000) · doi:10.7551/mitpress/6626.001.0001 · Cited by 25
Re-evaluating GPT-4's bar exam performancePreprint
Martínez, E. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4441311 · Cited by 22
Complexity and Entropy in Legal Language
Friedrich, R. (2021) · Frontiers in Physics · doi:10.3389/fphy.2021.671882 · Cited by 19
Large Language Models as Corporate LobbyistsPreprint
Nay, J. J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4316615 · Cited by 16
Barton, B. H. (2003) · bepress Legal Repository · Cited by 16
Even lawyers do not like legalese
Martínez, E., Mollica, F., & Gibson, E. A. F. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2302672120 · Cited by 15
GPT as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA CapabilitiesPreprint
Bommarito, J., Bommarito, M. J., Katz, D. M., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.04408 · Cited by 15
Self-citationAll the Wild Possibilities: Technology That Attacks Barriers to Access to Justice
Staudt, R. W. (2009) · bepress Legal Repository · Cited by 14
Measuring and Modeling the U.S. Regulatory Ecosystem
Bommarito II, M. J., & Katz, D. M. (2017) · Journal of Statistical Physics · doi:10.1007/s10955-017-1846-3 · Cited by 13
Self-citationIn-Context Retrieval-Augmented Language ModelsPreprint
Ram, O., Levine, Y., Dalmedigos, I., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.00083 · Cited by 12
Hadfield, G. K. (2009) · FLASH - Fordham Law Archive of Scholarship & History (Fordham University) · Cited by 9
Using Recognizing Textual Entailment as a Core Engine for Answer Validation
Wang, R., & Neumann, G. (2008) · Lecture notes in computer science · doi:10.1007/978-3-540-85760-0_50 · Cited by 7
Howarth, J. W. (2022) · Stanford University Press eBooks · doi:10.1515/9781503633698 · Cited by 7
Natural Language Processing in the Legal DomainPreprint
Hartung, D., Katz, D. M., Bommarito, M. J., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.12039 · Cited by 7
Self-citationAssessing America’s Access to Civil Justice Crisis
Sandefur, R. L., & Teufel, J. A. (2021) · bepress Legal Repository · Cited by 6
Transformer Based Language Models for Similar Text Retrieval and RankingPreprint
Qadrud-Din, J., Rabiou, A. B., Walker, R., Soni, R., Gajek, M., Pack, G., & Rangaraj, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.04588 · Cited by 6
Caplan, L. (2019) · Daedalus · doi:10.1162/daed_a_00531 · Cited by 4
Dagan, I., Roth, D., Sammons, M., & Zanzotto, F. M. (2013) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-02151-0_1 · Cited by 4
Griggs, M. (2019) · doi:10.37419/lr.v7.i1.1 · Cited by 3
Retrieval-Generation Synergy Augmented Large Language ModelsPreprint
Feng, Z., Feng, X., Zhao, D., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.05149 · Cited by 1
The Pardy-Ruhl Dialogue on Ecosystem Management, Part IV: Narrowing and Sharpening the Questions
Ruhl, J. B. (2007) · Pace Environmental Law Review · doi:10.58948/0738-6206.1052
Supplementary material from "GPT-4 passes the Bar Exam"
Martin Katz, D., James Bommarito, M., Gao, S., et al. (2024) · Figshare · doi:10.6084/m9.figshare.c.7031287
Self-citationBLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Workshop, B., :, Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.05100
Research works printed but not linked: 14
- Tiersma PM. 1999 Legal language. Chicago, IL: University of Chicago Press.
- Rhode DL. 2004 Access to justice. Oxford, UK: Oxford University Press.
- Von Neumann J. 1958 The computer and the brain. New Haven, CT: Yale University Press.
- Vaswani A Shazeer N Parmar N Uszkoreit J Jones L Gomez AN Kaiser L Polosukhin I. 2017 Attention is all you need. In Advances in neural information processing systems vol. 30.
- Christiano PF Leike J Brown T Martic M Legg S Amodei D. 2017 Deep reinforcement learning from human preferences. Advances in neural information processing systems vol. 30.
- Devlin J Chang MW Lee K Toutanova K. 2018 BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019 pp. 4171–4186.
- Kenton JDMWC Toutanova LK. 2019 BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies pp. 4171–4186.
- Brown T Mann B Ryder N Subbiah M Kaplan JD Dhariwal P Neelakantan A Shyam P Sastry G Askell A Agarwal S. 2020 Language models are few-shot learners. In Advances in neural information processing systems pp. 1877–1901.
- Barrett D Hill F Santoro A Morcos A Lillicrap T. 2018 Measuring abstract reasoning in neural networks. In International conference on machine learning pp. 511–520. PMLR.
- Wang A Pruksachatkun Y Nangia N Singh A Michael J Hill F Levy O Bowman S. 2019 Superglue: a stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems 32.
- Mitchell M Krakauer DC. 2022 The Debate Over Understanding in AI’s Large Language Models. (http://arxiv.org/abs/2210.13966) · arXiv
- Fawei B Wyner A Pan J. 2016 Passing a USA national bar exam: a first corpus for experimentation. In LREC 2016 Tenth International Conference on Language Resources and Evaluation pp. 23–30.
- Wang Z Panda R Karlinsky L Feris R Sun H Kim Y. 2023 Multitask prompt tuning enables parameter-efficient transfer learning. In The Eleventh International Conference on Learning Representations.
- Wei J et al. 2022 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems 35 24 824-24 837.
Cited by
Sorted by citations
Grassini, S. (2023) · Education Sciences · doi:10.3390/educsci13070692 · Cited by 1,048
Yenduri, G., Ramalingam, M., Selvi, G. C., Supriya, Y., Srivastava, G., Maddikunta, P. K. R., Raj, G. D., Jhaveri, R. H., Prabadevi, B., Wang, W., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3389497 · Cited by 916
The promise and challenges of generative AI in education
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2394886 · Cited by 543
Bengesi, S., El-Sayed, H., Sarker, M. K., Houkpati, Y., Irungu, J., & Oladunni, T. (2024) · IEEE Access · doi:10.1109/access.2024.3397775 · Cited by 470
Metadata
- Authors
- Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P.
- Affiliations
- Illinois Institute of Technology (US)
- Casetext, Inc., USA
- Stanford University (US)
- Corresponding author
- Katz, Daniel Martin
- Venue
- Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
- Publisher
- The Royal Society
- Published
- 2024-02-26
- Volume
- 382
- Issue
- 2270
- Pages
- 20230254-20230254
- ISSN
- 1364-503X (ISSN-L)
- 1471-2962
- PMC
- PMC10894685
- OpenAlex
- W4392773006
- Semantic Scholar
- 0e8297549b4ec852ce8fd55dee9ae21501805af2
- Cited by
- 420
- References
- 63
- Citations by year
- 2026: 91 · 2025: 116 · 2024: 127
- FWCI
- 532.03
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences (publishedVersion, cc-by)
- SSRN Electronic Journal (acceptedVersion)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Law › Artificial Intelligence in Law
- Social Sciences › Law › Legal Education and Practice Innovations
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- GPT-4, bar examination, zero-shot performance, large language models, ChatGPT, legal education, legal services
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
