The DOI matches its Crossref record; 78 of the 79 references it cites were found in a registry or an index, and 1 are not checked yet; none of the 18 sources its claims cite is checked for support yet.
GPT-4 passes the bar exam
Open accessAbstract
In this paper, we experimentally evaluate the zero-shot performance of GPT-4 against prior generations of GPT on the entire uniform bar examination (UBE), including not only the multiple-choice multistate bar examination (MBE), but also the open-ended multistate essay exam (MEE) and multistate performance test (MPT) components. On the MBE, GPT-4 significantly outperforms both human test-takers and prior models, demonstrating a 26% increase over ChatGPT and beating humans in five of seven subject areas. On the MEE and MPT, which have not previously been evaluated by scholars, GPT-4 scores an average of 4.2/6.0 when compared with much lower scores for ChatGPT. Graded across the UBE components, in the manner in which a human test-taker would be, GPT-4 scores approximately 297 points, significantly in excess of the passing threshold for all UBE jurisdictions. These findings document not just the rapid and remarkable advance of large language model performance generally, but also the potential for such models to support the delivery of legal services in society. This article is part of the theme issue ‘A complexity science approach to law and governance’.
Verifying this paper
8 of 9 steps done · this page updates itself — you can leave and come back
- Registry recordVerified
- Full textRead the full text
- Reference list63 references found
- Reference checkChecking each cited work — 78 of 79
- StatisticsRecomputed
- Claims86 claims anchored to a quote
- Cited sourcesChecked
- Self-checkChecked
- VerificationFindings verified
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
GPT-4 achieved zero-shot passing performance on the entire Uniform Bar Exam, including the MBE, MEE, and MPT, with an overall UBE score of approximately 297 points. The authors argue this demonstrates rapid LLM progress and potential to support legal services delivery.
Key findings
- On the MBE, GPT-4 scored 75.7% accuracy, a 26.5% increase over ChatGPT, beating the average human test-taker by more than 7% and outperforming humans in five of seven subject areas.
- On the MEE, GPT-4 averaged 4.2/6.0 versus ChatGPT’s 3.0/6.0; on the MPT, GPT-4 averaged 4.2/6.0 versus ChatGPT’s 2.8/6.0.
- Combined UBE score was 297 points for GPT-4 and 213 points for ChatGPT, exceeding passing thresholds ranging from 260 to 273 points across UBE jurisdictions.
- GPT-4 was Pareto dominant over prior GPT models on the MBE, with largest gains in Contracts and Evidence, though Civil Procedure was its weakest subject.
Methods
- The authors evaluated zero-shot performance on the full UBE: official MBE practice questions and the most recently released July 2022 MEE and MPT questions.
- MBE grading was automated, with three runs per prompt/parameter set and average scores reported; OpenAI ran a contamination check on the test questions.
- MEE and MPT responses were graded by two authors—a tenured law professor and a multi-jurisdiction licensed attorney—using conservative scoring against state bar representative “good” answers, with additional peer review of selected samples.
Limitations
- The authors state they could not replicate the NCBE’s full-scale grading process, and MEE/MPT scoring involved subjective qualitative assessment despite conservative standards and peer checks.
- The paper states GPT-4 still made errors, including on a testamentary trust distribution, a civil procedure diversity jurisdiction question, and a real property future interest/Rule Against Perpetuities subquestion.
- The text makes evident that MEE/MPT evaluation used only the July 2022 exam, the MBE questions could not be released, and results reflect minimal zero-shot prompting rather than more exhaustive prompt engineering or few-shot techniques.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
46 Own result · 12 Own interpretation · 10 Own method · 9 Not checked yet · 9 No basis given
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| GPT-4 | 75.7% |”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it meets or exceeds the approximate passing threshold in all seven subject matter areas.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plainn = 7- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Civil Procedure is both the worst subject for GPT-4, ChatGPT and human test-takers.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Contracts and Evidence are the topics with the largest overall improvement.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“First, it has difficulty calculating the distribution of assets from a testamentary trust which has been deemed to be invalid.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 scores an average of 4.2/6.0 when compared with much lower scores for ChatGPT.”
§ GPT-4 passes the bar exam > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“a more than 35% raw increase in Evidence.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 is now Pareto dominant over all prior versions of GPT”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the MBE, GPT-4 significantly outperforms both human test-takers and prior models”
§ GPT-4 passes the bar exam > Abstract
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“On the MBE, GPT-4 significantly outperforms both human test-takers and prior models”
§ GPT-4 passes the bar exam > Abstract
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 24
- Probably not a substantial claim. 24
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-3.0 can vaguely recite some of the relevant principles and rules but does not properly connect those principles to the facts and consistently reaches improper legal conclusions.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT fails to properly identify all four prongs of Rule 702 of the Federal Rules of Evidence (FRE). This results in a failure to discuss Rule 702(a).”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4, by contrast, correctly discusses both FRE Rule 403 and Rule 404(b) and does not devote attention to extraneous issues.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“consider the second subquestion on the MEE Evidence problem (figures 7 and 10) where ChatGPT fails to address a relevant rule (FRE 403) and instead focuses significant attention on non-relevant rules (FRE 701 and 702).”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the average student answers more than three in ten questions incorrectly.”
§ GPT-4 passes the bar exam > Results > Multistate bar exam results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These type of patterns repeat themselves over the remainder of both MPT-1 and MPT-2.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The July 2022 MPT featured two questions, each over the publicly available ChatGPT 4096 token limit—5297 tokens for MPT-1 and 5188 tokens for MPT-2.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Subquestion three of MPT-1, as displayed in figures 16 and 20, provides another example of the distinction in quality between GPT-4 and ChatGPT.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, in the first subquestion of MPT-1, both GPT-4 and ChatGPT produce a lengthy and otherwise reasonable looking answer.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| MPT 1 - Hixon Marriages | 4.2/6.0 | 3.0/6.0 |”
- Also stated in: § Multistate performance test results · plain
“| MPT 2 - In re Briotti | 4.1/6.0 | 2.5/6.0 |”
- Also stated in: § Multistate performance test results · plain
“| overall score | 4.2 | 2.8 |”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Multistate performance test results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 obtains an overall UBE score of 297 points4 while ChatGPT obtains a score of 213 points.5”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In figure 14, GPT-4 not only correctly identifies that Columbia Law is controlling but provides reasonable arguments as to why Columbia should be selected under the ‘significant relationship’ test.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In figure 18, ChatGPT correctly identifies that it should address the Restatement of the Conflicts of Law as this is core to the answer. However, it incorrectly cites §6 as the proper source when it should cite §283 as the source of the ‘significant relationship’ test.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT fails to distinguish between the circumstances and assigns authority to the Franklin Courts that they do not possess.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both of the models we considered produced long-form answers to the respective problems, but there are some fairly straightforward differences in the quality of output produced.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As displayed in table 8, different jurisdictions impose different UBE passing score thresholds ranging from 260 (for states such as Alabama and Minnesota) to 273 points (for Arizona).”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although GPT-4 obtains within-category passing-level scores for both the MEE and MPT”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Next, it fails to grasp the call of the question and provides an incorrect answer on a civil procedure question regarding diversity jurisdiction after the joinder of a necessary party.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both GPT-4 and, to a lesser extent, ChatGPT were able to largely avoid the trap of citing legal principles, cases, or other materials which would otherwise appear to be on point but would not be responsive to the requirements of the MPT.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, as presented in table 5, our final MEE grade is 4.2 out of 6 points for GPT-4 and 3 out of 6 points for ChatGPT.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The complexity of the law [12–15] imposes real consequences for many individuals and organizations [16,17].”
§ GPT-4 passes the bar exam > Introduction
What the sources say
- Not checked yetcites [12] Bourcier D, Mazzega P. 2007 Toward measures of complexity in legal systems. In Proceedings of the 11th international conference on Artificial intelligence and law, pp. 211–215. source
- Not checked yetcites [13] Ruhl J, Katz D, Bommarito M. 2017 Harnessing legal complexity. Science 355, 1377- 1378. (10.1126/science.aag3013) 28360284 source
- Not checked yetcites [14] Bommarito II M, Katz D. 2017 Measuring and modeling the US regulatory ecosystem. J. Stat. Phys. 168, 1125- 1135. (10.1007/s10955-017-1846-3) source
- Not checked yetcites [15] Katz DM, Coupette C, Beckedorf J, Hartung D. 2020 Complex societies and the growth of the law. Sci. Rep. 10, 1- 14. (10.1038/s41598-020-73623-x) 33127960 PMC7599339 source
- Not checked yetcites [16] Staudt RW. 2008 All the wild possibilities: technology that attacks barriers to access to justice. Loy. LAL Rev. 42, 1117.
- Not checked yetcites [17] Sandefur RL, Teufel J. 2020 Assessing America’s Access to Civil Justice Crisis. UC Irvine L. Rev. 11, 753.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
In the paper
“the rate of performance increase from text-davinci-001 to text-davinci-003 strongly suggested that passing performance could ‘occur within the next 0–18 months’ [8].”
§ GPT-4 passes the bar exam > Introduction
What the sources say
- Not checked yetcites [8] Bommarito MJ, Katz DM. 2022 GPT Takes the Bar Exam. Available at SSRN 4314839. source
boostedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Two of this study’s authors, a tenured law professor and an attorney licensed in multiple jurisdictions, reviewed the model output and collaboratively assigned scores to each of the MEE questions.”
- Also stated in: § Multistate essay examination results · plain
“we debated each score with the view that we should be somewhat conservative in the assignment of MEE scores.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Multistate essay examination results · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“All grading is automated.”
§ GPT-4 passes the bar exam > Data and methods > Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“the ability to interpret such complex documents often requires years of study.”
§ GPT-4 passes the bar exam > Introduction
What the sources say
- Not checked yetcites [6] Mertz E. 2007 The language of law school: learning to ‘think like a lawyer’. USA: Oxford University Press. source
- Not checked yetcites [7] Martínez E, Mollica F, Gibson E. 2022 Poor writing, not specialized concepts, drives processing difficulty in legal language. Cognition 224, 105070. (10.1016/j.cognition.2022.105070) 35257980 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“legal systems have struggled to assist with the quantity, quality, and accessibility of legal services demanded by society [17–19].”
§ GPT-4 passes the bar exam > Introduction
What the sources say
- Not checked yetcites [17] Sandefur RL, Teufel J. 2020 Assessing America’s Access to Civil Justice Crisis. UC Irvine L. Rev. 11, 753.
- Not checked yetcites [18] Rhode DL. 2004 Access to justice. Oxford, UK: Oxford University Press.
- Not checked yetcites [19] Susskind RE. 2019 Online courts and the future of justice. Oxford, UK: Oxford University Press. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“a combined score of 266 points is enough to pass in jurisdictions such as Illinois, New York and the District of Columbia, while a score of 270 points would pass in the vast majority of states which use the UBE.”
§ GPT-4 passes the bar exam > The Uniform Bar Exam > Description of the Uniform Bar Exam
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Experiments involved three runs for each set of prompts and parameters; we report the average score across all runs in our results.”
§ GPT-4 passes the bar exam > Data and methods > Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“but also the potential for such models to support the delivery of legal services in society.”
§ GPT-4 passes the bar exam > Abstract
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: Bar exam performance is a valid indicator of a model's potential to support the delivery of legal services in society.
None of the sources cited for this claim have been checked yet.
In the paper
“Legal language is notoriously complex [3–5]”
§ GPT-4 passes the bar exam > Introduction
What the sources say
- Not checked yetcites [3] Friedrich R. 2021 Complexity and entropy in legal language. Front. Phys. 9, 671882. (10.3389/fphy.2021.671882) source
- Not checked yetcites [4] Katz DM, Bommarito MJ. 2014 Measuring the complexity of the law: the United States Code. Artif. Intell. Law 22, 337- 374. (10.1007/s10506-014-9160-8) source
- Not checked yetcites [5] Ruhl JB. 2007 Law’s complexity: a primer. Ga. St. UL Rev. 24, 885. (10.58948/0738-6206.1052)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“These representative good answers are made available by state bar associations and reflect actual MEE and MPT answers produced by real examinees.”
§ GPT-4 passes the bar exam > Data and methods > Data
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“While the exact sequence of questions administered is not identical to any actual exam as administered”
§ GPT-4 passes the bar exam > Data and methods > Data
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Franklin Court cannot, without personal jurisdiction over Tucker, take action against a property outside its borders.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“The UBE has three components (MBE, MEE and MPT) which are typically, but not always, weighed using the approach highlighted in table 1.”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4, like prior models, may still hallucinate sources, incorrectly interpret facts or fail to follow ethical requirements”
§ GPT-4 passes the bar exam > Conclusion
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
In the paper
“As the NCBE describes it, ‘the MPT is not a test of substantive law; the Library materials provide sufficient substantive information to complete the task.’”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
What the sources say
- Not checked yetcites NCBE
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Consequently, the MPT requires a suspension of knowledge, whereby the examinee must, for the period of the test, imagine themselves in a jurisdiction that may contradict their actual knowledge of real law.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As we noted in the grading of the MEE, we recognize that there is a subjective aspect to this sort of analysis”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“its high MBE percentile provided it with substantial latitude; GPT-4 would likely pass even with a much lower MEE or MPT score in some or all jurisdictions.”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“However, thanks to assistance from OpenAI, we were able to use an ‘8K’ version of ChatGPT that has a wider context window of 8193 tokens, and could thus accommodate the length of the overall materials.”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Prior models were unable to handle longer documents in zero-shot tasks due to their token limits.
None of the sources cited for this claim have been checked yet.
In the paper
“Indeed, the instructions provided with the test remind the test-taker that even ‘if the cases appear familiar to you, do not assume that they are precisely the same as you have read before. Read them thoroughly, as if they all were new to you.’”
§ GPT-4 passes the bar exam > Results > Multistate performance test results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In total, our analysis highlights that GPT-4 has indeed passed the Bar and has done so by a significant margin.”
§ GPT-4 passes the bar exam > Results > Combined results and comparison to the UBE passing threshold
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Most jurisdictions use a six-point scale, and a score of four or higher is generally considered passing.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It should be noted that several of these topics where GPT-4 struggles are also areas where law students and bar examinees would also likely struggle.”
§ GPT-4 passes the bar exam > Results > Multistate essay examination results
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The Rule Against Perpetuities is considered by many to be among the most difficult issues in all of law.
The paper relies on this without support.
Bar exam performance is a valid indicator of a model's potential to support the delivery of legal services in society.
“but also the potential for such models to support the delivery of legal services in society.”
“we sought an exemplary challenge to demonstrate this potential to both the legal domain and general scientific community.”
The paper relies on this without support.
The NCBE BarNow average performance by topic is comparable to the performance of human test-takers on the specific MBE exam used in this study.
“Table 3 offers both the NCBE Bar Now average accuracy by question category as well as the overall approximate national average MBE performance for recent test-takers.”
“it outperforms the average NCBE BarNow test-taker in five out of seven categories (Civil Procedure, Contracts, Criminal Law and Procedure, Evidence and Real Property).”
The paper relies on this without support.
The July 2022 MEE and MPT questions are representative of the UBE for the purpose of evaluating GPT-4's ability to pass the bar.
“For the MEE and the MPT, we collected the most recently released questions from the July 2022 Bar Examination.”
“GPT-4 can ‘pass the Bar’ in all UBE jurisdictions.”
The paper relies on this without support.
Most jurisdictions use a six-point scale, and a score of four or higher is generally considered passing.
“Most jurisdictions leverage the six-point scale, where a score of four or higher is generally considered passing.”
The paper relies on this without support.
The Rule Against Perpetuities is considered by many to be among the most difficult issues in all of law.
“In particular, the Rule Against Perpetuities is considered by many to be among the most difficult issues in all of law.”
The paper relies on this without support.
Most real-life examinees who otherwise pass the Bar Exam are unable to complete an end-to-end MEE that is free from errors.
“In addition, it should be mentioned that most real life examinees who otherwise pass the Bar Exam are unable to complete an end-to-end MEE that is free from errors.”
The paper relies on this without support.
Prior models were unable to handle longer documents in zero-shot tasks due to their token limits.
“As many have noted, prior models were unable to handle longer documents in zero-shot tasks due to their token limits.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“GPT has been lauded for its ability to exhibit human-level performance on a variety of tasks (e.g., passing the Bar Exam or acing the SAT test), and better performance than existing LLMs (47, 48).”
From GPT is an effective tool for multilingual psychological text analysis (2024) · cites this paper as [47]
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892
Advances in neural information processing systems 7
Cios, K. J., & Shields, M. E. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,093
A logical calculus of the ideas immanent in nervous activity
McCulloch, W. S., & Pitts, W. (1943) · The Bulletin of Mathematical Biophysics · doi:10.1007/bf02478259 · Cited by 18,566
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
A Survey of the Usages of Deep Learning for Natural Language Processing
Otter, D. W., Medina, J. R., & Kalita, J. K. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2020.2979670 · Cited by 1,836
Research works printed but not linked: 13
- Tiersma PM. 1999 Legal language. Chicago, IL: University of Chicago Press.
- Rhode DL. 2004 Access to justice. Oxford, UK: Oxford University Press.
- Von Neumann J. 1958 The computer and the brain. New Haven, CT: Yale University Press.
- Vaswani A Shazeer N Parmar N Uszkoreit J Jones L Gomez AN Kaiser L Polosukhin I. 2017 Attention is all you need. In Advances in neural information processing systems vol. 30.
- Christiano PF Leike J Brown T Martic M Legg S Amodei D. 2017 Deep reinforcement learning from human preferences. Advances in neural information processing systems vol. 30.
- Devlin J Chang MW Lee K Toutanova K. 2018 BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019 pp. 4171–4186.
- Kenton JDMWC Toutanova LK. 2019 BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies pp. 4171–4186.
- Brown T Mann B Ryder N Subbiah M Kaplan JD Dhariwal P Neelakantan A Shyam P Sastry G Askell A Agarwal S. 2020 Language models are few-shot learners. In Advances in neural information processing systems pp. 1877–1901.
- Barrett D Hill F Santoro A Morcos A Lillicrap T. 2018 Measuring abstract reasoning in neural networks. In International conference on machine learning pp. 511–520. PMLR.
- Wang A Pruksachatkun Y Nangia N Singh A Michael J Hill F Levy O Bowman S. 2019 Superglue: a stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems 32.
- Fawei B Wyner A Pan J. 2016 Passing a USA national bar exam: a first corpus for experimentation. In LREC 2016 Tenth International Conference on Language Resources and Evaluation pp. 23–30.
- Wang Z Panda R Karlinsky L Feris R Sun H Kim Y. 2023 Multitask prompt tuning enables parameter-efficient transfer learning. In The Eleventh International Conference on Learning Representations.
- Wei J et al. 2022 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems 35 24 824-24 837.
Cited by
Sorted by citations
Grassini, S. (2023) · Education Sciences · doi:10.3390/educsci13070692 · Cited by 1,048
Yenduri, G., Ramalingam, M., Selvi, G. C., Supriya, Y., Srivastava, G., Maddikunta, P. K. R., Raj, G. D., Jhaveri, R. H., Prabadevi, B., Wang, W., et al. (2024) · IEEE Access · doi:10.1109/access.2024.3389497 · Cited by 916
The promise and challenges of generative AI in education
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2394886 · Cited by 543
Bengesi, S., El-Sayed, H., Sarker, M. K., Houkpati, Y., Irungu, J., & Oladunni, T. (2024) · IEEE Access · doi:10.1109/access.2024.3397775 · Cited by 470
GPT is an effective tool for multilingual psychological text analysis
Rathje, S., Mirea, D.-M., Sucholutsky, I., Marjieh, R., Robertson, C. E., & Van Bavel, J. J. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2308950121 · Cited by 297
Wamba, S. F., Guthrie, C. H., de Queiroz, M. M., & Minner, S. (2023) · International Journal of Production Research · doi:10.1080/00207543.2023.2294116 · Cited by 357
Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024) · Journal of Legal Analysis · doi:10.1093/jla/laae003 · Cited by 559
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516
“As a result, when confronted with problems necessitating domain-specific knowledge, such as medical (Li et al., 2023g; Singhal et al., 2023) and legal (Yu et al., 2022; Katz et al., 2023) questions, these models may exhibit pronounced hallucinations, often manifesting as factual fabrication.”
Massey, P. A., Montgomery, C., & Zhang, A. S. (2023) · Journal of the American Academy of Orthopaedic Surgeons · doi:10.5435/jaaos-d-23-00396 · Cited by 198
The now and future of ChatGPT and GPT in psychiatry
Cheng, S., Chang, C., Chang, W., Wang, H., Liang, C., Kishimoto, T., Chang, J. P., Kuo, J. S., & Su, K. (2023) · Psychiatry and Clinical Neurosciences · doi:10.1111/pcn.13588 · Cited by 216
Huang, X., Ruan, W., Huang, W., Jin, G., Dong, Y., Wu, C., Bensalem, S., Mu, R., Yi, Q., Zhao, X., et al. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10824-0 · Cited by 228
Explainable Generative AI (GenXAI): a survey, conceptualization, and research agenda
Schneider, J. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10916-x · Cited by 214
Shahriar, S., Lund, B. D., Mannuru, N. R., Arshad, M., Hayawi, K., Bevara, R. V. K., Mannuru, A., & Batool, L. (2024) · Applied Sciences · doi:10.3390/app14177782 · Cited by 211
Fields, J., Chovanec, K., & Madiraju, P. (2024) · IEEE Access · doi:10.1109/access.2024.3349952 · Cited by 188
Evaluating ChatGPT Performance on the Orthopaedic In-Training Examination
Kung, J. E., Marshall, C., Gauthier, C., Gonzalez, T. A., & Jackson, J. B. (2023) · JBJS Open Access · doi:10.2106/jbjs.oa.23.00056 · Cited by 137
Using ChatGPT for human–computer interaction research: a primer
Tabone, W., & de Winter, J. (2023) · Royal Society Open Science · doi:10.1098/rsos.231053 · Cited by 140
Can ChatGPT Pass High School Exams on English Language Comprehension?
de Winter, J. C. F. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-023-00372-z · Cited by 115
Mihalache, A., Huang, R. S., Popovic, M. M., & Muni, R. H. (2023) · Medical Teacher · doi:10.1080/0142159x.2023.2249588 · Cited by 113
Comparative Evaluation of LLMs in Clinical Oncology
Rydzewski, N. R., Dinakaran, D., Zhao, S. G., Ruppin, E., Türkbey, B., Citrin, D. E., & Patel, K. R. (2024) · NEJM AI · doi:10.1056/aioa2300151 · Cited by 116
Advances of machine learning in materials science: Ideas and techniques
Chong, S. S., Ng, Y. S., Wang, H., & Zheng, J. (2023) · Frontiers of Physics · doi:10.1007/s11467-023-1325-z · Cited by 114
Twenty‐first century technologies and language education: Charting a path forward
Kern, R. (2024) · Modern Language Journal · doi:10.1111/modl.12924 · Cited by 124
Generating meaning: active inference and the scope and limits of passive AI
Pezzulo, G., Parr, T., Cisek, P., Clark, A., & Friston, K. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2023.10.002 · Cited by 80
How should the advancement of large language models affect the practice of science?
Binz, M., Alaniz, S., Roskies, A., Aczel, B., Bergstrom, C. T., Allen, C., Schad, D., Wulff, D., West, J. D., Zhang, Q., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2401227121 · Cited by 74
Šavelka, J., & Ashley, K. D. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1279794 · Cited by 92
Wandelt, S., Zheng, C., Wang, S., Liu, Y., & Sun, X. (2024) · Applied Sciences · doi:10.3390/app14177455 · Cited by 86
GPT-aided diagnosis on agricultural image based on a new light YOLOPC
Qing, J., Deng, X., Lan, Y., & Li, Z. (2023) · Computers and Electronics in Agriculture · doi:10.1016/j.compag.2023.108168 · Cited by 70
Duah, J. E., & McGivern, P. (2024) · The International Journal of Information and Learning Technology · doi:10.1108/ijilt-11-2023-0213 · Cited by 67
The impact of generative AI on information processing: Evidence from the ban of ChatGPT in Italy
Bertomeu, J., Lin, Y., Liu, Y., & Ni, Z. (2025) · Journal of Accounting and Economics · doi:10.1016/j.jacceco.2025.101782 · Cited by 89
Creative Learning for Sustainability in a World of AI: Action, Mindset, Values
Henriksen, D., Mishra, P., & Stern, R. (2024) · Sustainability · doi:10.3390/su16114451 · Cited by 74
Conceptualizing generative AI as style engines: Application archetypes and implications
Riemer, K., & Peter, S. (2024) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2024.102824 · Cited by 63
Visual cognition in multimodal large language models
Schulze Buschoff, L. M., Akata, E., Bethge, M., & Schulz, E. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00963-y · Cited by 83
Greco, C. M., & Tagarelli, A. (2023) · Artificial Intelligence and Law · doi:10.1007/s10506-023-09374-7 · Cited by 87
From concept to manufacturing: evaluating vision-language models for engineering design
Picard, C., Edwards, K. M., Doris, A. C., Man, B., Giannone, G., Alam, M. F., & Ahmed, F. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11290-y · Cited by 57
Re-evaluating GPT-4’s bar exam performance
Martínez, E. (2024) · Artificial Intelligence and Law · doi:10.1007/s10506-024-09396-9 · Cited by 96
Fact-checking information from large language models can decrease headline discernment
DeVerna, M., Yan, H. Y., Yang, K.-C., & Menczer, F. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2322823121 · Cited by 107
Khademi, A. (2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.28 · Cited by 72
Rizzo, M. G., Cai, N., & Constantinescu, D. S. (2023) · Journal of Orthopaedics · doi:10.1016/j.jor.2023.11.056 · Cited by 54
The model student: GPT-4 performance on graduate biomedical science exams
Stribling, D., Xia, Y., Amer, M. K., Graim, K. S., Mulligan, C. J., & Renne, R. (2024) · Scientific Reports · doi:10.1038/s41598-024-55568-7 · Cited by 60
Potential Benefits and Risks of Artificial Intelligence in Education
Özer, M. (2024) · Bartın University Journal of Faculty of Education · doi:10.14686/buefad.1416087 · Cited by 72
Can you spot the bot? Identifying AI-generated writing in college essays
Waltzer, T., Pilegard, C., & Heyman, G. D. (2024) · International Journal for Educational Integrity · doi:10.1007/s40979-024-00158-3 · Cited by 49
Experiential Narratives in Marketing: A Comparison of Generative AI and Human Content
Wen, Y., & Laporte, S. (2024) · Journal of Public Policy & Marketing · doi:10.1177/07439156241297973 · Cited by 62
How to train your stochastic parrot: large language models for political texts
Ornstein, J. T., Blasingame, E., & Truscott, J. S. (2025) · Political Science Research and Methods · doi:10.1017/psrm.2024.64 · Cited by 51
Large language models: an overview of foundational architectures, recent trends, and a new taxonomy
Mienye, I. D., Jere, N., Obaido, G., Ogunruku, O. O., Esenogho, E., & Modisane, C. (2025) · Discover Applied Sciences · doi:10.1007/s42452-025-07668-w · Cited by 46
Enhancing Smart-Contract Security through Machine Learning: A Survey of Approaches and Techniques
Jiang, F., Chao, K., Xiao, J., Liu, Q., Gu, K., Wu, J., & Cao, Y. (2023) · Electronics · doi:10.3390/electronics12092046 · Cited by 49
Teaching and AI in the postdigital age: Learning from teachers’ perspectives
Moylan, R., Code, J., & O'Brien, H. L. (2024) · Teaching and Teacher Education · doi:10.1016/j.tate.2024.104851 · Cited by 60
Are chatbots the new relationship experts? Insights from three studies
Vowels, L. M. (2024) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2024.100077 · Cited by 42
Can large language models help augment English psycholinguistic datasets?
Trott, S. (2024) · Behavior Research Methods · doi:10.3758/s13428-024-02337-z · Cited by 58
Benchmarking Human–AI collaboration for common evidence appraisal tools
Woelfle, T., Hirt, J., Janiaud, P., Kappos, L., Ioannidis, J. P. A., & Hemkens, L. G. (2024) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2024.111533 · Cited by 46
ChatGPT: Literate or intelligent about UN sustainable development goals?
Raghu Raman, R., Lathabai, H. H., Mandal, S., Das, P., Kaur, T., & Nedungadi, P. (2024) · PLoS ONE · doi:10.1371/journal.pone.0297521 · Cited by 46
Garabet, R., Mackey, B. P., Cross, J., & Weingarten, M. S. (2023) · Medical Science Educator · doi:10.1007/s40670-023-01956-z · Cited by 45
Leveraging LLM respondents for item evaluation: A psychometric analysis
Liu, Y., Bhandari, S., & Pardos, Z. A. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13570 · Cited by 65
de Winter, J. (2024) · Scientometrics · doi:10.1007/s11192-024-04939-y · Cited by 34
Wong, L. P. W. (2024) · Merits · doi:10.3390/merits4040027 · Cited by 45
How do judges use large language models? Evidence from Shenzhen
Liu, J. Z., & Li, X. (2024) · The Journal of Legal Analysis · doi:10.1093/jla/laae009 · Cited by 40
Knoedler, L., Knoedler, S. A., Hoch, C. C., Prantl, L., Frank, K., Soiderer, L., Cotofana, S. C., Dorafshar, A. H., Schenck, T. L., Vollbach, F. H., et al. (2024) · Scientific Reports · doi:10.1038/s41598-024-63997-7 · Cited by 33
Large language models for automatic equation discovery of nonlinear dynamics
Du, M., Chen, Y., Wang, Z., Nie, L., & Zhang, D. (2024) · Physics of Fluids · doi:10.1063/5.0224297 · Cited by 56
What drives the corporate payoffs of using generative artificial intelligence?
Bughin, J. (2024) · Structural Change and Economic Dynamics · doi:10.1016/j.strueco.2024.09.011 · Cited by 35
Scalable Microscale Artificial Synapses of Lead Halide Perovskite with Femtojoule Energy Consumption
de Boer, J. J., & Ehrler, B. (2024) · ACS Energy Letters · doi:10.1021/acsenergylett.4c02360 · Cited by 29
García-Torres, D., Vicente Ripoll, M. A., Fernández Peris, C., & Mira Solves, J. J. (2024) · Healthcare · doi:10.3390/healthcare12222241 · Cited by 29
Wahidur, R. S. M., Kim, S., Choi, H., Bhatti, D. S., & Lee, H.-N. (2025) · IEEE Access · doi:10.1109/access.2025.3542125 · Cited by 40
WorldBench: Quantifying Geographic Disparities in LLM Factual Recall
Moayeri, M., Tabassi, E., & Feizi, S. (2024) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3630106.3658967 · Cited by 40
Strategic behavior of large language models and the role of game structure versus contextual framing
Lorè, N., & Heydari, B. (2024) · Scientific Reports · doi:10.1038/s41598-024-69032-z · Cited by 28
Performance analysis of large language models in the domain of legal argument mining
Zubaer, A. A., Granitzer, M., & Mitrović, J. (2023) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2023.1278796 · Cited by 37
On Cyber-Social Learning: A Critique of Artificial Intelligence in Education
Cope, B., & Kalantzis, M. (2024) · Postdigital science and education · doi:10.1007/978-3-031-64487-0_1 · Cited by 31
Large Language Models in Legal Systems: A Survey
Dehghani, F., Dehghani, R., Ardebili, Y. N., & Rahnamayan, S. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-05924-3 · Cited by 39
Wen, Y., Zhao, X., Li, X., & Zang, Y. (2025) · Sustainability · doi:10.3390/su17031127 · Cited by 30
Gadekallu, T. R., Yenduri, G., Kaluri, R., Rajput, D. S., Lakshmanna, K., Fang, K., Chen, J., & Wang, W. W. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2400 · Cited by 31
What will ChatGPT revolutionize in the financial industry?
Ali, H., & Aysan, A. F. (2023) · Modern Finance · doi:10.61351/mf.v1i1.67 · Cited by 71
Embracing Artificial Intelligence to Improve Self-Directed Learning: A Cybersecurity Classroom Study
Marquardson, J. (2024) · Information Systems Education Journal · doi:10.62273/wzby3952 · Cited by 33
AI and the Future of Skills, Volume 2
OECD (2023) · Educational research and innovation · doi:10.1787/a9fe53cb-en · Cited by 43
Beyond Generating Code: Evaluating GPT on a Data Visualization Course
Chen, Z., Zhang, C., Wang, Q., Troidl, J., Warchol, S., Beyer, J., Gehlenborg, N., & Pfister, H. (2023) · IEEE VIS Workshop on Visualization Education, Literacy, and Activities (EduVis) · doi:10.1109/eduvis60792.2023.00009 · Cited by 40
ChatGPT Knows Your Attacks: Synthesizing Attack Trees Using LLMs
Gadyatskaya, O., & Papuc, D. (2023) · Communications in computer and information science · doi:10.1007/978-981-99-7969-1_18 · Cited by 22
Bajohr, H. (2024) · Poetics Today · doi:10.1215/03335372-11092990 · Cited by 29
Teaching accounting in the era of ChatGPT – The student perspective
Sundkvist, C. H., & Kulset, E. M. (2024) · Journal of Accounting Education · doi:10.1016/j.jaccedu.2024.100932 · Cited by 63
Vats, S., Zope, A., De, S., Sharma, A., Bhattacharya, U., Nigam, S. K., Guha, S. K., Rudra, K., & Ghosh, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.831 · Cited by 21
Harašta, J., Novotná, T., & Šavelka, J. (2024) · Artificial Intelligence and Law · doi:10.1007/s10506-024-09422-w · Cited by 35
Kim, S., Kim, K., & Wonjeong Jo, C. (2024) · Preventive Medicine Reports · doi:10.1016/j.pmedr.2024.102723 · Cited by 18
Finch, L. A., Broach, V. A., Feinberg, J., Al‐Niaimi, A., Abu‐Rustum, N. R., Zhou, Q. C., Iasonos, A. E., & Dennis, S. (2024) · Gynecologic Oncology · doi:10.1016/j.ygyno.2024.07.007 · Cited by 19
Hurley, N. C., Schroeder, K. M., & Hess, A. S. (2023) · Transfusion · doi:10.1111/trf.17526 · Cited by 22
Karoui, A. T., Sayari, S., Dammak, W., & Jeribi, A. (2024) · Risks · doi:10.3390/risks12030052 · Cited by 18
English, R., Nash, R., & Mackenzie, H. (2026) · Innovations in Education and Teaching International · doi:10.1080/14703297.2024.2446236 · Cited by 17
LAB-Bench: Measuring Capabilities of Language Models for Biology Research
Laurent, J. M., Janizek, J. D., Ruzo, M., Hinks, M. M., Hammerling, M. J., Narayanan, S., Ponnapati, M., White, A. D., & Rodriques, S. G. (2024) · arXiv · doi:10.48550/arxiv.2407.10362 · Cited by 21
“…data (i.e. a huge fraction of the internet), and the scale of such training has allowed even general models to demonstrate emergent abilities to solve complex problems they were not explicitly trained for, including in medicine [25, 48], law [22], writing code [43], or other broad capabilities [7].”
BackgroundTrends in research on ChatGPT and adoption-related issues discussed in articles: a narrative review
Kim, S.-J. (2023) · Science Editing · doi:10.6087/kcse.321 · Cited by 17
Enhancing early depression detection with AI: a comparative use of NLP models
Hadžić, B., Mohammed, P., Danner, M., Ohse, J., Zhang, Y., Shiban, Y., & Rätsch, M. (2024) · SICE Journal of Control Measurement and System Integration · doi:10.1080/18824889.2024.2342624 · Cited by 17
Enhancing Classroom Learning: ChatGPT's Integration and Educational Challenges
AlAli, R. M., & Wardat, Y. (2024) · International Journal of Religion · doi:10.61707/znwnxd43 · Cited by 25
Johns, B., Carroll, K., Medina, C., Lewark, R., & Walliser, J. C. (2024) · INCOSE International Symposium · doi:10.1002/iis2.13201 · Cited by 17
ChatGPT’s performance in university admissions tests in mathematics
Udías, Á., Alonso‐Ayuso, A., Álfaro, C., Algar, M. J., Cuesta, M., Fernández-Isabel, A., Gómez, J., Lancho, C., Cano, E. L., de Diego, I. M., et al. (2024) · International Electronic Journal of Mathematics Education · doi:10.29333/iejme/15517 · Cited by 26
Lu, M., Gao, F., Tang, X., & Chen, L. (2024) · iScience · doi:10.1016/j.isci.2024.109451 · Cited by 19
Lotto, C., Sheppard, S. C., Anschuetz, W., Stricker, D., Molinari, G., Huwendiek, S., & Anschuetz, L. (2024) · OTO Open · doi:10.1002/oto2.70018 · Cited by 16
Classifying legal interpretations using large language models
Dugac, G., & Altwicker, T. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-025-09447-9 · Cited by 19
Recommender Systems Meet Large Language Model Agents: A Survey
Zhu, X., Wang, Y., Gao, H., Xu, W., Wang, C., Liu, Z., Wang, K., Jin, M., Pang, L. X., Weng, Q., et al. (2025) · Foundations and Trends® in Privacy and Security · doi:10.1561/3300000050 · Cited by 26
Usage of Large Language Model for Code Generation Tasks: A Review
Stefano Bistarelli, B., Fiore, M., Mercanti, I., & Mongiello, M. (2025) · SN Computer Science · doi:10.1007/s42979-025-04241-5 · Cited by 29
Natural and Artificial Intelligence: A Comparative Analysis of Cognitive Aspects
Abbate, F. (2023) · Minds and Machines · doi:10.1007/s11023-023-09646-w · Cited by 15
Gray, M., Šavelka, J., Oliver, W. M., & Ashley, K. D. (2024) · Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences · doi:10.1098/rsta.2023.0155 · Cited by 15
AI, Law and beyond. A transdisciplinary ecosystem for the future of AI & Law
Bex, F. (2024) · Artificial Intelligence and Law · doi:10.1007/s10506-024-09404-y · Cited by 21
Carl, N., Nguyen, L., Haggenmüller, S., Joachim Hetz, M., Theres Winterstein, J., Otto Hartung, F., Gruene, B., Nikolas Kather, J., Holland-Letz, T., Stephan Michel, M., et al. (2024) · European Urology Open Science · doi:10.1016/j.euros.2024.10.009 · Cited by 15
Quantum many-body physics calculations with large language models
Pan, H., Mudur, N., Taranto, W., Tikhanovskaya, M., Venugopalan, S., Bahri, Y., Brenner, M. P., & Kim, E.-A. (2025) · Communications Physics · doi:10.1038/s42005-025-01956-y · Cited by 15
An LLMs-based neuro-symbolic legal judgment prediction framework for civil cases
Wei, B., Yu, Y., Gan, L., & Wu, F. Q. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-025-09433-1 · Cited by 15
Distraction is All You Need for Multimodal Large Language Model Jailbreaking
Yang, Z., Fan, J., Yan, A., Gao, E., Lin, X., Li, T., Mo, K., & Dong, C. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00884 · Cited by 16
Metadata
- Authors
- Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P.
- Affiliations
- Illinois Institute of Technology (US)
- Casetext, Inc., USA
- Stanford University (US)
- Corresponding author
- Katz, Daniel Martin
- Venue
- Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
- Publisher
- The Royal Society
- Published
- 2024-02-26
- Volume
- 382
- Issue
- 2270
- Pages
- 20230254-20230254
- ISSN
- 1364-503X (ISSN-L)
- 1471-2962
- PMC
- PMC10894685
- OpenAlex
- W4392773006
- Semantic Scholar
- 0e8297549b4ec852ce8fd55dee9ae21501805af2
- Cited by
- 420
- References
- 63
- Citations by year
- 2026: 91 · 2025: 116 · 2024: 127
- FWCI
- 532.03
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences (publishedVersion, cc-by)
- SSRN Electronic Journal (acceptedVersion)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Law › Artificial Intelligence in Law
- Social Sciences › Law › Legal Education and Practice Innovations
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- GPT-4, bar examination, zero-shot performance, large language models, ChatGPT, legal education, legal services
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
