CoChat
Sign Up
Cited by 420Artificial Intelligence in LawLegal Education and Practice InnovationsTopic Modeling

GPT-4 passes the bar exam

Open access
Katz, Daniel Martin, Bommarito, Michael James, Gao, Shang, Arredondo, Pablo·2024·Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
DOI 10.1098/rsta.2023.0254PMC PMC10894685
Open this paper in CoChat — free account

Abstract

In this paper, we experimentally evaluate the zero-shot performance of GPT-4 against prior generations of GPT on the entire uniform bar examination (UBE), including not only the multiple-choice multistate bar examination (MBE), but also the open-ended multistate essay exam (MEE) and multistate performance test (MPT) components. On the MBE, GPT-4 significantly outperforms both human test-takers and prior models, demonstrating a 26% increase over ChatGPT and beating humans in five of seven subject areas. On the MEE and MPT, which have not previously been evaluated by scholars, GPT-4 scores an average of 4.2/6.0 when compared with much lower scores for ChatGPT. Graded across the UBE components, in the manner in which a human test-taker would be, GPT-4 scores approximately 297 points, significantly in excess of the passing threshold for all UBE jurisdictions. These findings document not just the rapid and remarkable advance of large language model performance generally, but also the potential for such models to support the delivery of legal services in society. This article is part of the theme issue ‘A complexity science approach to law and governance’.

Verified 2026-10-09 · 2 references not checked yet

AI snapshot

Written from the paper’s full text.

TL;DR

GPT-4 achieved zero-shot passing performance on the entire Uniform Bar Exam, including the MBE, MEE, and MPT, with an overall UBE score of approximately 297 points. The authors argue this demonstrates rapid LLM progress and potential to support legal services delivery.

Key findings

  • On the MBE, GPT-4 scored 75.7% accuracy, a 26.5% increase over ChatGPT, beating the average human test-taker by more than 7% and outperforming humans in five of seven subject areas.
  • On the MEE, GPT-4 averaged 4.2/6.0 versus ChatGPT’s 3.0/6.0; on the MPT, GPT-4 averaged 4.2/6.0 versus ChatGPT’s 2.8/6.0.
  • Combined UBE score was 297 points for GPT-4 and 213 points for ChatGPT, exceeding passing thresholds ranging from 260 to 273 points across UBE jurisdictions.
  • GPT-4 was Pareto dominant over prior GPT models on the MBE, with largest gains in Contracts and Evidence, though Civil Procedure was its weakest subject.

Methods

  • The authors evaluated zero-shot performance on the full UBE: official MBE practice questions and the most recently released July 2022 MEE and MPT questions.
  • MBE grading was automated, with three runs per prompt/parameter set and average scores reported; OpenAI ran a contamination check on the test questions.
  • MEE and MPT responses were graded by two authors—a tenured law professor and a multi-jurisdiction licensed attorney—using conservative scoring against state bar representative “good” answers, with additional peer review of selected samples.

Limitations

  • The authors state they could not replicate the NCBE’s full-scale grading process, and MEE/MPT scoring involved subjective qualitative assessment despite conservative standards and peer checks.
  • The paper states GPT-4 still made errors, including on a testamentary trust distribution, a civil procedure diversity jurisdiction question, and a real property future interest/Rule Against Perpetuities subquestion.
  • The text makes evident that MEE/MPT evaluation used only the July 2022 exam, the MBE questions could not be released, and results reflect minimal zero-shot prompting rather than more exhaustive prompt engineering or few-shot techniques.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1098/rsta.2023.0254

Cites

75 of 77 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 18 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    GPT-4 achieves 75.7% accuracy on the MBE.

    “| GPT-4 | 75.7% |”
    Quote in the paperSubstantial claim
  • Findingown data

    GPT-4 meets or exceeds the approximate passing threshold in all seven MBE subject matter areas.

    “it meets or exceeds the approximate passing threshold in all seven subject matter areas.”
    Quote in the paperSubstantial claim
  • Findingown data

    Civil Procedure is the worst MBE subject for GPT-4, ChatGPT and human test-takers.

    “Civil Procedure is both the worst subject for GPT-4, ChatGPT and human test-takers.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Contracts and Evidence are the MBE topics with the largest overall improvement for GPT-4.

    “Contracts and Evidence are the topics with the largest overall improvement.”
    Quote in the paperNot a substantial claim
  • Findingnone given

    GPT-4 has difficulty calculating the distribution of assets from a testamentary trust that has been deemed invalid.

    “First, it has difficulty calculating the distribution of assets from a testamentary trust which has been deemed to be invalid.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 24
  • Classified as probably not a substantial claim. 24

  • May combine two claims.

    Findingown data

    GPT-3.0 produces very thin output on the MEE Evidence question, vaguely recites relevant principles but does not properly connect them to facts and consistently reaches improper legal conclusions.

    “GPT-3.0 can vaguely recite some of the relevant principles and rules but does not properly connect those principles to the facts and consistently reaches improper legal conclusions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    ChatGPT fails to properly identify all four prongs of Rule 702 of the Federal Rules of Evidence, resulting in a failure to discuss Rule 702(a).

    “ChatGPT fails to properly identify all four prongs of Rule 702 of the Federal Rules of Evidence (FRE). This results in a failure to discuss Rule 702(a).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4 correctly discusses both FRE Rule 403 and Rule 404(b) on the MEE Evidence problem and does not devote attention to extraneous issues.

    “GPT-4, by contrast, correctly discusses both FRE Rule 403 and Rule 404(b) and does not devote attention to extraneous issues.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    ChatGPT fails to address FRE 403 on the second subquestion of the MEE Evidence problem and instead focuses significant attention on non-relevant rules FRE 701 and 702.

    “consider the second subquestion on the MEE Evidence problem (figures 7 and 10) where ChatGPT fails to address a relevant rule (FRE 403) and instead focuses significant attention on non-relevant rules (FRE 701 and 702).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The average student answers more than three in ten MBE questions incorrectly.

    “the average student answers more than three in ten questions incorrectly.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning

    These patterns repeat themselves over the remainder of both MPT-1 and MPT-2.

    “These type of patterns repeat themselves over the remainder of both MPT-1 and MPT-2.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    The July 2022 MPT featured two questions, each over the publicly available ChatGPT 4096 token limit: 5297 tokens for MPT-1 and 5188 tokens for MPT-2.

    “The July 2022 MPT featured two questions, each over the publicly available ChatGPT 4096 token limit—5297 tokens for MPT-1 and 5188 tokens for MPT-2.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Subquestion three of MPT-1 provides another example of the distinction in quality between GPT-4 and ChatGPT.

    “Subquestion three of MPT-1, as displayed in figures 16 and 20, provides another example of the distinction in quality between GPT-4 and ChatGPT.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    In the first subquestion of MPT-1, both GPT-4 and ChatGPT produce a lengthy and otherwise reasonable looking answer.

    “For example, in the first subquestion of MPT-1, both GPT-4 and ChatGPT produce a lengthy and otherwise reasonable looking answer.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4 scored 4.2/6.0 on MPT-1 and 4.1/6.0 on MPT-2, with an overall MPT score of 4.2; ChatGPT scored 3.0/6.0 on MPT-1 and 2.5/6.0 on MPT-2, with an overall MPT score of 2.8.

    “| MPT 1 - Hixon Marriages | 4.2/6.0 | 3.0/6.0 |”
    • Also stated in: § Multistate performance test results · plain
      “| MPT 2 - In re Briotti | 4.1/6.0 | 2.5/6.0 |”
    • Also stated in: § Multistate performance test results · plain
      “| overall score | 4.2 | 2.8 |”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GPT-4 obtains an overall UBE score of 297 points while ChatGPT obtains a score of 213 points.

    “GPT-4 obtains an overall UBE score of 297 points4 while ChatGPT obtains a score of 213 points.5”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GPT-4 correctly identifies that Columbia Law is controlling and provides reasonable arguments as to why Columbia should be selected under the 'significant relationship' test.

    “In figure 14, GPT-4 not only correctly identifies that Columbia Law is controlling but provides reasonable arguments as to why Columbia should be selected under the ‘significant relationship’ test.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    ChatGPT correctly identifies that it should address the Restatement of the Conflicts of Law but incorrectly cites §6 instead of §283 as the source of the 'significant relationship' test.

    “In figure 18, ChatGPT correctly identifies that it should address the Restatement of the Conflicts of Law as this is core to the answer. However, it incorrectly cites §6 as the proper source when it should cite §283 as the source of the ‘significant relationship’ test.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    ChatGPT fails to distinguish between the circumstances and assigns authority to the Franklin Courts that they do not possess.

    “ChatGPT fails to distinguish between the circumstances and assigns authority to the Franklin Courts that they do not possess.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Both models produced long-form answers to the MPT problems, but there are fairly straightforward differences in the quality of output produced.

    “Both of the models we considered produced long-form answers to the respective problems, but there are some fairly straightforward differences in the quality of output produced.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    Different jurisdictions impose different UBE passing score thresholds ranging from 260 (Alabama and Minnesota) to 273 points (Arizona).

    “As displayed in table 8, different jurisdictions impose different UBE passing score thresholds ranging from 260 (for states such as Alabama and Minnesota) to 273 points (for Arizona).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GPT-4 obtains within-category passing-level scores for both the MEE and MPT.

    “Although GPT-4 obtains within-category passing-level scores for both the MEE and MPT”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4 fails to grasp the call of the question and provides an incorrect answer on a civil procedure question regarding diversity jurisdiction after the joinder of a necessary party.

    “Next, it fails to grasp the call of the question and provides an incorrect answer on a civil procedure question regarding diversity jurisdiction after the joinder of a necessary party.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Both GPT-4 and, to a lesser extent, ChatGPT were able to largely avoid the trap of citing legal principles, cases, or other materials that would appear on point but would not be responsive to the MPT requirements.

    “Both GPT-4 and, to a lesser extent, ChatGPT were able to largely avoid the trap of citing legal principles, cases, or other materials which would otherwise appear to be on point but would not be responsive to the requirements of the MPT.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The final MEE grade is 4.2 out of 6 points for GPT-4 and 3 out of 6 points for ChatGPT.

    “Overall, as presented in table 5, our final MEE grade is 4.2 out of 6 points for GPT-4 and 3 out of 6 points for ChatGPT.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    The complexity of the law imposes real consequences for many individuals and organizations.

    “The complexity of the law [12–15] imposes real consequences for many individuals and organizations [16,17].”
    Quote in the paperNot a substantial claim0 of 6 sources checked
    • cites [12] Bourcier D, Mazzega P. 2007 Toward measures of complexity in legal systems. In Proceedings of the 11th international conference on Artificial intelligence and law, pp. 211–215. source

      Not checked in the free run.

    • cites [13] Ruhl J, Katz D, Bommarito M. 2017 Harnessing legal complexity. Science 355, 1377- 1378. (10.1126/science.aag3013) 28360284 source

      Not checked in the free run.

    • cites [14] Bommarito II M, Katz D. 2017 Measuring and modeling the US regulatory ecosystem. J. Stat. Phys. 168, 1125- 1135. (10.1007/s10955-017-1846-3) source

      Not checked in the free run.

    • cites [15] Katz DM, Coupette C, Beckedorf J, Hartung D. 2020 Complex societies and the growth of the law. Sci. Rep. 10, 1- 14. (10.1038/s41598-020-73623-x) 33127960 PMC7599339 source

      Not checked in the free run.

    • cites [16] Staudt RW. 2008 All the wild possibilities: technology that attacks barriers to access to justice. Loy. LAL Rev. 42, 1117.

      Not checked in the free run.

    • cites [17] Sandefur RL, Teufel J. 2020 Assessing America’s Access to Civil Justice Crisis. UC Irvine L. Rev. 11, 753.

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The rate of performance increase from text-davinci-001 to text-davinci-003 strongly suggested that passing performance could occur within the next 0-18 months.

    “the rate of performance increase from text-davinci-001 to text-davinci-003 strongly suggested that passing performance could ‘occur within the next 0–18 months’ [8].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [8] Bommarito MJ, Katz DM. 2022 GPT Takes the Bar Exam. Available at SSRN 4314839. source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data

    Two authors (a tenured law professor and an attorney licensed in multiple jurisdictions) reviewed model output and collaboratively assigned scores to each MEE question, using a conservative standard.

    “Two of this study’s authors, a tenured law professor and an attorney licensed in multiple jurisdictions, reviewed the model output and collaboratively assigned scores to each of the MEE questions.”
    • Also stated in: § Multistate essay examination results · plain
      “we debated each score with the view that we should be somewhat conservative in the assignment of MEE scores.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    All MBE grading was automated.

    “All grading is automated.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    The ability to interpret complex legal documents often requires years of study.

    “the ability to interpret such complex documents often requires years of study.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [6] Mertz E. 2007 The language of law school: learning to ‘think like a lawyer’. USA: Oxford University Press. source

      Not checked in the free run.

    • cites [7] Martínez E, Mollica F, Gibson E. 2022 Poor writing, not specialized concepts, drives processing difficulty in legal language. Cognition 224, 105070. (10.1016/j.cognition.2022.105070) 35257980 source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Premisecitation

    Legal systems have struggled to assist with the quantity, quality, and accessibility of legal services demanded by society.

    “legal systems have struggled to assist with the quantity, quality, and accessibility of legal services demanded by society [17–19].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [17] Sandefur RL, Teufel J. 2020 Assessing America’s Access to Civil Justice Crisis. UC Irvine L. Rev. 11, 753.

      Not checked in the free run.

    • cites [18] Rhode DL. 2004 Access to justice. Oxford, UK: Oxford University Press.

      Not checked in the free run.

    • cites [19] Susskind RE. 2019 Online courts and the future of justice. Oxford, UK: Oxford University Press. source

      Not checked in the free run.

  • Classified as probably not a substantial claim. May combine two claims.

    Premisenone given

    A combined UBE score of 266 points is enough to pass in jurisdictions such as Illinois, New York and the District of Columbia, while 270 points would pass in the vast majority of states which use the UBE.

    “a combined score of 266 points is enough to pass in jurisdictions such as Illinois, New York and the District of Columbia, while a score of 270 points would pass in the vast majority of states which use the UBE.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    Experiments involved three runs for each set of prompts and parameters, and the reported score is the average across all runs.

    “Experiments involved three runs for each set of prompts and parameters; we report the average score across all runs in our results.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    These findings document the potential for such models to support the delivery of legal services in society.

    “but also the potential for such models to support the delivery of legal services in society.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Bar exam performance is a valid indicator of a model's potential to support the delivery of legal services in society.

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    Legal language is notoriously complex.

    “Legal language is notoriously complex [3–5]”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [3] Friedrich R. 2021 Complexity and entropy in legal language. Front. Phys. 9, 671882. (10.3389/fphy.2021.671882) source

      Not checked in the free run.

    • cites [4] Katz DM, Bommarito MJ. 2014 Measuring the complexity of the law: the United States Code. Artif. Intell. Law 22, 337- 374. (10.1007/s10506-014-9160-8) source

      Not checked in the free run.

    • cites [5] Ruhl JB. 2007 Law’s complexity: a primer. Ga. St. UL Rev. 24, 885. (10.58948/0738-6206.1052)

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    The representative good answers reflect actual MEE and MPT answers produced by real examinees.

    “These representative good answers are made available by state bar associations and reflect actual MEE and MPT answers produced by real examinees.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The exact sequence of MBE questions administered is not identical to any actual exam as administered.

    “While the exact sequence of questions administered is not identical to any actual exam as administered”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The Franklin Court cannot, without personal jurisdiction over Tucker, take action against a property outside its borders.

    “The Franklin Court cannot, without personal jurisdiction over Tucker, take action against a property outside its borders.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    The UBE has three components (MBE, MEE and MPT) which are typically, but not always, weighed using the approach highlighted in table 1.

    “The UBE has three components (MBE, MEE and MPT) which are typically, but not always, weighed using the approach highlighted in table 1.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    GPT-4, like prior models, may still hallucinate sources, incorrectly interpret facts or fail to follow ethical requirements.

    “GPT-4, like prior models, may still hallucinate sources, incorrectly interpret facts or fail to follow ethical requirements”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The MPT is not a test of substantive law; the Library materials provide sufficient substantive information to complete the task.

    “As the NCBE describes it, ‘the MPT is not a test of substantive law; the Library materials provide sufficient substantive information to complete the task.’”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites NCBE

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    The MPT requires a suspension of knowledge, whereby the examinee must imagine themselves in a jurisdiction that may contradict their actual knowledge of real law.

    “Consequently, the MPT requires a suspension of knowledge, whereby the examinee must, for the period of the test, imagine themselves in a jurisdiction that may contradict their actual knowledge of real law.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    There is a subjective aspect to the MPT grading analysis.

    “As we noted in the grading of the MEE, we recognize that there is a subjective aspect to this sort of analysis”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    GPT-4's high MBE percentile provided it with substantial latitude; GPT-4 would likely pass even with a much lower MEE or MPT score in some or all jurisdictions.

    “its high MBE percentile provided it with substantial latitude; GPT-4 would likely pass even with a much lower MEE or MPT score in some or all jurisdictions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductreasoning

    An 8K version of ChatGPT with a wider context window of 8193 tokens was used, which could accommodate the length of the overall materials.

    “However, thanks to assistance from OpenAI, we were able to use an ‘8K’ version of ChatGPT that has a wider context window of 8193 tokens, and could thus accommodate the length of the overall materials.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Prior models were unable to handle longer documents in zero-shot tasks due to their token limits.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The MPT instructions remind the test-taker that even if cases appear familiar, they should not assume they are precisely the same as previously read and should read them thoroughly as if new.

    “Indeed, the instructions provided with the test remind the test-taker that even ‘if the cases appear familiar to you, do not assume that they are precisely the same as you have read before. Read them thoroughly, as if they all were new to you.’”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    GPT-4 has indeed passed the Bar and has done so by a significant margin.

    “In total, our analysis highlights that GPT-4 has indeed passed the Bar and has done so by a significant margin.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Most jurisdictions use a six-point scale, and a score of four or higher is generally considered passing.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Several topics where GPT-4 struggles are also areas where law students and bar examinees would likely struggle.

    “It should be noted that several of these topics where GPT-4 struggles are also areas where law students and bar examinees would also likely struggle.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The Rule Against Perpetuities is considered by many to be among the most difficult issues in all of law.
43 of 43

  • The paper relies on this without support.

    Bar exam performance is a valid indicator of a model's potential to support the delivery of legal services in society.

    “but also the potential for such models to support the delivery of legal services in society.”
    “we sought an exemplary challenge to demonstrate this potential to both the legal domain and general scientific community.”

  • The paper relies on this without support.

    The NCBE BarNow average performance by topic is comparable to the performance of human test-takers on the specific MBE exam used in this study.

    “Table 3 offers both the NCBE Bar Now average accuracy by question category as well as the overall approximate national average MBE performance for recent test-takers.”
    “it outperforms the average NCBE BarNow test-taker in five out of seven categories (Civil Procedure, Contracts, Criminal Law and Procedure, Evidence and Real Property).”

  • The paper relies on this without support.

    The July 2022 MEE and MPT questions are representative of the UBE for the purpose of evaluating GPT-4's ability to pass the bar.

    “For the MEE and the MPT, we collected the most recently released questions from the July 2022 Bar Examination.”
    “GPT-4 can ‘pass the Bar’ in all UBE jurisdictions.”

  • The paper relies on this without support.

    Most jurisdictions use a six-point scale, and a score of four or higher is generally considered passing.

    “Most jurisdictions leverage the six-point scale, where a score of four or higher is generally considered passing.”

  • The paper relies on this without support.

    The Rule Against Perpetuities is considered by many to be among the most difficult issues in all of law.

    “In particular, the Rule Against Perpetuities is considered by many to be among the most difficult issues in all of law.”

  • The paper relies on this without support.

    Most real-life examinees who otherwise pass the Bar Exam are unable to complete an end-to-end MEE that is free from errors.

    “In addition, it should be mentioned that most real life examinees who otherwise pass the Bar Exam are unable to complete an end-to-end MEE that is free from errors.”

  • The paper relies on this without support.

    Prior models were unable to handle longer documents in zero-shot tasks due to their token limits.

    “As many have noted, prior models were unable to handle longer documents in zero-shot tasks due to their token limits.”
7 of 7

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • GPT has exhibited human-level performance on a variety of tasks and better performance than existing LLMs.

    “GPT has been lauded for its ability to exhibit human-level performance on a variety of tasks (e.g., passing the Bar Exam or acing the SAT test), and better performance than existing LLMs (47, 48).”

    From GPT is an effective tool for multilingual psychological text analysis (2024) · cites this paper as [47]

Cites

Flagged first, then by citations

Research works printed but not linked: 14

  1. Tiersma PM. 1999 Legal language. Chicago, IL: University of Chicago Press.
  2. Rhode DL. 2004 Access to justice. Oxford, UK: Oxford University Press.
  3. Von Neumann J. 1958 The computer and the brain. New Haven, CT: Yale University Press.
  4. Vaswani A Shazeer N Parmar N Uszkoreit J Jones L Gomez AN Kaiser L Polosukhin I. 2017 Attention is all you need. In Advances in neural information processing systems vol. 30.
  5. Christiano PF Leike J Brown T Martic M Legg S Amodei D. 2017 Deep reinforcement learning from human preferences. Advances in neural information processing systems vol. 30.
  6. Devlin J Chang MW Lee K Toutanova K. 2018 BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019 pp. 4171–4186.
  7. Kenton JDMWC Toutanova LK. 2019 BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies pp. 4171–4186.
  8. Brown T Mann B Ryder N Subbiah M Kaplan JD Dhariwal P Neelakantan A Shyam P Sastry G Askell A Agarwal S. 2020 Language models are few-shot learners. In Advances in neural information processing systems pp. 1877–1901.
  9. Barrett D Hill F Santoro A Morcos A Lillicrap T. 2018 Measuring abstract reasoning in neural networks. In International conference on machine learning pp. 511–520. PMLR.
  10. Wang A Pruksachatkun Y Nangia N Singh A Michael J Hill F Levy O Bowman S. 2019 Superglue: a stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems 32.
  11. Mitchell M Krakauer DC. 2022 The Debate Over Understanding in AI’s Large Language Models. (http://arxiv.org/abs/2210.13966) · arXiv
  12. Fawei B Wyner A Pan J. 2016 Passing a USA national bar exam: a first corpus for experimentation. In LREC 2016 Tenth International Conference on Language Resources and Evaluation pp. 23–30.
  13. Wang Z Panda R Karlinsky L Feris R Sun H Kim Y. 2023 Multitask prompt tuning enables parameter-efficient transfer learning. In The Eleventh International Conference on Learning Representations.
  14. Wei J et al. 2022 Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems 35 24 824-24 837.

Cited by

Sorted by citations

Related

Authors
Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P.
Affiliations
  • Illinois Institute of Technology (US)
  • Casetext, Inc., USA
  • Stanford University (US)
Corresponding author
Katz, Daniel Martin
ORCID
Venue
Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
Publisher
The Royal Society
Published
2024-02-26
Volume
382
Issue
2270
Pages
20230254-20230254
ISSN
  • 1364-503X (ISSN-L)
  • 1471-2962
DOI
10.1098/rsta.2023.0254
PMC
PMC10894685
OpenAlex
W4392773006
Semantic Scholar
0e8297549b4ec852ce8fd55dee9ae21501805af2
Cited by
420
References
63
Citations by year
2026: 91 · 2025: 116 · 2024: 127 · 2023: 86
FWCI
532.03
Type
journal article
Language
en
Open access
Licence
Locations
  • Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences (publishedVersion, cc-by)
  • SSRN Electronic Journal (acceptedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Topics
  • Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
  • Social Sciences › Law › Legal Education and Practice Innovations
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
GPT-4, bar examination, zero-shot performance, large language models, ChatGPT, legal education, legal services

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review