CoChat
Sign Up
PreprintCited by 18Artificial Intelligence in LawTopic ModelingLegal Language and Interpretation

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Explaining Legal Concepts with Augmented Large Language Models (GPT-4)

Open access
Savelka, Jaromir, Ashley, Kevin D., Gray, Morgan A., Westermann, Hannes, Xu, Huihui·2023·arXiv
DOI 10.48550/arxiv.2306.09525arXiv 2306.09525
Open this paper in CoChat — free account

Abstract

Interpreting the meaning of legal open-textured terms is a key task of legal professionals. An important source for this interpretation is how the term was applied in previous court cases. In this paper, we evaluate the performance of GPT-4 in generating factually accurate, clear and relevant explanations of terms in legislation. We compare the performance of a baseline setup, where GPT-4 is directly asked to explain a legal term, to an augmented approach, where a legal information retrieval module is used to provide relevant context to the model, in the form of sentences from case law. We found that the direct application of GPT-4 yields explanations that appear to be of very high quality on their surface. However, detailed analysis uncovered limitations in terms of the factual accuracy of the explanations. Further, we found that the augmentation leads to improved quality, and appears to eliminate the issue of hallucination, where models invent incorrect statements. These findings open the door to the building of systems that can autonomously retrieve relevant sentences from case law and condense them into a useful explanation for legal scholars, educators or practicing lawyers alike.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.09525

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 1 references

Research works printed but not linked: 30

  1. [1]1. Asai, A., Yu, X., Kasai, J., Hajishirzi, H.: One question answering model for many languages with cross-lingual dense passage retrieval. Advances in Neural Information Processing Systems 34 , 7547-7560 (2021)
  2. [3]3. Blair-Stanek, A., Holzenberger, N., Van Durme, B.: Can gpt-3 perform statutory reasoning? arXiv preprint arXiv:2302.06100 (2023) · arXiv
  3. [2]2. Ashley, K.D., Rissland, E.L.: Law, learning and representation. Artificial Intelligence 150 (1-2), 17-58 (2003)
  4. [4]4. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33 , 1877-1901 (2020)
  5. [6]6. Daci, J.: Legal principles, legal values and legal norms: are they the same or different? Academicus International Scientific Journal 02 , 109-115 (2010)
  6. [5]5. Chen, D., Fisch, A., Weston, J., Bordes, A.: Reading wikipedia to answer opendomain questions. arXiv preprint arXiv:1704.00051 (2017) · arXiv
  7. [7]7. Endicott, T.: Vagueness in Law. Oxford University Press (2000)
  8. [9]9. Hamilton, S.: Blind judgement: Agent-based supreme court modelling with gpt. arXiv preprint arXiv:2301.05327 (2023) · arXiv
  9. [10]10. Hart, H.L.: The Concept of Law. Clarendon Press, 2nd edn. (1994)
  10. [12]12. Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.t., Rocktäschel, T., et al.: Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems 33 , 9459-9474 (2020)
  11. [11]11. Katz, D.M., Bommarito, M.J., Gao, S., Arredondo, P.: Gpt-4 passes the bar exam. Available at SSRN 4389233 (2023)
  12. [13]13. MacCormick, D.N., Summers, R.S.: Interpreting Statutes: A Comparative Study. Routledge, Darmouth (1991)
  13. [15]15. Nakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al.: Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332 (2021) · arXiv
  14. [14]14. Mialon, G., Dessì, R., Lomeli, M., Nalmpantis, C., Pasunuru, R., Raileanu, R., Rozière, B., Schick, T., Dwivedi-Yu, J., Celikyilmaz, A., et al.: Augmented language models: a survey. arXiv preprint arXiv:2302.07842 (2023) · arXiv
  15. [16]16. Nguyen, H.T., Goebel, R., Toni, F., Stathis, K., Satoh, K.: How well do sota legal reasoning models support abductive reasoning? arXiv preprint arXiv:2304.06912 (2023) · arXiv
  16. [18]18. Paquin, L.C., Blanchard, F., Thomasset, C.: Loge-expert: from a legal expert system to an information system for non-lawyers. In: Proceedings of the 3rd international conference on Artificial intelligence and law. pp. 254-259 (1991)
  17. [19]19. Salaün, O., Troussel, A., Longhais, S., Westermann, H., Langlais, P., Benyekhlef, K.: Conditional abstractive summarization of court decisions for laymen and insights from human evaluation. In: Legal Knowledge and Information Systems, pp. 123-132. IOS Press (2022)
  18. [20]20. Savelka, J.: Unlocking practical applications in legal domain: Evaluation of gpt for zero-shot semantic annotation of legal texts. arXiv preprint arXiv:2305.04417 (2023) · arXiv
  19. [22]22. Šavelka, J., Ashley, K.D.: Discovering explanatory sentences in legal case decisions using pre-trained language models. In: Findings of the Association for Computational Linguistics: EMNLP 2021. pp. 4273-4283 (2021)
  20. [21]21. Šavelka, J., Ashley, K.D.: Extracting case law sentences for argumentation about the meaning of statutory terms. In: Proceedings of the third workshop on argument mining (ArgMining2016). pp. 50-59 (2016)
  21. [23]23. Šavelka, J., Ashley, K.D.: Legal information retrieval for understanding statutory terms. Artificial Intelligence and Law pp. 1-45 (2021)
  22. [25]25. Šavelka, J., Harašta, J.: Open texture in law, legal certainty and logical analysis of natural language. Logic in the Theory and Practice of Lawmaking pp. 159-171 (2015)
  23. [24]24. Savelka, J., Ashley, K.D.: On the role of past treatment of terms from written laws in legal reasoning. New Developments in Legal Reasoning and Logic: From Ancient Law to Modern Legal Systems pp. 379-395 (2022)
  24. [27]27. Savelka, J., Xu, H., Ashley, K.D.: Improving sentence retrieval from case law for statutory interpretation. In: Proceedings of the seventeenth international conference on artificial intelligence and law. pp. 113-122 (2019)
  25. [29]29. Shuster, K., Xu, J., Komeili, M., Ju, D., Smith, E.M., Roller, S., Ung, M., Chen, M., Arora, K., Lane, J., et al.: Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXiv:2208.03188 (2022) · arXiv
  26. [31]31. Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.T., Jin, A., Bos, T., Baker, L., Du, Y., et al.: Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239 (2022) · arXiv
  27. [30]30. Tan, Jinzhe, H.W., Benyekhlef, K.: Chatgpt as an artificial lawyer? In: Proceedings of The First Workshop on Artificial Intelligence for Access to Justice (2023)
  28. [34]34. Westermann, H., Benyekhlef, K.: Justicebot: A methodology for building augmented intelligence tools for laypeople to increase access to justice. In: Proceedings of ICAIL 2023 (2023)
  29. [33]33. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., Zhou, D.: Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022) · arXiv
  30. [37]37. Yu, F., Quartey, L., Schilder, F.: Legal prompting: Teaching a language model to think like a lawyer (2022). https://doi.org/10.48550/ARXIV.2212.01326 , https://arxiv.org/abs/2212.01326 · DOI

Cited by

Sorted by citations

See all 18

Metadata

Authors
Savelka, J., Ashley, K. D., Gray, M. A., Westermann, H., & Xu, H.
Affiliations
  • Carnegie Mellon University (US)
  • University of Pittsburgh (US)
  • University of St. Thomas - Minnesota (US)
  • Maastricht University (NL)
Corresponding author
Gray, Morgan A.
ORCID
Westermann, Hannes
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
1556-5068 (ISSN-L)
DOI
10.48550/arxiv.2306.09525
arXiv
2306.09525
OpenAlex
W7143531414
Cited by
18
References
1
Citations by year
2026: 1 · 2025: 7 · 2024: 6 · 2023: 4
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • SSRN Electronic Journal (acceptedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Research Publications (Maastricht University) (submittedVersion, other-oa)
  • arXiv (Cornell University) (cc-by)
Funders
University of Pittsburgh
Topics
  • Social Sciences › Law › Artificial Intelligence in Law
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Social Sciences › Law › Legal Language and Interpretation
Keywords
GPT-4, large language models, legal information retrieval, case law, factual accuracy, hallucinations, legal education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Explaining Legal Concepts with Augmented Large Language Models (GPT-4). https://app.cochat.ai/papers/10.48550/arxiv.2306.09525
The paper
Savelka, J., Ashley, K. D., Gray, M. A., Westermann, H., & Xu, H. (2023). Explaining Legal Concepts with Augmented Large Language Models (GPT-4). arXiv. https://doi.org/10.48550/arxiv.2306.09525