CoChat
Sign Up
PreprintCited by 4

Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities Emergence

Nay, John J., Karamardian, David, Lawsky, Sarah B., Tao, Wenting, Bhat, Meghana, Jain, Raghav, Lee, Aaron Travis, Choi, Jonathan H. +1 more·2023·arXiv
DOI 10.48550/arxiv.2306.07075arXiv 2306.07075
Open this paper in CoChat — free account

Abstract

Better understanding of Large Language Models' (LLMs) legal analysis abilities can contribute to improving the efficiency of legal services, governing artificial intelligence, and leveraging LLMs to identify inconsistencies in law. This paper explores LLM capabilities in applying tax law. We choose this area of law because it has a structure that allows us to set up automated validation pipelines across thousands of examples, requires logical reasoning and maths skills, and enables us to test LLM capabilities in a manner relevant to real-world economic lives of citizens and companies. Our experiments demonstrate emerging legal understanding capabilities, with improved performance in each subsequent OpenAI model release. We experiment with retrieving and utilising the relevant legal authority to assess the impact of providing additional legal context to LLMs. Few-shot prompting, presenting examples of question-answer pairs, is also found to significantly enhance the performance of the most advanced model, GPT-4. The findings indicate that LLMs, particularly when combined with prompting enhancements and the correct legal texts, can perform at high levels of accuracy but not yet at expert tax lawyer levels. As LLMs continue to advance, their ability to reason about law autonomously could have significant implications for the legal profession and AI governance.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2306.07075

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs, particularly when combined with prompting enhancements and correct legal texts, could perform better but not yet at expert tax lawyer levels.

    “Nay et al. [139] indicated that LLMs, particularly when combined with prompting enhancements and the correct legal texts, could perform better but not yet at expert tax lawyer levels.”

    From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [139]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 31

  1. Reppert J, Rachbach B, George C, Stebbing L, Byun J, Appleton M, et al. Iterated Decomposition: Improving Science Q&A by Supervising Reasoning Processes [Internet]. arXiv [cs.CL]. 2023 [cited 2023 Mar 31]. Available from: https://arxiv.org/abs/2301.01751 · arXiv
  2. Diao S, Wang P, Lin Y, Zhang T. Active Prompting with Chain-of-Thought for Large Language Models [Internet]. arXiv [cs.CL]. 2023 [cited 2023 Mar 31]. Available from: https://arxiv.org/abs/2302.12246 · arXiv
  3. Zhou Y, Maharjan S, Liu B. Scalable Prompt Generation for Semi-supervised Learning with Language Models [Internet]. arXiv [cs.CL]. 2023 [cited 2023 Mar 31]. Available from: https://arxiv.org/abs/2302.09236 · arXiv
  4. Kim G, Baldi P, McAleer S. Language Models can Solve Computer Tasks. arXiv preprint arXiv:2303.17491. 2023 Mar 31 · arXiv
  5. Press O, Zhang M, Min S, Schmidt L, Smith NA, Lewis M. Measuring and Narrowing the Compositionality Gap in Language Models. arXiv preprint arXiv:2210.03350. 2022 Oct 7. · arXiv
  6. Yao S, Zhao J, Yu D, Du N, Shafran I, Narasimhan K, et al. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv preprint arXiv:2210.03629. 2023 Oct 10 · arXiv
  7. Jin Z, Levine S, Gonzalez F, Kamal O, Sap M, Sachan M, et al. When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment [Internet]. arXiv [cs.CL]. 2022 [cited 2023 Mar 31]. Available from: https://arxiv.org/abs/2210.01478 · arXiv
  8. Lu A, Zhang H, Zhang Y, Wang X, Yang D. Bounding the Capabilities of Large Language Models in Open Text Generation with Prompt Constraints [Internet]. arXiv [cs.CL]. 2023 [cited 2023 Mar 31]. Available from: https://arxiv.org/abs/2302.09185 [Internet]. 2023 [cited 2023 Mar 31]. Available from: https://openreview.net/forum?id=em4xg1Gvxa 38 · arXiv
  9. Ishibashi Y, Bollegala D, Sudoh K, Nakamura S. Evaluating the Robustness of Discrete Prompts [Internet]. arXiv [cs.CL]. 2023 [cited 2023 Mar 31]. Available from: https://arxiv.org/abs/2302.05619 · arXiv
  10. Moghaddam SR, Honey CJ. Boosting Theory-of-Mind Performance in Large Language Models via Prompting. arXiv preprint arXiv:2304.11490. 2023. · arXiv
  11. Andreas J. Language Models as Agent Models. arXiv preprint arXiv:2212.01681. 2022 · arXiv
  12. FAIR, Bakhtin A, Brown N, Dinan E, Farina G, Flaherty C, Fried D, Goff A, Gray J, Hu H, Jacob AP, Komeili M, Konath K, Kwon M, Lerer A, Lewis M, Miller AH, Mitts S, Renduchintala A, Roller S, Rowe D, Shi W, Spisak J, Wei A, Wu D, Zhang H, Zijlstra M. Human-level play in the game of Diplomacy by combining language models with strategic reasoning. Science. 2022 Nov 25;378(6624):1067-1074. Available from: https://www.science.org/doi/10.1126/science.ade9097. doi: 10.1126/science.ade9097. PMID: 34976589. · DOI · Paper page
  13. Shinn N, Labash B, Gopinath A. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366. 2023 Mar 28 · arXiv
  14. Yang S, Nachum O, Du Y, Wei J, Abbeel P, Schuurmans D. Foundation Models for Decision Making: Problems, Methods, and Opportunities [Internet]. arXiv [cs.AI]. 2023 [cited 2023 Mar 31]. Available from: https://arxiv.org/abs/2303.04129 · arXiv
  15. Li AC, Brown E, Efros AA, Pathak D. Internet Explorer: Targeted Representation Learning on the Open Web [Internet]. arXiv [cs.LG]. 2023 [cited 2023 Mar 31]. Available from: https://arxiv.org/abs/2302.14051 · arXiv
  16. Carta T, Romac C, Wolf T, Lamprier S, Sigaud O, Oudeyer PY. Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning. arXiv preprint arXiv:2302.02662. 2023 Feb · arXiv
  17. Valmeekam K, Sreedharan S, Marquez M, Olmo A, Kambhampati S. On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark) [Internet]. arXiv [cs.AI]. 2023 [cited 2023 Mar 31]. Available from: https://arxiv.org/abs/2302.06706 · arXiv
  18. Wang Z, Cai S, Liu A, Ma X, Liang Y. Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents. arXiv preprint arXiv:2302.01560. 2023 Feb 3 · arXiv
  19. Li BZ, Chen W, Sharma P, Andreas J. LaMPP: Language Models as Probabilistic Priors for Perception and Action. arXiv preprint arXiv:2302.02801. 2023 Feb 7. · arXiv
  20. Liu Y, Iter D, Xu Y, Wang S, Xu R, Zhu C. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. arXiv preprint arXiv:2303.16634. 2023 · arXiv
  21. Liu A, Wu Z, Michael J, Suhr A, West P, Koller A, Swayamdipta S, Smith NA, Choi Y. We're Afraid Language Models Aren't Modeling Ambiguity. arXiv preprint arXiv:2304.14399. 2023 · arXiv
  22. Mialon G, Dessì R, Lomeli M, Nalmpantis C, Pasunuru R, Raileanu R, Rozière B, Schick T, Dwivedi-Yu J, Celikyilmaz A, Grave E, LeCun Y, Scialom T. Augmented Language Models: a Survey. arXiv. 2023 Feb 21. arXiv:2302.07842 · arXiv
  23. Peng B, Galley M, He P, Cheng H, Xie Y, Hu Y, et al. Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback. arXiv preprint. 2023 Feb 28:2302.12813
  24. Zhou S, Alon U, Xu FF, Wang Z, Jiang Z, Neubig G. DocPrompting: Generating Code by Retrieving the Docs. arXiv preprint arXiv:2207.05987. 2023 · arXiv
  25. Wu Y, Fan Y, Liang PP, Azaria A, Li Y, Mitchell TM. Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manuals. arXiv preprint arXiv:2302.04449. 2023 Feb 1 · arXiv
  26. Chen D, Yih W. Open-Domain Question Answering. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts. Online: Association for Computational Linguistics; 2020. p. 34-37. Available from: https://aclanthology.org/2020.acl-tutorials.8. doi: 10.18653/v1/2020.acl-tutorials.8 · DOI
  27. Sil A, Sen J, Iyer B, Franz M, Fadnis K, Bornea M, et al. PrimeQA: The Prime Repository for State-of-the-Art Multilingual Question Answering Research and Development. arXiv preprint arXiv:2301.09715. 2023 Jan 26 · arXiv
  28. Sun Z, Wang X, Tay Y, Yang Y, Zhou D. Recitation-Augmented Language Models. arXiv preprint arXiv:2210.01296. 2023 · arXiv
  29. Khattab O, Santhanam K, Li XL, Hall D, Liang P, Potts C, Zaharia M. Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP. arXiv preprint arXiv:2212.14024. 2023 · arXiv
  30. Ye J, Wu Z, Feng J, Yu T, Kong L. Compositional Exemplars for In-context Learning. arXiv preprint arXiv:2302.05698. 2023 Feb 15 · arXiv
  31. Ram O, Levine Y, Dalmedigos I, Muhlgay D, Shashua A, Leyton-Brown K, Shoham Y. In-Context Retrieval-Augmented Language Models. arXiv preprint arXiv:2302.00083. 2023 Feb · arXiv

Cited by

Sorted by citations

Authors
Nay, J. J., Karamardian, D., Lawsky, S. B., Tao, W., Bhat, M., Jain, R., Lee, A. T., Choi, J. H., & Kasai, J.
Venue
arXiv
Publisher
arXiv
Published
2023
DOI
10.48550/arxiv.2306.07075
arXiv
2306.07075
References
0
Type
Preprint
Licence
creativecommons.org/licenses/by/4.0/legalcode

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review