Legalbench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
Open accessAbstract
The advent of large language models (LLMs) and their adoption by the legal community has given rise to the question: what types of legal reasoning can LLMs perform? To enable greater study of this question, we present LegalBench: a collaboratively constructed legal reasoning benchmark consisting of 162 tasks covering six different types of legal reasoning. LegalBench was built through an interdisciplinary process, in which we collected tasks designed and hand-crafted by legal professionals. Because these subject matter experts took a leading role in construction, tasks either measure legal reasoning capabilities that are practically useful, or measure reasoning skills that lawyers find interesting. To enable cross-disciplinary conversations about LLMs in the law, we additionally show how popular legal frameworks for describing legal reasoning—which distinguish between its many forms—correspond to LegalBench tasks, thus giving lawyers and LLM developers a common vocabulary. This paper describes LegalBench, presents an empirical evaluation of 20 open-source and commercial LLMs, and illustrates the types of research explorations LegalBench enables.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Lawyers are often taught to analyze situations with an IRAC framework-first identify the issue and governing legal rule, then analyze the facts with that rule to arrive at a conclusion.
“Lawyers are often taught to analyze situations with an IRAC framework-first identify the issue (I) and governing legal rule (R), then analyze (A) the facts with that rule to arrive at a conclusion (C) (Guha et al. 2023).”
From Hallucination‐Free? Assessing the Reliability of Leading <scp>AI</scp> Legal Research Tools (2025) · cites this paper as Guha et al. 2023
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Self-citationOn the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Self-citationLlama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469 · see citing works
Opportunities and obstacles for deep learning in biology and medicine
Ching, T., Himmelstein, D. S., Beaulieu-Jones, B. K., Kalinin, A. A., Do, B. T., Way, G. P., Ferrero, E., Agapow, P.-M., Zietz, M., Hoffman, M. M., et al. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0387 · Cited by 2,310 · see citing works
The Cambridge handbook of thinking and reasoning
Holyoak, K. J., & Morrison, B. (2005) · Cited by 2,133 · see citing works
Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity RecognitionPreprint
Sang, E. T. K. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0209010 · Cited by 1,677 · see citing works
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture notes in computer science · doi:10.1007/11736790_9 · Cited by 1,674 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
WizardLM: Empowering Large Pre-Trained Language Models to Follow Complex InstructionsPreprint
Xu, C., Sun, Q., Zheng, K., Geng, X.-B., Zhao, P., Feng, J., Tao, C., Lin, Q.-W., & Jiang, D. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2304.12244 · Cited by 1,362 · see citing works
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2205.14135 · Cited by 1,214 · see citing works
Self-citationSuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051 · see citing works
Neural Network Acceptability Judgments
Warstadt, A., Singh, A., & Bowman, S. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00290 · Cited by 984 · see citing works
Obar, J. A., & Oeldorf-Hirsch, A. (2020) · Information, Communication & Society · doi:10.1080/1369118x.2018.1486870 · Cited by 771 · see citing works
Training Verifiers to Solve Math Word Problems
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.14168 · Cited by 575 · see citing works
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · see citing works
Holistic Evaluation of Language Models
Bommasani, R., Liang, P., & Lee, T. (2023) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.15007 · Cited by 480 · see citing works
Does Anyone Read the Fine Print? Consumer Attention to Standard-Form Contracts
Bakos, Y., Marotta-Wurgler, F., & Trossen, D. R. (2014) · The Journal of Legal Studies · doi:10.1086/674424 · Cited by 436 · see citing works
(2021) · International Conference on Learning Representations · Cited by 435 · see citing works
Legal Judgment Prediction via Topological Learning
Zhong, H., Guo, Z., Tu, C., Xiao, C., Liu, Z., & Sun, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1390 · Cited by 369 · see citing works
Learning to Predict Charges for Criminal Cases with Legal Basis
Luo, B., Feng, Y., Xu, J., Zhang, X., & Zhao, D. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1289 · Cited by 329 · see citing works
The Creation and Analysis of a Website Privacy Policy Corpus
Wilson, S., Schaub, F., Dara, A. A., Liu, F., Cherivirala, S., Leon, P. G., Andersen, M. S., Zimmeck, S., Sathyendra, K. M., Russell, N. C., et al. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1126 · Cited by 283 · see citing works
Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., et al. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.824 · Cited by 253 · see citing works
CLAUDETTE: an automated detector of potentially unfair clauses in online terms of service
Lippi, M., Pałka, P., Contissa, G., Lagioia, F., Micklitz, H., Giovanni Sartor, B., & Torroni, P. (2019) · Artificial Intelligence and Law · doi:10.1007/s10506-019-09243-2 · Cited by 240 · see citing works
True Few-Shot Learning with Language Models
Perez, E., Kiela, D., & Cho, K. (2021) · arXiv · doi:10.48550/arxiv.2105.11447 · Cited by 226 · see citing works
Text summarization from legal documents: a survey
Kanapala, A., Pal, S., & Pamula, R. (2017) · Artificial Intelligence Review · doi:10.1007/s10462-017-9566-2 · Cited by 194 · see citing works
(2019) · bepress Legal Repository · Cited by 192 · see citing works
Deep learning in law: early adaptation and legal word embeddings trained on large corpora
Chalkidis, I., & Kampas, D. (2018) · Artificial Intelligence and Law · doi:10.1007/s10506-018-9238-9 · Cited by 190 · see citing works
MAPS: Scaling Privacy Compliance Analysis to a Million Apps
Zimmeck, S., Story, P., Smullen, D., Ravichander, A., Wang, Z., Reidenberg, J. R., Russell, N. C., & Sadeh, N. (2019) · Proceedings on Privacy Enhancing Technologies · doi:10.2478/popets-2019-0037 · Cited by 177 · see citing works
ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation
Malik, V., Sanjay, R., Nigam, S. K., Ghosh, K., Guha, S. K., Bhattacharya, A., & Modi, A. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.313 · Cited by 176 · see citing works
The Litigation State Public Regulation and Private Lawsuits in the United States
Farhang, S. (2010) · doi:10.1515/9781400836789 · Cited by 168 · see citing works
Penedo, G., Malartic, Q., Hesslow, D., Cojocaru, R., Cappelli, A., Alobeidli, H., Pannier, B., Almazrouei, E., & Launay, J. (2023) · arXiv · doi:10.48550/arxiv.2306.01116 · Cited by 164 · see citing works
Peng, B., Galley, M., He, P., Cheng, H., Xie, Y., Hu, Y., Huang, Q., Liden, L., Yu, Z., Chen, W., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.12813 · Cited by 156 · see citing works
Hungry Hungry Hippos: Towards Language Modeling with State Space Models
Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., & Ré, C. (2022) · arXiv · doi:10.48550/arxiv.2212.14052 · Cited by 123 · see citing works
Self-citationNLP Based Latent Semantic Analysis for Legal Text Summarization
Merchant, K., & Pande, Y. (2018) · International Conference on Advances in Computing, Communications and Informatics (ICACCI) · doi:10.1109/icacci.2018.8554831 · Cited by 120 · see citing works
A Survey on Legal Judgment Prediction: Datasets, Metrics, Models and Challenges
Cui, J., Shen, X., & Wen, S. (2023) · IEEE Access · doi:10.1109/access.2023.3317083 · Cited by 117 · see citing works
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English
Chalkidis, I., Jana, A., Hartung, D., Bommarito, M. J., Androutsopoulos, I., Katz, D. M., & Aletras, N. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3936759 · Cited by 111 · see citing works
Artificial Intelligence and Legal Analytics: New Tools for Law Practice in the Digital Age
Zataraín, J. M. N. (2018) · SCRIPTed A Journal of Law Technology & Society · doi:10.2966/scrip.150118.156 · Cited by 103 · see citing works
Question Answering for Privacy Policies: Combining Computational and Legal Perspectives
Ravichander, A., Black, A. W., Wilson, S., Norton, T. B., & Sadeh, N. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1500 · Cited by 103 · see citing works
ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts
Koreeda, Y., & Manning, C. D. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.164 · Cited by 102 · see citing works
The No Reading Problem in Consumer Contract Law
Ayres, I., & Schwartz, A. (2013) · eYLS (Yale Law School) · Cited by 97 · see citing works
Hyena Hierarchy: Towards Larger Convolutional Language Models
Poli, M., Massaroli, S., Nguyen, E., Fu, D. Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., & Ré, C. (2023) · arXiv · doi:10.48550/arxiv.2302.10866 · Cited by 96 · see citing works
Self-citationNatural Language Processing in the Legal DomainPreprint
Hartung, D., Katz, D. M., Bommarito, M. J., Gerlach, L., Jana, A., & Soh, J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4336224 · Cited by 92 · see citing works
Rethinking the field of automatic prediction of court decisions
Medvedeva, M., Wieling, M. B., & Vols, M. (2022) · Artificial Intelligence and Law · doi:10.1007/s10506-021-09306-3 · Cited by 89 · see citing works
Overview and Discussion of the Competition on Legal Information Extraction/Entailment (COLIEE) 2021
Rabelo, J. C. B., Goebel, R. G., Kim, M., Kano, Y., Yoshioka, M., & Satoh, K. (2022) · The Review of Socionetwork Strategies · doi:10.1007/s12626-022-00105-z · Cited by 89 · see citing works
Chalkidis, I., Fergadiotis, M., & Androutsopoulos, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.559 · Cited by 89 · see citing works
Ask Me Anything: A simple strategy for prompting language modelsPreprint
Arora, S., Narayan, A., Chen, M. F., Orr, L., Guha, N., Bhatia, K., Chami, I., Sala, F., & Ré, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.02441 · Cited by 84 · see citing works
Self-citationCampaign Finance and American Democracy
Dawood, Y. A. (2015) · Annual Review of Political Science · doi:10.1146/annurev-polisci-010814-104523 · Cited by 80 · see citing works
AI Assistance in Legal Analysis: An Empirical Study
Choi, J. H., & Schwarcz, D. (2023) · doi:10.2139/ssrn.4539836 · Cited by 80 · see citing works
Self-citationRevisiting Transformer-based Models for Long Document Classification
Dai, X., Chalkidis, I., Darkner, S., & Elliott, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.534 · Cited by 76 · see citing works
Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmark
Niklaus, J., Chalkidis, I., & Stuermer, M. (2021) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2021.nllp-1.3 · Cited by 70 · see citing works
Self-citationSelective annotation makes language models better few-shot learners
Su, H., Kasai, J., Wu, C. H., Shi, W., Wang, T., Xin, J., Zhang, R., Ostendorf, M., Zettlemoyer, L., Smith, N. A., et al. (2022) · arXiv · doi:10.48550/arxiv.2209.01975 · Cited by 70 · see citing works
Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset
Henderson, P., Krass, M., Zheng, L., Guha, N., Manning, C. D., Jurafsky, D., & Ho, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2119 · Cited by 62 · see citing works
Self-citationChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmarkPreprint
Chalkidis, I. (2023) · Social Science Research Network · doi:10.2139/ssrn.4385460 · Cited by 57 · see citing works
Paul, S., Goyal, P., & Ghosh, S. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21363 · Cited by 53 · see citing works
LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain
Niklaus, J., Matoshi, V., Rani, P., Galassi, A., Stürmer, M., & Chalkidis, I. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.200 · Cited by 50 · see citing works
Self-citationPile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset
Henderson, P., Krass, M. S., Zheng, L., Guha, N., Manning, C. D., Jurafsky, D., & Ho, D. E. (2022) · arXiv · doi:10.48550/arxiv.2207.00220 · Cited by 49 · see citing works
Self-citationLeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development
Chalkidis, I., Garneau, N., Goanță, C., Katz, D. M., & Søgaard, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.865 · Cited by 48 · see citing works
Semantic Segmentation of Legal Documents via Rhetorical Roles
Malik, V., Sanjay, R., Guha, S. K., Hazarika, A., Nigam, S. K., Bhattacharya, A., & Modi, A. (2022) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2022.nllp-1.13 · Cited by 48 · see citing works
Oral Arguments Before the Supreme Court
Wrightsman, L. S. (2008) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195368628.001.0001 · Cited by 48 · see citing works
Bootleg: Chasing the Tail with Self-Supervised Named Entity DisambiguationPreprint
Orr, L., Leszczynski, M., Arora, S., Wu, S., Guha, N., Xiao, L., & Ré, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10363 · Cited by 48 · see citing works
Self-citationRAFT: A Real-World Few-Shot Text Classification BenchmarkPreprint
Alex, N., Lifland, E., Tunstall, L., Thakur, A., Maham, P., Riedel, C. J., Hine, E., Ashurst, C., Sedille, P., Carlier, A., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.14076 · Cited by 47 · see citing works
Exploring the Effectiveness of Prompt Engineering for Legal Reasoning Tasks
Yu, F., Quartey, L., & Schilder, F. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.858 · Cited by 47 · see citing works
Legal Judgment Prediction: A Survey of the State of the Art
Feng, Y., Li, C., & Ng, V. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/765 · Cited by 44 · see citing works
Modeling law search as prediction
Dadgostari, F., Guim, M., Beling, P. A., Livermore, M. A., & Rockmore, D. N. (2020) · Artificial Intelligence and Law · doi:10.1007/s10506-020-09261-5 · Cited by 39 · see citing works
Self-citationFairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing
Chalkidis, I., Pasini, T., Zhang, S., Tomada, L., Schwemer, S. F., & Søgaard, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.301 · Cited by 36 · see citing works
A Dataset for Statutory Reasoning in Tax Law Entailment and Question AnsweringPreprint
Holzenberger, N., Blair-Stanek, A., & Durme, B. V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.05257 · Cited by 34 · see citing works
Self-citationLegal Prompting: Teaching a Language Model to Think Like a LawyerPreprint
Yu, F., Quartey, L., & Schilder, F. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.01326 · Cited by 34 · see citing works
Large Language Models as Tax Attorneys: A Case Study in Legal Capabilities EmergencePreprint
Nay, J. J., Karamardian, D., Lawsky, S. B., Tao, W., Bhat, M., Jain, R., Lee, A. T., Choi, J. H., & Kasai, J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4476325 · Cited by 32 · see citing works
Self-citationCan GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?Preprint
Šavelka, J., Ashley, K. D., Gray, M., Westermann, H., & Xu, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.13906 · Cited by 31 · see citing works
Predicting and understanding law-making with word vectors and an ensemble model
Nay, J. J. (2017) · PLoS ONE · doi:10.1371/journal.pone.0176999 · Cited by 30 · see citing works
Self-citationLegal Tech, Civil Procedure, and the Future of Adversarialism
Engstrom, D. F., & Gelbach, J. B. (2021) · Penn Carey Law Legal Scholarship Repository (University of Pennsylvania) · Cited by 29 · see citing works
MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understanding
Wang, S., Scardigli, A., Tang, L., Chen, W., Levkin, D., Chen, A., Ball, S., Woodside, T., Zhang, O. R., & Hendrycks, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1019 · Cited by 26 · see citing works
Multi-granular Legal Topic Classification on Greek Legislation
Papaloukas, C., Chalkidis, I., Athinaios, K., Pantazi, D.-A., & Koubarakis, M. (2021) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2021.nllp-1.6 · Cited by 25 · see citing works
Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities
Shen, Z., Lo, K., Yu, L., Dahlberg, N., Schlanger, M., & Downey, D. (2022) · neural information processing systems · doi:10.52202/068431-0956 · Cited by 24 · see citing works
A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Prediction
Hwang, W., Lee, D., Cho, K., Lee, H., & Seo, M. (2022) · neural information processing systems · doi:10.52202/068431-2358 · Cited by 24 · see citing works
How to Use Large Language Models for Empirical Legal Research
Choi, J. H. (2024) · Journal of Institutional and Theoretical Economics JITE · doi:10.1628/jite-2024-0006 · Cited by 23 · see citing works
Self-citationSüzgün, M., Melas-Kyriazi, L., Sarkar, S. K., Kominers, S. D., & Shieber, S. M. (2023) · neural information processing systems · doi:10.52202/075280-2524 · Cited by 19 · see citing works
The Limitations of Supply Chain Disclosure Regimes
Chilton, A. S., & Sarfaty, G. A. (2016) · bepress Legal Repository · Cited by 18 · see citing works
Self-citationPredicting Consumer ContractsPreprint
Kolt, N. (2022) · SSRN Electronic Journal · doi:10.15779/z382b8vc90 · Cited by 15 · see citing works
Self-citationRosa, G., Bonifacio, L., Jeronymo, V., Abonizio, H., Lotufo, R., & Nogueira, R. (2022) · arXiv.org · doi:10.48550/arxiv.2205.15172 · Cited by 15 · see citing works
Legal Case Document Summarization: Extractive and Abstractive Methods and their EvaluationPreprint
Shukla, A., Bhattacharya, P., Poddar, S., Mukherjee, R., Ghosh, K., Goyal, P., & Ghosh, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.07544 · Cited by 13 · see citing works
Artificial Intelligence for Adjudication: The Social Security Administration and AI GovernancePreprint
Glaze, K., Ho, D. E., Ray, G. K., & Tsang, C. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3935950 · Cited by 11 · see citing works
Self-citationLegalBench: Prototyping a Collaborative Benchmark for Legal ReasoningPreprint
Guha, N., Ho, D. E., Nyarko, J., & Ré, C. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2209.06120 · Cited by 11 · see citing works
Self-citationValues Embedded in Legal Artificial Intelligence
Surden, H. (2022) · IEEE Technology and Society Magazine · doi:10.1109/mts.2022.3147542 · Cited by 10 · see citing works
Legal Transformer Models May Not Always HelpPreprint
Geng, S., Lebret, R., & Aberer, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.06862 · Cited by 10 · see citing works
Generative InterpretationPreprint
Hoffman, D. A., & Arbel, Y. A. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4526219 · Cited by 10 · see citing works
The Promise of AI in an Open Justice System
Pah, A. R., Schwartz, D. L., Sanga, S., Alexander, C. S., Hammond, K. J., & Amaral, L. A. N. (2022) · AI Magazine · doi:10.1002/aaai.12039 · Cited by 9 · see citing works
Simple Hardware-Efficient Long Convolutions for Sequence ModelingPreprint
Fu, D. Y., Epstein, E. L., Nguyen, E., Thomas, A. W., Zhang, M., Dao, T., Atri Rudra, B., & Ré, C. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.06646 · Cited by 9 · see citing works
Self-citationU-CREAT: Unsupervised Case Retrieval using Events extrAcTion
Joshi, A., Sharma, A., Tanikella, S. K., & Modi, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.777 · Cited by 8 · see citing works
How Smart Are Smart Readers? LLMs and the Future of the No-Reading Problem
Arbel, Y. A., & Becher, S. I. (2025) · Cambridge University Press eBooks · doi:10.1017/9781009279079.019 · Cited by 8 · see citing works
The lobbying manual : a complete guide to federal lobbying law and practice
Luneburg, W. V., Susman, T. M., Gordon, R. H., & Practice, R. (2009) · American Bar Association eBooks · Cited by 7 · see citing works
One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial SupportPreprint
Stern, R., Rasiah, V., Matoshi, V., Bose, S. B., Stuermer, M., Chalkidis, I., Ho, D. E., & Niklaus, J. (2023) · arXiv (Cornell University) · doi:10.24451/arbor.19713 · Cited by 7 · see citing works
Self-citationFrom Policy Confusion to Doctrinal Clarity: Successor Liability from the Prospective of Big DataPreprint
Fagan, F. (2014) · SSRN Electronic Journal · doi:10.2139/ssrn.2488819 · Cited by 6 · see citing works
Self-citationLegal Syllogism Prompting: Teaching Large Language Models for Legal Judgment PredictionPreprint
Jiang, C., & Yang, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.08321 · Cited by 6 · see citing works
A Computational Analysis of Oral Argument in the Supreme CourtPreprint
Dickinson, G. M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.05373 · Cited by 5 · see citing works
Self-citationEmbroid: Unsupervised Prediction Smoothing Can Improve Few-Shot Classification
Guha, N., Chen, M. F., Bhatia, K., Mirhoseini, A., Sala, F., & Ré, C. (2023) · neural information processing systems · doi:10.52202/075280-2762 · Cited by 4 · see citing works
Self-citationPrivate Enforcement in the StatesPreprint
Zambrano, D. A., Guha, N., Peters, A., & Xia, J. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4365144 · Cited by 3 · see citing works
Self-citationShelter Check: Proactively Finding Tax Minimization Strategies via AI
Blair-Stanek, A., Holzenberger, N., & Durme, B. V. (2022) · HAL · Cited by 2 · see citing works
Self-citationDefeating the Empire of FormsPreprint
Hoffman, D. A. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4334425 · Cited by 2 · see citing works
Breaking News: Drafting Client Alerts to Prepare for PracticePreprint
Silver, C. (2020) · SSRN Electronic Journal · Cited by 1 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
OPT: Open Pre-trained Transformer Language Models
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. (2022) · arXiv · doi:10.48550/arxiv.2205.01068
CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
Hendrycks, D., Burns, C., Chen, A., & Ball, S. (2021) · arXiv · doi:10.48550/arxiv.2103.06268
Flagged first, then by citations
Cited by
Sorted by citations
A Comprehensive Overview of Large Language Models
Naveed, H., Khan, A. U., Qiu, S., Saqib, M., Anwar, S., Usman, M., Akhtar, N., Barnes, N., & Mian, A. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3744746 · Cited by 1,257 · see citing works
Katz, D. M., Bommarito, M. J., Gao, S., & Arredondo, P. (2024) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2023.0254 · Cited by 420 · see citing works
Self-citationA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024) · Journal of Legal Analysis · doi:10.1093/jla/laae003 · Cited by 288 · see citing works
Self-citationHallucination‐Free? Assessing the Reliability of Leading <scp>AI</scp> Legal Research Tools
Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. (2025) · Journal of Empirical Legal Studies · doi:10.1111/jels.12413 · Cited by 182 · see citing works
Self-citationLarge Language Models on Graphs: A Comprehensive Survey
Jin, B., Liu, G., Han, C., Jiang, M., Ji, H., & Han, J. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3469578 · Cited by 204 · see citing works
Wiratunga, N., Abeyratne, R., Jayawardena, L., Martin, K., Massie, S., Nkisi-Orji, I., Weerasinghe, R., Liret, A., & Fleish, B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-63646-2_29 · Cited by 136 · see citing works
Datasets for large language models: a comprehensive survey
Liu, Y., Cao, J., Liu, C., Ding, K., & Jin, L. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11403-7 · Cited by 159 · see citing works
Cheong, I., Xia, K., Feng, K. J. K., Chen, Q. Z., & Zhang, A. X. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3659048 · Cited by 91 · see citing works
The role of large language models in agriculture: harvesting the future with LLM intelligence
Shaikh, T. A., Rasool, T., Veningston, K., & Yaseen, S. M. (2024) · Progress in Artificial Intelligence · doi:10.1007/s13748-024-00359-4 · Cited by 61 · see citing works
Chandra, M., Naik, S., Ford, D., Okoli, E., De Choudhury, M., Ershadi, M., Ramos, G., Hernandez, J., Bhattacharjee, A., Warreth, S., et al. (2025) · Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3715275.3732063 · Cited by 44 · see citing works
Abdollahi, M., Yeganli, S. F., Baharloo, M. M., & Baniasadi, A. (2024) · Electronics · doi:10.3390/electronics14010120 · Cited by 47 · see citing works
Wahidur, R. S. M., Kim, S., Choi, H., Bhatti, D. S., & Lee, H.-N. (2025) · IEEE Access · doi:10.1109/access.2025.3542125 · Cited by 35 · see citing works
From hallucinations to hazards: benchmarking LLMs for hazard analysis in safety-critical systems
Dokas, Ι. M. (2025) · Safety Science · doi:10.1016/j.ssci.2025.107056 · Cited by 30 · see citing works
Large Language Models are legal but they are not: Making the case for a powerful LegalLLM
Jayakumar, T., Farooqui, F., & Farooqui, L. (2023) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2023.nllp-1.22 · Cited by 29 · see citing works
Sociotechnical Implications of Generative Artificial Intelligence for Information Access
Mitra, B., Cramer, H. S. M., & Gurevich, O. (2024) · The information retrieval series · doi:10.1007/978-3-031-73147-1_7 · Cited by 20 · see citing works
Classifying legal interpretations using large language models
Dugac, G., & Altwicker, T. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-025-09447-9 · Cited by 17 · see citing works
Can GPT Alleviate the Burden of Annotation?
Gray, M., Šavelka, J., Oliver, W. M., & Ashley, K. D. (2023) · Frontiers in artificial intelligence and applications · doi:10.3233/faia230961 · Cited by 15 · see citing works
Boosting court judgment prediction and explanation using legal entities
Benedetto, I., Koudounas, A., Vaiani, L., Pastor, E., Cagliero, L., Tarasconi, F., & Baralis, E. (2024) · Artificial Intelligence and Law · doi:10.1007/s10506-024-09397-8 · Cited by 17 · see citing works
Unfair clause detection in terms of service across multiple languages
Galassi, A., Lagioia, F., Jabłonowska, A., & Lippi, M. (2024) · Artificial Intelligence and Law · doi:10.1007/s10506-024-09398-7 · Cited by 14 · see citing works
Prompts and Large Language Models: A New Tool for Drafting, Reviewing and Interpreting Contracts?
Wang, B. T. (2024) · Law Technology and Humans · doi:10.5204/lthj.3483 · Cited by 14 · see citing works
Can large language models generate geospatial code?
Hou, S., Shen, Z., Liang, J., Jiao, H., Zhao, A., Qing, Y., Peng, D., Gui, Z., Guan, X., Xiang, L., et al. (2025) · Geo-spatial Information Science · doi:10.1080/10095020.2025.2535523 · Cited by 17 · see citing works
Kim, J. H., & Kim, H. R. (2024) · Research Square · doi:10.21203/rs.3.rs-4328966/v1 · Cited by 13 · see citing works
Causal Artificial Intelligence in Legal Language Processing: A Systematic Review
Tritto, P. P., & Ponce-Espinosa, H. (2025) · Entropy · doi:10.3390/e27040351 · Cited by 22 · see citing works
Towards AI-Assisted Mapmaking: Assessing the Capabilities of GPT-4o in Cartographic Design
Memduhoğlu, A. (2025) · ISPRS International Journal of Geo-Information · doi:10.3390/ijgi14010035 · Cited by 13 · see citing works
Iyenghar, P. (2025) · Electronics · doi:10.3390/electronics14183624 · Cited by 11 · see citing works
Computational Legal Studies Comes of Age
Chau, B. K., & Livermore, M. A. (2024) · European journal of empirical legal studies · doi:10.62355/ejels.19684 · Cited by 11 · see citing works
Self-citationLeveraging large language models for abstractive summarization of Italian legal news
Benedetto, I., Cagliero, L., Ferro, M., Tarasconi, F., Bernini, C., & Giacalone, G. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-025-09431-3 · Cited by 12 · see citing works
Artificial intelligence and interspecific law
Gervais, D. J., & Nay, J. J. (2023) · Science · doi:10.1126/science.adi8678 · Cited by 10 · see citing works
Self-citationThe Factuality of Large Language Models in the Legal Domain
Hamdani, R. E., Bonald, T., Malliaros, F. D., Holzenberger, N., & Suchanek, F. M. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679961 · Cited by 11 · see citing works
Self-citationBenchmarking Large Language Models on Homework Assessment in Circuit Analysis
Chen, L., Qin, Z., Guo, Y., Rohde, J. A., & Zhang, Y. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00501-w · Cited by 9 · see citing works
tinyBenchmarks: evaluating LLMs with fewer examples
Polo, F. M., Weber, L., Choshen, L., Sun, Y., Xu, G., & Yurochkin, M. (2024) · arXiv · doi:10.48550/arxiv.2402.14992 · Cited by 8 · see citing works
von Schwerin, M., & Reichert, M. (2024) · Future Internet · doi:10.3390/fi16120459 · Cited by 8 · see citing works
Identifying open-texture in regulations using LLMs
Guitton, C., Gubelmann, R., Karray, G., Mayer, S., & Tamò‐Larrieux, A. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-025-09450-0 · Cited by 8 · see citing works
Interacting with AI at Work: Perceptions and Opportunities from the UK Judiciary
Solovey, E., Flanagan, B., & Chen, D. L. (2025) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3729176.3729192 · Cited by 8 · see citing works
WIP: Large Language Model-Enhanced Smart Tutor for Undergraduate Circuit Analysis
Chen, L., Xie, H., Rohde, J. A., & Zhang, Y. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328403 · Cited by 8 · see citing works
Lee, H.-R., & Kim, S. (2024) · Research Square · doi:10.21203/rs.3.rs-4394715/v1 · Cited by 8 · see citing works
The Large Language Model GreekLegalRoBERTa
Saketos, V., Pantazi, D.-A., & Koubarakis, M. (2024) · Hellenic Conference on Artificial Intelligence · doi:10.1145/3688671.3688770 · Cited by 7 · see citing works
Liu, Z., Kou, J., Zhang, W., Gu, C., Fang, X., Huang, Z., Hao, Y., Li, H., Lu, X., Yin, A., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189137 · Cited by 9 · see citing works
A Reusable Prompting Framework for Applying Large Language Models to Legal Tasks
Sriram, S., Vijayaraj, N., Krishna, G. R., Bhanu, S. V., Choudhary, G., & Murugan, T. (2026) · IEEE Access · doi:10.1109/access.2026.3650941 · Cited by 8 · see citing works
LEXam: Benchmarking Legal Reasoning on 340 Law ExamsPreprint
Fan, Y., Ni, J., Merane, J., Salimbeni, E., Tian, Y., Hermstrüwer, Y., Huang, Y., Akhtar, M., Geering, F., Dreyer, O., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5265144 · Cited by 25 · see citing works
Self-citationLegal knowledge infusion for large language models: A survey
Liu, Q., Yu, H., Wang, Q., Xu, Q., Li, J., Zou, Z., Mao, R., & Cambria, E. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103426 · Cited by 7 · see citing works
LENS: Layers of Evaluation of Hallucination in GenAI Systems
Gu, C., Zhang, W., Huang, Z., Kou, J., Liu, Z., Zhao, C., Liu, C., Zhang, L., Lin, W., Wang, Z., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189150 · Cited by 16 · see citing works
Security and Privacy of Large Language Models: Threat Taxonomy, Ethical Implications, and Governance
Pribisalić, M., & Martinčić-Ipšić, S. (2026) · AI · doi:10.3390/ai7050152 · Cited by 9 · see citing works
Nzobonimpa, S., Savard, J., Caron, I., & Lawarée, J. (2024) · AI & Society · doi:10.1007/s00146-024-02103-x · Cited by 6 · see citing works
ElecBench: A large language model benchmark in electric power domain
Zhang, S., Huang, Q., Zhang, Q., Liang, X. M., Liu, W., Gao, K., Zhou, F., & Shi, C. (2025) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2025.112310 · Cited by 6 · see citing works
Never Start from Scratch: Expediting On-Device LLM Personalization via Explainable Model Selection
Wang, H., Yang, B., Yin, X., & Gao, W. (2025) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/3711875.3729132 · Cited by 5 · see citing works
WisWheat: A Three-Tiered Vision-Language Dataset for Wheat Management
Yuan, B., Song, S., Fernández, J. A., Luo, Y., Baktashmotlagh, M., & Huang, Z. H. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3758256 · Cited by 5 · see citing works
DesignRepair: Dual-Stream Design Guideline-Aware Frontend Repair with Large Language Models
Yuan, M., Chen, J., Xing, Z., Quigley, A. J., Luo, Y., Luo, T., Mohammadi, G., Lu, Q., & Zhu, L. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00109 · Cited by 8 · see citing works
The Law and NLP: Bridging Disciplinary Disconnects
Mahari, R., Stammbach, D., Ash, E., & Pentland, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.224 · Cited by 8 · see citing works
GongBu: Easily Fine-tuning LLMs for Domain-specific Adaptation
Zhang, B., Tian, Y., Wang, S., Tu, Z., Chu, D., & Shen, Z. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679233 · Cited by 5 · see citing works
Empower LlaMa 2 for Advanced Logical Reasoning in Natural Language UnderstandingPreprint
Watanabe, N., Kinasaka, K., & Nakamura, A. (2024) · Research Square · doi:10.21203/rs.3.rs-4333417/v1 · Cited by 5 · see citing works
Assis, G., Freitas, C., & Paes, A. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.5814 · Cited by 5 · see citing works
Orchestrating Human-AI Teams: The Manager Agent as aUnifying Research Challenge
Masters, C., Vellanki, A., Shangguan, J., Kultys, B., Gilmore, J., Moore, A. P., & Albrecht, S. V. (2025) · International Conference on Distributed Artificial Intelligence (DAI) · doi:10.1145/3772429.3772439 · Cited by 6 · see citing works
From single-agent to multi-agent: a comprehensive review of LLM-based legal agents
Yang, S., Yang, Z., Liu, Y., & Wang, H. (2025) · AI Agent · doi:10.20517/aiagent.2025.06 · Cited by 4 · see citing works
What is the Best Model? Application-Driven Evaluation for Large Language Models
Lian, S., Zhao, K., Liu, X., Lei, X., Yang, B., Zhang, W., Wang, K., & Liu, Z. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-9437-9_6 · Cited by 4 · see citing works
Prabhakar, P., & Pati, P. B. (2024) · Studies in computational intelligence · doi:10.1007/978-981-97-8460-8_15 · Cited by 4 · see citing works
Sheik, R., Reji, S. A., Sharon, A., Avisha, M., & Nirmala, S. J. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-025-09459-5 · Cited by 5 · see citing works
Capturing Legal Reasoning Paths from Facts to Law in Court Judgments using Knowledge Graphs
Kondo, R., Matsuoka, R., Yoshida, T., Yamasawa, K., & Hisano, R. (2025) · International Conference on Knowledge Capture · doi:10.1145/3731443.3771354 · Cited by 4 · see citing works
Transforming Legal Workflows: A Deep Dive into NLP Solutions for Legal Challenges
Irfan, H., Peerzada, S., & Mansoor, M. (2024) · International Conference on Emerging Technologies · doi:10.1109/icet63392.2024.10935072 · Cited by 3 · see citing works
Legal Hallucinations and the Adoption of Artificial Intelligence in the Judiciary
Herrera-Tapias, B. A., & Guzmán, D. H. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.03.158 · Cited by 7 · see citing works
The European Court of Justice Dataset: Legal Issues and Doctrine
Šadl, U., Brekke, S. A., Zurita, L. L., & Kim, E. H. (2025) · Journal of law & empirical analysis. · doi:10.1177/2755323x251337809 · Cited by 3 · see citing works
Towards a Corpus Methodology for LLMs in the Legal Domain
Athaydes, A., Mota, L. B. B., Neto, F. H. D. A. M., da Silva, S. R., Mane, B., Claro, D. B., Souza, M., & Lisboa, A. B. C. (2025) · Simpósio Brasileiro de Tecnologia da Informação e da Linguagem Humana (STIL) · doi:10.5753/stil.2025.37831 · Cited by 3 · see citing works
Precedent Aware Legal Reasoning System for Turkish Jurisdiction with RAG Supported LLMs
Genç, U., Keskin, Ö. F., Akman, N. P., Berkol, A., & Birant, D. (2025) · International Symposium on Multidisciplinary Studies and Innovative Technologies (ISMSIT) · doi:10.1109/ismsit67332.2025.11267996 · Cited by 4 · see citing works
AI for Statutory Simplification: A Comprehensive State Legal Corpus and Labor Benchmark
Hariri, E., & Ho, D. E. (2025) · International Conference on Artificial Intelligence and Law · doi:10.1145/3769126.3769256 · Cited by 5 · see citing works
Self-citationDolomites : Domain-Specific Long-Form Methodical Tasks
Malaviya, C., Agrawal, P., Ganchev, K., Srinivasan, P., Huot, F., Berant, J., Yatskar, M., Das, D., Lapata, M., & Alberti, C. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00727 · Cited by 6 · see citing works
Wu, X., Jo, S., Zeng, Y., Das, A., Zhang, T., Patel, P. S., Wei, Y., Li, L., Gao, S., Zhang, J., et al. (2024) · IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI ...) · doi:10.1109/bhi62660.2024.10913581 · Cited by 3 · see citing works
Self-citationMhia-Alddin, D. A., & Hussein, A. M. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-4862-9.ch002 · Cited by 3 · see citing works
Unveiling the Potential of Large Language Models
Varghese, N., & Ramasamy, G. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-5288-5.ch015 · Cited by 3 · see citing works
XtremeLLMs: Towards Extremely Large Language ModelsPreprint
Mienye, I. D., Swart, T. G., & Obaido, G. (2024) · Preprints.org · doi:10.20944/preprints202408.1483.v1 · Cited by 3 · see citing works
Conrad, J. G., Ganguly, D., Ghosh, K., & Ghosh, S. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-025-09487-1 · Cited by 3 · see citing works
Yaacoub, J.-P. A., Noura, H., Salman, O., & Pujolle, G. (2025) · Annals of Telecommunications · doi:10.1007/s12243-025-01134-9 · Cited by 2 · see citing works
Automatic Legal Writing Evaluation of LLMs
Pires, R., Malaquias, R. D., & Nogueira, R. (2025) · International Conference on Artificial Intelligence and Law · doi:10.1145/3769126.3769227 · Cited by 7 · see citing works
Using Large Language Models to Automate Annotation and Part-of-Math Tagging of Math Equations
Shan, R., & Youssef, A. S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-66997-2_1 · Cited by 2 · see citing works
Multilingual Scraper of Privacy Policies and Terms of Service
Bernhard, D., Nenadic, L., Bechtold, S., & Kubíček, K. (2025) · doi:10.1145/3709025.3712215 · Cited by 2 · see citing works
Software engineering meets legal texts: LLMs for auto detection of contract smells
Dechtiar, M., Katz, D. M., & Wang, H. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100639 · Cited by 4 · see citing works
Unifying Corroborative and Contributive Attributions in Large Language Models
Worledge, T., Shen, J. H., Meister, N., Winston, C., & Guestrin, C. (2024) · IEEE Conference on Secure and Trustworthy Machine Learning (SaTML) · doi:10.1109/satml59370.2024.00039 · Cited by 2 · see citing works
Designing for the Future of Information Access with Generative Information Retrieval
Murdock, V., Lee, C., & Hersh, W. (2024) · The information retrieval series · doi:10.1007/978-3-031-73147-1_9 · Cited by 1 · see citing works
LLM Non-text-focused Data Generation for Payroll Fraud Detection
Ouesleti, F., ElBedoui, K., Barhoumi, W., & Aissaoui, M. A. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-7238-7_26 · Cited by 1 · see citing works
The Cocoruta Hub: Open and Curated Corpora, Datasets and Language Models on Brazilian Ocean Law
Santo, F. O. D. E., Peres, S. M., Gonçalves, B., Ortega, F. J. M., Matos, V. B., de Lima, A. P., Brandão, A. A. F., & Cozman, F. G. (2025) · Journal of the Brazilian Computer Society · doi:10.5753/jbcs.2025.5791 · Cited by 1 · see citing works
Enhanced Retrieval-Augmented Generation for Chinese Legal Question Answering
Zhang, S., Sun, W., & Wen, Y. (2025) · International Conference on Cloud Computing, Big Data Application and Software Engineering (CBASE) · doi:10.1109/cbase67452.2025.11335575 · Cited by 1 · see citing works
Wang, Z., Li, L., Chen, L., Xu, X., Li, M., & Su, Y. (2025) · International Conference on Wireless Communication, Networking and Internet of Things (WCNIoT) · doi:10.1109/wcniot67424.2025.11381315 · Cited by 1 · see citing works
LLM-assisted formalization for deterministic detection of statutory inconsistency in tax law
Yadamsuren, B., Platt, S. K., & Diaz, M. (2026) · Artificial Intelligence and Law · doi:10.1007/s10506-026-09525-6 · Cited by 1 · see citing works
Faridi, A. R., Masood, F., Keshvi, & Yunus, T. (2026) · Wiley Interdisciplinary Reviews Data Mining and Knowledge Discovery · doi:10.1002/widm.70122 · Cited by 2 · see citing works
DMind Benchmark: Toward a Holistic Assessment of LLM Capabilities across the Web3 Domain
Huang, E., Sun, P., Wang, S., Lin, Z., Chen, A., Hu, K., Ouyang, J., Li, F., Zhang, Z., Wang, H., et al. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817512 · Cited by 1 · see citing works
Legal Judgement Prediction via Contrastive Learning Based-Retrieval and Semantic Embedding
Wei, C., Zhang, Q., Xu, J., Wang, Y., Huang, Z., & Hao, T. (2024) · IEEE International Symposium on Product Compliance Engineering - Asia (ISPCE-ASIA) · doi:10.1109/ispce-asia64773.2024.10756244 · Cited by 1 · see citing works
Toward a Quantum-Inspired Framework for Modelling Legal Rules
Godfrey, N. (2024) · Quantum Economics and Finance · doi:10.1177/29767032241298302 · Cited by 1 · see citing works
Predicting potentially abusive clauses in Chilean terms of services with natural language processing
Löffler, C., Freile, A. M., & Pizarro, T. R. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-025-09462-w · Cited by 3 · see citing works
Trustworthiness Evaluation of Large Language Models
Chen, P., & Liu, S. (2025) · Springer eBooks · doi:10.1007/978-3-031-76770-8_12 · Cited by 2 · see citing works
Economy Watchers Survey Provides Datasets and Tasks for Japanese Financial Domain
Suzuki, M., & Sakaji, H. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715304 · Cited by 1 · see citing works
Legal Mitra: Simplified Legal Resources and Guidance Platform
Chikka, K., Chavan, Y., Gupta, A., Trivedi, M., Jawale, S., & Mhatre, S. (2025) · International Conference for Emerging Technology (INCET) · doi:10.1109/incet64471.2025.11140002 · Cited by 1 · see citing works
PhishBENCH: A Benchmark for Legal Tasks Related to Phishing Attacks
Katoikos, I., Kosmopoulos, D., Iglezakis, I., Kiortsi, P., Thomopoulos, G. A., & Fidas, C. A. (2025) · IEEE International Conference on Tools with Artificial Intelligence (ICTAI) · doi:10.1109/ictai66417.2025.00198
Legal Compliance Evaluation of Smart Contracts Generated by Large Language Models
Wijayakoon, C., Dong, H., Bandara, H. M. N. D., Tari, Z., & Soin, A. (2025) · IEEE International Conference on Blockchain and Cryptocurrency (ICBC) · doi:10.1109/icbc64466.2025.11114661 · Cited by 3 · see citing works
Hagan, M. D. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5146722 · Cited by 1 · see citing works
Self-citationSohail, A., Patel, O. P., Choi, J., Venditti, J. C. S., & Wu, A. J. (2025) · medRxiv · doi:10.1101/2025.07.06.25330972 · Cited by 1 · see citing works
Self-citationAkarajaradwong, P., Lertprasertphakorn, W., Chaksangchaichot, C., & Nutanong, S. (2026) · arXiv · doi:10.48550/arxiv.2605.25652
Legal NLP Meets MiCAR: Advancing the Analysis of Crypto White Papers
Camassa, C. (2023) · Natural Legal Language Processing Workshop (NLLP) · doi:10.18653/v1/2023.nllp-1.14 · Cited by 2 · see citing works
Professor GPT: Having a Large Language Model Write a Commentary on Freedom of AssemblyPreprint
Engel, C., & Kruse, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4994131
Legal Judgment Prediction Through Argument Analysis
Azmi, Mistica, M., Levy, I., & Hovy, E. H. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0348-0_4
The Negotiation Trap An Experiment on a Large Language ModelPreprint
Engel, C. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5076291
- Authors
- Guha, N., Nyarko, J., Ho, D. E., Ré, C., Chilton, A., Narayana, A., Chohlas-Wood, A., Peters, A., Waldon, B., Rockmore, D., Zambrano, D., Talisman, D., Hoque, E., Surani, F., Fagan, F., Sarfaty, G., Dickinson, G. M., Porat, H., Hegland, J., Wu, J., Nudell, J., Niklaus, J., Nay, J., Choi, J. H., Tobia, K., Hagan, M., Ma, M., Livermore, M. A., Rasumov-Rahe, N., Holzenberger, N., Kolt, N., Henderson, P., Rehaag, S., Goel, S., Gao, S., Williams, S., Gandhi, S., Zur, T., Iyer, V., & Li, Z.
- Affiliations
- Stanford University (US)
- University of Chicago (US)
- Georgetown University (US)
- Dartmouth College (US)
- University of California, Santa Barbara (US)
- South Texas College of Law (US)
- University of Toronto (CA)
- University of Nebraska–Lincoln (US)
- Harvard University (US)
- University of Bern (CH)
- New York University (US)
- University of Minnesota (US)
- University of Virginia (US)
- Institut Polytechnique de Paris (FR)
- Princeton University (US)
- York University (CA)
- California Western School of Law (US)
- Indiana University Bloomington (US)
- Corresponding author
- Guha, Neel
- ORCID
- Guha, Neel
- Nyarko, Julian
- Ho, Daniel E.
- Ré, Christopher
- Chilton, Adam
- Chohlas-Wood, Alex
- Waldon, Brandon
- Rockmore, Daniel
- Zambrano, Diego
- Surani, Faiz
- Fagan, Frank
- Sarfaty, Galit
- Dickinson, Gregory M.
- Porat, Haggai
- Wu, Jessica
- Nudell, Joe
- Niklaus, Joel
- Nay, John
- Tobia, Kevin
- Hagan, Margaret
- Ma, Megan
- Livermore, Michael A.
- Holzenberger, Nils
- Henderson, Peter
- Rehaag, Sean
- Goel, Sharad
- Gao, Shang
- Williams, Spencer
- Iyer, Varun
- Li, Zehua
- Venue
- SSRN Electronic Journal
- Publisher
- Elsevier BV
- Published
- 2023
- Volume
- 4583531
- Pages
- 1-143
- ISSN
- 1556-5068 (ISSN-L)
- OpenAlex
- W4389403907
- Cited by
- 154
- References
- 106
- Citations by year
- 2026: 36 · 2025: 74 · 2024: 38 · 2023: 6
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Locations
- SSRN Electronic Journal (acceptedVersion)
- HAL (submittedVersion)
- bepress Legal Repository (submittedVersion)
- Topics
- Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Law › Legal Education and Practice Innovations
- Keywords
- legal reasoning, large language models, benchmark design, lawyers, LLM evaluation, interdisciplinary collaboration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review