Legalbench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

Guha, Neel, Nyarko, Julian, Ho, Daniel E., Ré, Christopher, Chilton, Adam, Narayana, Aditya, Chohlas-Wood, Alex, Peters, Austin +32 more·2023·SSRN Electronic Journal
DOI 10.2139/ssrn.4583531

Cited by (156)

Items 101–156 of 156

  1. SMOOTHIE: Label Free Language Model Routing
    Neel Guha, Mayee F. Chen, Trevor Chow, Ishan S. Khare, Christopher Ré · 2025 · Qeios
  2. RTPbench: A Natural Language to Temporal Logic Translation Benchmark for Multi Robot Heterogeneous Task Planning
    Baiheng Cao, Yaonan Wang, Yijia Zhang, Bo Ouyang · 2024 · IEEE International Conference on Unmanned Systems (ICUS)
  3. Evidence of conceptual mastery in the application of rules by Large Language Models
    José Luiz Nunes, Guilherme Almeida, Brian Flanagan · 2025 · SSRN Electronic Journal
  4. AI and the Future of Disputing: Naming, Blaming, Claiming, and Preventing
    Ethan Katsh, Talia Schwartz‐Maor, Orna Rabinovich‐Einy · 2025 · Conflict Resolution Quarterly
  5. The Medium Is Not the Message: Deconfounding Text Embeddings via Linear Concept Erasure
    Yu Fan, Yang Tian, Shauli Ravfogel, Mrinmaya Sachan, Elliott Ash, Alexander Hoyle · 2025 · SSRN Electronic Journal
  6. A Unified Retrieval Framework with Document Ranking and EDU Filtering for Multi-document Summarization
    Shiyin Tan, Jaeeon Park, Dongyuan Li, Renhe Jiang, Manabu Okumura · 2025 · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)
  7. Multi-agent Interaction
    Chung-Chi Chen, Hiroya Takamura · 2025 · SpringerBriefs in intelligent systems
  8. Technical and inspirational philosophy of law: Lawyers' reasoning and artificial intelligence
    Bojan Spaić · 2025 · Glasnik Advokatske komore Vojvodine
  9. Transformer-Based Extraction of Statutory Definitions from the U.S. Code
    Arpana Hosabettu, Harsh Shah · 2025 · IEEE World AI IoT Congress (AIIoT)
  10. The Puzzle of Anemic “Legal Tech” and the Future of Legal Services
    David Freeman Engstrom, Jiahui Lu · 2025 · Cambridge University Press eBooks
  11. Lessons from the Field
    David Freeman Engstrom, David Freeman Engstrom · 2025 · Cambridge University Press eBooks
  12. Coding Anti-Discrimination Jurisprudence: A Hybrid Computational Model of the Arlington Heights Test
    Kevin Scott Jobe · 2025 · Journal of Social Computing
  13. Evaluating the Ethical Judgment of Large Language Models in Financial Market Abuse Cases
    Avinash Kumar Pandey, Swati Rajwal · 2025 · ACM International Conference on AI in Finance (ICAIF)
  14. Long Math Reasoning Problem Generation
    Changwei Li, Guangping Huang, Zihao Zhou, Qiufeng Wang · 2025 · Lecture notes in computer science
  15. Resolving Legal Ambiguities for Safe MASS Navigation: A Socio-Technical Approach Using Human-Machine Learning
    Shubhi Verma, Dany Varghese, Alfie Anthony Treloar, Alan J. Hunter, Alireza Tamaddoni‐Nezhad · 2025
  16. Legal Retrieval Augmented Generation with Structured Retrieval and Iterative Refinement
    Chaitanya Dhananjay Jadhav, Chang Liu, Jun Zhao · 2025 · Conference on Privacy, Security and Trust
  17. LegisFlow: Enhancing Korean Legal Research with Temporal-Aware LLM Interfaces
    Junghwan Kim, Hyeonseok Jeon, Dongseok Heo, Jung Lee, Bongwon Suh · 2025 · ACM Symposium on User Interface Software and Technology (UIST)
  18. On the Diminishing Returns of Complex Robust RAG Training in the Era of Powerful LLMs
    Hanxing Ding, Shuchang Tao, Liang Pang, Zihao Wei, Liwei Chen, Kun Xu, Huawei Shen, Xueqi Cheng · 2025 · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP)
  19. Reducing Drafting Backlogs with LLMs: A Feasibility Study on Thai Legal Petitions
    Sathana Laolugsanalerd, Chuenkamon Wirunhasangsit, Jamnean Kongthavorn, Natthawut Kertkeidkachorn, Peerapon Vateekul · 2025 · International Joint Symposium on Artificial Intelligence and Natural Language Processing (iSAI-NLP)
  20. Identifying Legal Holdings with LLMs: A Systematic Study of Performance, Scale, and Memorization
    Chuck Arvin · 2025 · International Conference on Artificial Intelligence and Law
  21. Can Large Language Models Reliably Extract Jurisdictional Variations? An Empirical Study on UK Statutory Texts
    Safia Kanwal, Livio Robaldo, Hafsa Shareef Dar, Davide Liga, Joseph Anim · 2025 · International Conference on Frontiers of Information Technology (FIT)
  22. Human-Machine Legal Preference Benchmarking: An Experimental Pilot Evaluating the Comparative Performance of LLM Models, Enterprise Tools, and Humans on Legal Tasks
    Megan Ma, Riya Ranjan · 2025 · International Conference on Artificial Intelligence and Law
  23. Non-Herculean Data
    Cox C · 2025 · International Conference on Artificial Intelligence and Law
  24. Large language models in interpreting legal documents
    Simone Corbo · 2026 · Elsevier eBooks
  25. CAN LARGE LANGUAGE MODELS ACT AS “CO-AUDITORS”?
    Hakan Emekci · 2026 · Denetişim
  26. Document Haystack: A Long Context Multimodal Image/Document Understanding Vision LLM Benchmark
    Goeric Huybrechts, Srikanth Ronanki, Sai Muralidhar Jayanthi, Jack FitzGerald, Srinivasan Veeravanallur · 2025 · IEEE International Conference on Computer Vision Workshops
  27. A Preliminary Analysis of the Current Limitation and Future Directions of AI Applied to the Legal Domain Based on a SLR
    Andrei Mikriukov, Yaroslav Plaksin, Alessandro Ravveduto, Mariia Snigireva, Giancarlo Succi, Alexander G. Tormasov, Екатерина Трофимова · 2025 · International Conference on Data Science and Intelligent Systems (DSIS)
  28. LLM-as-a-judge is bad, based on AI attempting the exam qualifying for the member of the Polish National Board of Appeal
    Michał Karp, Anna Kubaszewska, Magdalena Król, Robert Król, Witold Wydmański, Aleksander Smywiński-Pohl, Mateusz Szymański · 2026 · Artificial Intelligence and Law
  29. Çok Modlu Yapay Zekâda Görsel Dil Modelleri: Mimari Temeller ve Sektörel Uygulamalar
    İrem Cakcak, Burhan Duman · 2026 · Özgür Yayınları eBooks
  30. MPCCD-MLF: A dataset of multi-round professional consultation conversations in the medical, legal and financial domains
    Cong Luo, Qidong Yan, Dejie Wang, Ning Ma · 2026 · China Scientific Data
  31. FairAgent: A Collaborative Multi-Agent System for Fair Competition Review
    Yuanqing Mao, Jinfei Ye, Cheng Yang, Chuncong Wang, Qiyu Chen, Yang Xu, Min Zhu, Hanrui Chen, Jiong Lin, Beining Wu et al. · 2026 · Electronics
  32. A Hybrid Multi-Agent System for Early Scam Detection in Crypto-Assets
    Mario Trerotola, Mimmo Parente, Davide Calvaresi · 2026 · Applied Sciences
  33. Grading Machines: Can AI Exam-Grading Replace Law Professors?
    Kevin L. Cope, Jens Frankenreiter, Scott Hirst, Eric A. Posner, Daniel B. Schwarcz, Dane Thorley · 2026 · Journal of law & empirical analysis.
  34. Building Benchmarks from the Ground Up: Community-Centered Evaluation of LLMs in Healthcare Chatbot Settings
    Hamna Hamna, Gayatri Bhat, Sourabrata Mukherjee, Faisal M. Lalani, Evan Hadfield, Divya Siddarth, Kalika Bali, Sunayana Sitaram · 2026 · CHI Conference on Human Factors in Computing Systems
  35. DraftNEPABench: A Benchmark for Drafting NEPA Document Sections with Coding Agents
    Anurag Acharya, Bishal Lakha, Rounak Meyur, Rohan Nuttall, Sarthak Chaturvedi, Anika Halappanavar, Leah Hare, Lin Zeng, Mike Parker, Sai Munikoti et al. · 2026 · ACM Conference on AI and Agentic Systems (CAIS)
  36. LegalKit: A Modular Toolkit for Efficient Legal AI Evaluation
    Shuo Miao, Haitao Li, Yueyue Wu, Qingyao Ai, Yiqun Liu · 2026 · International World Wide Web Conference (WWW)
  37. Beyond Storage: Toward Living Memory Systems for LLMs on the Edge
    Myles Foley, Valentin Radu, Massimo Pegoraro, Maryam Vahdat Pour, Frank Faricy, Mark Girolami · 2026 · International Conference on Mobile Systems, Applications, and Services
  38. From Legal Text to NP-Complete Decision Models: MPNet Retrieval and Policy Information Extraction
    Айгерим Айтим, Anel Auyezova, Б.К. Синчев, Aksulu Mukhanova · 2026 · Machine Learning and Knowledge Extraction
  39. Legal-TREND: Uncertainty Trend Modeling with Domain-Aware Retrieval for Legal Question Answering
    Shi Shu, Jianing Zhong, Jiahao Fang, Jiaqi Li · 2026 · International Seminar on Artificial Intelligence, Networking and Information Technology (AINIT)
  40. Version-Aware Knowledge-Graph-Enhanced Retrieval for Multi-Version Legal Question Answering
    晓宇 陈 · 2026 · Computer Science and Application
  41. CollectivIA: Two-Pipeline Multilingual Legal RAG for Moroccan Territorial Governance with LLM-Assisted and Regex-Based Chunking
    Firiel Zouak, Omar El Beqqali, Jamal Riffi · 2026 · Big Data and Cognitive Computing
  42. HIVE: Understanding Post-hallucination Reasoning in Vision Language Models
    Feng He, Zhenting Wang, Qifan Wang, Qiang Guan, Dongfang Liu, Ruixiang Tang, Qiankun Li · 2026 · Lecture notes in computer science
  43. HAR-SCP: Fine-Grained Advertising Law Auditing from Platform URLs with Multimodal Evidence Centralization and Statutory Constraint Propagation
    Huayang Hsu, Yuesheng Zhu, Guibo Luo, Hongye Liu · 2026 · Lecture notes in computer science
  44. Logic-based interpretability and analysis of neural legal AI models
    Yiping Tang, Rui Shao, Lingyan Yang, Yifei Qiao, Fang Wang · 2026 · Artificial Intelligence and Law
  45. Compliance-to-Code : Enhancing Financial Compliance Checking via Code Generation
    Siyuan Li, Jian Chen, Rui Yao, Xuming Hu, Peilin Zhou, Weihua Qiu, Simin Zhang, Chucheng Dong, Zhiyao Li, Qipeng Xie et al. · 2026 · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)
  46. Executable explanation traces for legal LLM predictions via retrieval-augmented codification
    Haoyang Chen, Kumiko Tanaka‐Ishii · 2026 · Frontiers in Artificial Intelligence
  47. Leveraging LLM agents and knowledge models for automated legal reasoning
    Vladia Celia Monteiro Pinheiro, Francisco C J Bonfim, S. L Silva, Alicia Neves, Henrique Santos, João A. Monteiro Neto, Jorge L B Araujo, Rilder S. Pires, Ricardo Alexandre da Silva Costa, Vasco Furtado · 2026 · AI Magazine
  48. Abordagem LGRAG-IoT: Contribuição para a Conformidade do Middleware EXEHDA com a Lei Geral de Proteção de Dados
    Rogério da Costa Albandes, Rafael Azevedo, Renata Abib, Anderson Passos, Ana Marilza Pernas, Adenauer Yamin · 2026
  49. Evaluation of Vanilla and RAG-Supported LLM Comprehension of the EU AI Act
    Eleni Tsalera, Gerasimos K. Pagiatakis, Andreas E. Papadakis · 2026 · Future Internet
  50. Technological Symmetry and Tax Assistance Through Artificial Intelligence
    Carlos Aldair Orta de la Rosa, Alejandro Javier Juarez Bolaños · 2026 · Communications in computer and information science
  51. JusDef: Defeasible Message Passing for Exception-Aware Legal Document Classification
    Awais Abdul Khaliq, Stefano Montanelli, Hercules Dalianis, Nuzhat Naqvi · 2026 · Lecture notes in computer science
  52. Towards Reliable Evaluation of Legal QA Systems Using LLM-as-a-Judge
    Agris Šostaks, Sergejs Rikačovs, Artūrs Sproģis, Andris Hohbergs, Uldis Lavrinovičs · 2026 · Lecture notes in business information processing
  53. Retrieval-Augmented Generation for Domain-Specific Assistance: a Legal Case Study
    Ritesh Sharma, Aarti M. Karande, Pranita Aware, Shardul Bhagade, Ritesh Sharma · 2026 · i-manager s Journal on Computer Science
  54. A Demand-Side Benchmark for Consumer-Facing Construction Cost Questions: Price-Figure Span, Output Consistency, and the Case for a Verifiable Reference Layer
    Toshikatsu Oga · 2026 · Intelligent infrastructure and construction
  55. A Semantic Resource Suite for Privacy Policy Formalization
    Rui Zhao, Vladyslav Melnychuk, Jesse Wright, Jun Zhao, Nigel Shadbolt · 2026 · Lecture notes in computer science
  56. The Chunking Crisis in Hierarchical Structural Law: Bridging the Ontological Gap in Legal RAG via Deterministic Ontology-Driven Ingestion
    Youssef Al Mouatamid, Marie Bonnin, Jihad Zahir · 2026 · Applied Sciences