Modeling Legal Reasoning: LM Annotation at the Edge of Human Agreement
Open accessAbstract
Generative language models (LMs) are increasingly used for document class-prediction tasks and promise enormous improvements in cost and efficiency.Existing research often examines simple classification tasks, but the capability of LMs to classify on complex or specialized tasks is less well understood.We consider a highly complex task that is challenging even for humans: the classification of legal reasoning according to jurisprudential philosophy.Using a novel dataset of historical United States Supreme Court opinions annotated by a team of domain experts, we systematically test the performance of a variety of LMs.We find that generative models perform poorly when given instructions (i.e.prompts) equal to the instructions presented to human annotators through our codebook.Our strongest results derive from fine-tuning models on the annotated dataset; the best performing model is an in-domain model, LEGAL-BERT.We apply predictions from this fine-tuned model to study historical trends in jurisprudence, an exercise that both aligns with prominent qualitative historical accounts and points to areas of possible refinement in those accounts.Our findings generally sound a note of caution in the use of generative LMs on complex tasks without finetuning and point to the continued relevance of human annotation-intensive classification methods.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Llama 2: Open Foundation and Fine-Tuned Chat Models
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2307.09288 · Cited by 2,604 · see citing works
Large Language Models are Zero-Shot Reasoners
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · arXiv · doi:10.48550/arxiv.2205.11916 · Cited by 2,013 · see citing works
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · see citing works
Scaling Instruction-Finetuned Language Models
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2210.11416 · Cited by 1,441 · see citing works
LEGAL-BERT: The Muppets straight out of Law School
Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Αλέτρας, Ν., & Androutsopoulos, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.261 · Cited by 1,197 · see citing works
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197 · see citing works
Llewellyn, K. N. (1950) · Vanderbilt law review · Cited by 186 · see citing works
Törnberg, P. (2023) · arXiv · doi:10.48550/arxiv.2304.06588 · Cited by 152 · see citing works
The Transformation of American Law, 1870-1960: The Crisis of Legal Orthodoxy.
Friedman, L. M., & Horwitz, M. J. (1993) · Contemporary Sociology A Journal of Reviews · doi:10.2307/2074484 · Cited by 128 · see citing works
The Common Law Tradition--Deciding Appeals
Westwood, H., & Llewellyn, K. N. (1961) · Columbia Law Review · doi:10.2307/1120104 · Cited by 56 · see citing works
Measuring the Fortress: Explaining Trends in Supreme Court and Circuit Court Dictionary Use
Calhoun, J. W. (2014) · bepress Legal Repository · Cited by 38 · see citing works
AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
He, X., Lin, Z., Gong, Y., Jin, A.-L., Zhang, H., Lin, C., Jiao, J., Yiu, S. M., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2303.16854 · Cited by 34 · see citing works
How to Use Large Language Models for Empirical Legal Research
Choi, J. H. (2024) · Journal of Institutional and Theoretical Economics JITE · doi:10.1628/jite-2024-0006 · Cited by 23 · see citing works
An Empirical Study of Statutory Interpretation in Tax LawPreprint
Choi, J. H. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3460962 · Cited by 16 · see citing works
Language Model Interpretability and Empirical Legal Studies
Livermore, M. A., Herron, F., & Rockmore, D. N. (2024) · Journal of Institutional and Theoretical Economics · doi:10.1628/jite-2024-0009 · Cited by 11 · see citing works
HISTORICAL TRENDS IN MACRO-JURISPRUDENCE: A LANGUAGE MODEL ASSESSMENT, 1870-2023Preprint
Stiglitz, E. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4886534 · Cited by 3 · see citing works
Self-citationBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
1 of the 21 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Aaron-Andrew Bruhl. 2018. Statutory interpretation and the rest of the iceberg: Divergences between the lower federal courts and the supreme court. Duke Law Journal, 68(1):1-73.
- William N. Eskridge. 1990. The new textualism. University of California Los Angeles Law Review, 37(4):621-692.
- Gilbert Gilmore. 2014. The Ages of American Law. Yale University Press, New Haven.
- Nancy Staudt, Lee Epstein, Peter Wiedenbeck, Rene Lindstiidt, and Ryan J. Vander Wielen. 2005. Judging statutes: Interpretive regimes. Loyola Law Review, 38(5):1909-1970.
Cited by
Sorted by citations
Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts
Halterman, A., & Keith, K. A. (2025) · Political Analysis · doi:10.1017/pan.2025.10017 · Cited by 44 · see citing works
Laboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settings
Wolfe, R., Slaughter, I., Han, B., Wen, B., Yang, Y., Rosenblatt, L., Herman, B., Brown, E., Qu, Z., Weber, N., et al. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658966 · Cited by 17 · see citing works
LEXam: Benchmarking Legal Reasoning on 340 Law ExamsPreprint
Fan, Y., Ni, J., Merane, J., Salimbeni, E., Tian, Y., Hermstrüwer, Y., Huang, Y., Akhtar, M., Geering, F., Dreyer, O., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5265144 · Cited by 25 · see citing works
Lawyers Should Not Trust AI: A call for an Open-source Legal Language ModelPreprint
Dahan, S., Bhambhoria, R. V., Liang, D., & Zhu, X. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4587092 · Cited by 4 · see citing works
Alyahya, G., & Aldayel, A. (2025) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v19i1.35803 · Cited by 3 · see citing works
Classifying proportionality - identification of a legal argument
Lüders, K., & Stohlmann, B. (2024) · Artificial Intelligence and Law · doi:10.1007/s10506-024-09415-9 · Cited by 2 · see citing works
Judge AI: A Case-Study of Large Language Models as Judges
Posner, E. A., & Saran, S. (2026) · Journal of law & empirical analysis. · doi:10.1177/2755323x261433614 · Cited by 20 · see citing works
Mining legal arguments to study judicial formalism
Koref, T., Held, L., Namazov, M., Thlija, Y., Kumru, H., & Habernal, I. (2026) · Artificial Intelligence and Law · doi:10.1007/s10506-026-09524-7 · Cited by 2 · see citing works
Learning How to Use Large Language Models for Empirical Legal Research
Stiglitz, E. H. (2024) · Journal of Institutional and Theoretical Economics JITE · doi:10.1628/jite-2024-0008 · Cited by 1 · see citing works
Self-citationJudicial self fashioning: Rhetorical performance in Supreme Court opinions
Thalken, R., Mimno, D., & Wilkens, M. J. (2024) · Discourse Studies · doi:10.1177/14614456241281142
Self-citationThalken, R., & Stiglitz, E. H. (2026) · Journal of Law and Courts · doi:10.1017/jlc.2025.10012
Self-citationAI-Assisted Moot Courts: Simulating Justice-Specific Questioning in Oral Arguments
Zhang, K., Nadeem, N., Zheng, L., Stammbach, D., & Henderson, P. (2026) · ACM Symposium on Computer Science and Law (CSLAW) · doi:10.1145/3788646.3789536
Kara, A., & Yıldırım, S. (2026) · Sakarya University Journal of Computer and Information Sciences · doi:10.35377/saucis...1835608
- Authors
- Thalken, R., Stiglitz, E., Mimno, D., & Wilkens, M.
- Affiliations
- Cornell University (US)
- Venue
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Published
- 2023
- Pages
- 9252-9265
- OpenAlex
- W4389520790
- Cited by
- 13
- References
- 21
- Citations by year
- 2026: 5 · 2025: 3 · 2024: 4 · 2023: 1
- FWCI
- 17.65
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1652536, 2019313)
- Topics
- Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- generative language models, legal annotations, fine-tuning, Legal-BERT, document classification
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
