On the Role of Negative Precedent in Legal Outcome Prediction
Open accessAbstract
Every legal case sets a precedent by developing the law in one of the following two ways. It either expands its scope, in which case it sets positive precedent, or it narrows it, in which case it sets negative precedent. Legal outcome prediction, the prediction of positive outcome, is an increasingly popular task in AI. In contrast, we turn our focus to negative outcomes here, and introduce a new task of negative outcome prediction. We discover an asymmetry in existing models’ ability to predict positive and negative outcomes. Where the state-of-the-art outcome prediction model we used predicts positive outcomes at 75.06 F1, it predicts negative outcomes at only 10.09 F1, worse than a random baseline. To address this performance gap, we develop two new models inspired by the dynamics of a court process. Our first model significantly improves positive outcome prediction score to 77.15 F1 and our second model more than doubles the negative outcome prediction performance to 24.01 F1. Despite this improvement, shifting focus to negative outcomes reveals that there is still much room for improvement for outcome prediction models. https://github.com/valvoda/Negative-Precedent-in-Legal-Outcome-Prediction
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Flawed legal counseling can severely infringe on rights, livelihoods, and liberties.
“While advances have been made, rigorous examination of risks is needed given that flawed legal counseling can severely infringe on rights, livelihoods, and liberties [6, 44, 85].”
From (A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advice (2024) · cites this paper as [85]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,881 · see citing works
Advances in Neural Information Processing Systems 14
(2002) · The MIT Press eBooks · doi:10.7551/mitpress/1120.001.0001 · Cited by 9,184 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
Stanford Encyclopedia of Philosophy
(2015) · Choice Reviews Online · doi:10.5860/choice.187763 · Cited by 5,364 · see citing works
LEGAL-BERT: The Muppets straight out of Law School
Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Αλέτρας, Ν., & Androutsopoulos, I. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.261 · Cited by 1,197 · see citing works
Αλέτρας, Ν., Tsarapatsanis, D., Preoțiuc-Pietro, D., & Lampos, V. (2016) · PeerJ Computer Science · doi:10.7717/peerj-cs.93 · Cited by 813 · see citing works
Legal Judgment Prediction via Topological Learning
Zhong, H., Guo, Z., Tu, C., Xiao, C., Liu, Z., & Sun, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1390 · Cited by 369 · see citing works
Neural Legal Judgment Prediction in English
Chalkidis, I., Androutsopoulos, I., & Aletras, N. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1424 · Cited by 352 · see citing works
The Nature and Authority of Precedent
Duxbury, N. (2008) · Cambridge University Press eBooks · doi:10.1017/cbo9780511818684 · Cited by 328 · see citing works
Kort, F. (1957) · American Political Science Review · doi:10.2307/1951767 · Cited by 211 · see citing works
CAIL2018: A Large-Scale Legal Dataset for Judgment PredictionPreprint
Xiao, C., Zhong, H., Guo, Z., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.02478 · Cited by 189 · see citing works
Distinguish Confusing Law Articles for Legal Judgment Prediction
Xu, N., Wang, P., Chen, L. F., Pan, L., Wang, X., & Zhao, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.280 · Cited by 186 · see citing works
Extractive summarisation of legal texts
Hachey, B., & Grover, C. (2007) · Artificial Intelligence and Law · doi:10.1007/s10506-007-9039-z · Cited by 153 · see citing works
Algorithms in the Criminal Justice System: Assessing the Use of Risk Assessments in Sentencing
Kehl, D. L., & Kessler, S. A. (2017) · Digital Access to Scholarship at Harvard (DASH) (Harvard University) · Cited by 131 · see citing works
LexGLUE: A Benchmark Dataset for Legal Language Understanding in English
Chalkidis, I., Jana, A., Hartung, D., Bommarito, M. J., Androutsopoulos, I., Katz, D. M., & Aletras, N. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3936759 · Cited by 111 · see citing works
Chalkidis, I., Fergadiotis, M., Tsarapatsanis, D., Aletras, N., Androutsopoulos, I., & Malakasiotis, P. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.22 · Cited by 78 · see citing works
Legal NERC with ontologies, Wikipedia and curriculum learning
Cardellino, C., Teruel, M., Alemany, L. A., & Villata, S. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-2041 · Cited by 37 · see citing works
NLP for Shallow Question Answering of Legal Documents Using Graphs
Monroy, A. L., Calvo, H., & Gelbukh, A. F. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-00382-0_40 · Cited by 26 · see citing works
The Unreasonable Effectiveness of the Baseline: Discussing SVMs in Legal Text Classification
Clavié, B., & Alphonsus, M. (2021) · Frontiers in artificial intelligence and applications · doi:10.3233/faia210317 · Cited by 16 · see citing works
Legal Knowledge and Information Systems
(2021) · Frontiers in artificial intelligence and applications · doi:10.3233/faia346 · Cited by 15 · see citing works
What Is Law for the European Court of Human RightsPreprint
Dzehtsiarou, K. (2018) · SSRN Electronic Journal · Cited by 12 · see citing works
What About the Precedent: An Information-Theoretic Analysis of Common Law
Valvoda, J., Pimentel, T., Stoehr, N., Cotterell, R., & Teufel, S. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.181 · Cited by 10 · see citing works
Self-citationThe Role of Precedent at the European Court of Human Rights: A Network Analysis of Case Citations
Lupu, Y., & Voeten, E. (2010) · OpenSIUC (Southern Illinois University Carbondale) · Cited by 5 · see citing works
From Case Law to Ratio Decidendi
Valvoda, J., & Ray, O. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-93794-6_2 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 5
- Kevin D. Ashley. 1988. Modelling Legal Argu- ment: Reasoning with Cases and Hypotheticals. Ph.D. thesis, USA. Order No: GAX88-13198.
- Henry Black. 2019. Black's Law Dictionary, 11th edition. Thomson Reuters.
- Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Pro- ceedings of the 3rd International Conference on Learning Representations (ICLR).
- Reed C. Lawlor. 1963. What computers can do: Analysis and prediction of judicial de- cisions. American Bar Association Journal, 49(4):337-344.
- Stuart S. Nagel. 1963. Applying correlation anal- ysis to case prediction. Texas Law Review, 42:1006.
Cited by
Sorted by citations
Cheong, I., Xia, K., Feng, K. J. K., Chen, Q. Z., & Zhang, A. X. (2024) · The 2024 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3630106.3659048 · Cited by 91 · see citing works
- Authors
- Valvoda, J., Cotterell, R., & Teufel, S.
- Affiliations
- University of Cambridge (GB)
- ETH Zurich (CH)
- Corresponding author
- Valvoda, Josef
- Cotterell, Ryan
- Teufel, Simone
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2023
- Volume
- 11
- Pages
- 34-48
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W4317898791
- Cited by
- 19
- References
- 25
- Citations by year
- 2026: 2 · 2025: 5 · 2024: 7 · 2023: 5
- FWCI
- 18.68
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Political Science and International Relations › Artificial Intelligence in Law
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- judicial decision prediction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
