Evaluating document representations for content-based legal literature recommendations
The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Abstract
Recommender systems assist legal professionals in finding relevant literature for supporting their case. Despite its importance for the profession, legal applications do not reflect the latest advances in recommender systems and representation learning research. Simultaneously, legal recommender systems are typically evaluated in small-scale user study without any public available benchmark datasets. Thus, these studies have limited reproducibility. To address the gap between research and practice, we explore a set of state-of-the-art document representation methods for the task of retrieving semantically related US case law. We evaluate text-based (e.g., fast-Text, Transformers), citation-based (e.g., DeepWalk, Poincaré), and hybrid methods. We compare in total 27 methods using two silver standards with annotations for 2,964 documents. The silver standards are newly created from Open Case Book and Wikisource and can be reused under an open license facilitating reproducibility. Our experiments show that document representations from averaged fastText word vectors (trained on legal corpora) yield the best results, closely followed by Poincaré citation embeddings. Combining fastText and Poincaré in a hybrid manner further improves the overall result. Besides the overall performance, we analyze the methods depending on document length, citation count, and the coverage of their recommendations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“recent work has utilized pretrained models for the recommendation of relevant citations and holding statements when writing legal texts [Zheng et al. 2021; Huang et al. 2021b; Ostendorff et al. 2021].”
From On the Opportunities and Risks of Foundation Models (2021) · cites this paper as Ostendorff et al. 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Bag of Tricks for Efficient Text Classification
Joulin, A., Grave, E., Bojanowski, P., & Mikolov, T. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers · doi:10.18653/v1/e17-2068 · Cited by 4,545
Signature Verification Using A "Siamese" Time Delay Neural Network
BROMLEY, J., BENTZ, J. W., BOTTOU, L., GUYON, I., LECUN, Y., MOORE, C., SÄCKINGER, E., & SHAH, R. (1993) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001493000339 · Cited by 2,077
SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation
Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., & Specia, L. (2017) · Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) · doi:10.18653/v1/s17-2001 · Cited by 1,562 · cited 1×
“SNLI and MNLI dataset [11, 54] or fine-tuned on the STS benchmark [13].”
Hyperbolic geometry of complex networks
Krioukov, D., Papadopoulos, F., Kitsak, M., Vahdat, A., & Boguñá, M. (2010) · Physical Review E · doi:10.1103/physreve.82.036106 · Cited by 1,059
Poincaré Embeddings for Learning Hierarchical Representations
Nickel, M., & Kiela, D. (2017) · arXiv · doi:10.48550/arxiv.1705.08039 · Cited by 667
Research works printed but not linked: 43
- Sanjeev Arora , Yingyu Liang , and Tengyu Ma . 2017 . A simple but though Baseline for Sentence Embeddings . In 5th International Conference on Learning Representations (ICLR 2017), Vol. 15 . 416--424. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but though Baseline for Sentence Embeddings. In 5th International Conference on Learning Representations (ICLR 2017), Vol. 15. 416--424.
- 10.2139/ssrn.3204926 · DOI
- 10.1109/ACCESS.2018.2890388 · DOI · Paper page
- Joeran Beel , Corinna Breitinger , Stefan Langer , Andreas Lommatzsch , and Bela Gipp . 2016. Towards reproducibility in recommender-systems research. User Modeling and User-Adapted Interaction (UMAI) 26 ( 2016 ). Joeran Beel, Corinna Breitinger, Stefan Langer, Andreas Lommatzsch, and Bela Gipp. 2016. Towards reproducibility in recommender-systems research. User Modeling and User-Adapted Interaction (UMAI) 26 (2016).
- Iz Beltagy , Matthew E. Peters , and Arman Cohan . 2020 . Longformer: The Long-Document Transformer. (2020). arXiv:2004.05150 Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long-Document Transformer. (2020). arXiv:2004.05150 · arXiv
- Paheli Bhattacharya Kripabandhu Ghosh Arindam Pal and Saptarshi Ghosh. 2020. Methods for Computing Legal Document Similarity: A Comparative Study. (2020). arXiv:2004.12307 Paheli Bhattacharya Kripabandhu Ghosh Arindam Pal and Saptarshi Ghosh. 2020. Methods for Computing Legal Document Similarity: A Comparative Study. (2020). arXiv:2004.12307 · arXiv
- David M Blei , Andrew Y Ng , and Michael I Jordan . 2003 . Latent dirichlet allocation . Journal of machine Learning research 3 , Jan (2003), 993 -- 1022 . David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research 3, Jan (2003), 993--1022.
- 10.1007/s10506-016-9184-3 · DOI
- 10.3233/978-1-61499-726-9-131 · doi:10.3233/978-1-61499-726-9-131
- 10.1162/tacl_a_00051 · DOI · Paper page
- Samuel R. Bowman , Gabor Angeli , Christopher Potts , and Christopher D. Manning . 2015. A large annotated corpus for learning natural language inference . Proc. of EMNLP ( 2015 ), 632--642. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. Proc. of EMNLP (2015), 632--642.
- 10.18653/v1/2020.findings-emnlp.261 · DOI · Paper page
- Mouzhi Ge , Carla Delgado-Battenfeld , and Dietmar Jannach . 2010. Beyond accuracy: evaluating recommender systems by coverage and serendipity . In Proc. of RecSys '10 . ACM Press , New York , New York, USA, 257. Mouzhi Ge, Carla Delgado-Battenfeld, and Dietmar Jannach. 2010. Beyond accuracy: evaluating recommender systems by coverage and serendipity. In Proc. of RecSys '10. ACM Press, New York, New York, USA, 257.
- 10.1145/2939672.2939754 · DOI · Paper page
- Nils Holzenberger , Andrew Blair-Stanek , and Benjamin Van Durme . 2020 . A dataset for statutory reasoning in tax law entailment and question answering . In Proc. of the 2020 Natural Legal Language Processing Workshop. 31--38 . Nils Holzenberger, Andrew Blair-Stanek, and Benjamin Van Durme. 2020. A dataset for statutory reasoning in tax law entailment and question answering. In Proc. of the 2020 Natural Legal Language Processing Workshop. 31--38.
- 10.1111/j.1469-8137.1912.tb05611.x · DOI · Paper page
- 10.1145/1980422.1980439 · DOI
- 10.18653/v1/w16-1609 · DOI · Paper page
- Quoc V. Le and Tomas Mikolov. 2014 . Distributed Representations of Sentences and Documents. Int. Conf. on Machine Learning 32 ( 2014 ), 1188--1196. Quoc V. Le and Tomas Mikolov. 2014. Distributed Representations of Sentences and Documents. Int. Conf. on Machine Learning 32 (2014), 1188--1196.
- 10.1145/3341161.3342864 · DOI
- Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. (2019). arXiv:1907.11692 Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. (2019). arXiv:1907.11692 · arXiv
- 10.1145/3140107.3140119 · doi:10.1145/3140107.3140119
- 10.1017/CBO9780511809071 · DOI · Paper page
- David Mellinkoff . 1963. The language of the law . Boston : Little Brown and Company ( 1963 ). David Mellinkoff. 1963. The language of the law. Boston: Little Brown and Company (1963).
- Tomas Mikolov Kai Chen Greg Corrado and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. (2013) 1--12. arXiv:1301.3781 Tomas Mikolov Kai Chen Greg Corrado and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. (2013) 1--12. arXiv:1301.3781 · arXiv
- Akshay Minocha , Navjyoti Singh , and Arjit Srivastava . 2015. Finding Relevant Indian Judgments using Dispersion of Citation Network . In Proc. of WWW '15 . ACM Press , New York , New York, USA, 1085--1088. Akshay Minocha, Navjyoti Singh, and Arjit Srivastava. 2015. Finding Relevant Indian Judgments using Dispersion of Citation Network. In Proc. of WWW '15. ACM Press, New York, New York, USA, 1085--1088.
- 10.1007/s10506-018-9236-y · DOI
- 10.1145/3383583.3398616 · doi:10.1145/3383583.3398616
- Malte Ostendorff , Peter Bourgonje , Maria Berger , Julian Moreno-Schneider , Georg Rehm , and Bela Gipp . 2019 . Enriching BERT with Knowledge Graph Embeddings for Document Classification . In Proc. of the 15th Conference on Natural Language Processing (KONVENS 2019). GSCL, Erlangen, Germany, 305--312. Malte Ostendorff, Peter Bourgonje, Maria Berger, Julian Moreno-Schneider, Georg Rehm, and Bela Gipp. 2019. Enriching BERT with Knowledge Graph Embeddings for Document Classification. In Proc. of the 15th Conference on Natural Language Processing (KONVENS 2019). GSCL, Erlangen, Germany, 305--312.
- 10.5555/1953048.2078195 · doi:10.5555/1953048.2078195
- 10.3115/v1/D14-1162 · DOI · Paper page
- Bryan Perozzi , Rami Al-Rfou , and Steven Skiena . 2014. DeepWalk: online learning of social representations . In Proc. of KDD '14 . ACM Press , New York , New York, USA, 701--710. Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. DeepWalk: online learning of social representations. In Proc. of KDD '14. ACM Press, New York, New York, USA, 701--710.
- 10.1145/3110025.3110086 · DOI · Paper page
- 10.18653/v1/D19-1410 · DOI · Paper page
- Benedek Rozemberczki Oliver Kiss and Rik Sarkar. 2020. An API Oriented Open-source Python Framework for Unsupervised Learning on Graphs. (2020). arXiv:2003.04819 Benedek Rozemberczki Oliver Kiss and Rik Sarkar. 2020. An API Oriented Open-source Python Framework for Unsupervised Learning on Graphs. (2020). arXiv:2003.04819 · arXiv
- 10.1145/361219.361220 · DOI · Paper page
- 10.1145/2910896.2910908 · DOI
- 10.1007/s10506-017-9195-8 · DOI
- A. Vaswani N. Shazeer N. Parmar J. Uszkoreit L. Jones A. N. Gomez L. Kaiser and I. Polosukhin. 2017. Attention Is All You Need. Advances in Neural Information Processing Systems 30 (Jun 2017) 5998--6008. A. Vaswani N. Shazeer N. Parmar J. Uszkoreit L. Jones A. N. Gomez L. Kaiser and I. Polosukhin. 2017. Attention Is All You Need. Advances in Neural Information Processing Systems 30 (Jun 2017) 5998--6008.
- 10.7717/peerj-cs.262 · DOI
- Lidan Wang , Ming Tan , and Jiawei Han . 2016 . FastHybrid: A hybrid model for efficient answer selection . Proc. of the 26th International Conference on Computational Linguistics (2016), 2378--2388. Lidan Wang, Ming Tan, and Jiawei Han. 2016. FastHybrid: A hybrid model for efficient answer selection. Proc. of the 26th International Conference on Computational Linguistics (2016), 2378--2388.
- 10.18653/v1/n18-1101 · DOI · Paper page
- Radboud Winkels , Alexander Boer , Bart Vredebregt , and Alexander Van Someren . 2014 . Towards a Legal Recommender System . In Frontiers in Artificial Intelligence and Applications , Vol. 271. 169 -- 178 . Radboud Winkels, Alexander Boer, Bart Vredebregt, and Alexander Van Someren. 2014. Towards a Legal Recommender System. In Frontiers in Artificial Intelligence and Applications, Vol. 271. 169--178.
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Metadata
- Authors
- Ostendorff, M., Ash, E., Ruas, T., Gipp, B., Moreno-Schneider, J., & Rehm, G.
- Affiliations
- Open Legal Data, Germany
- ETH Zurich, Switzerland
- University of Wuppertal, Germany
- DFKI GmbH, Germany
- Venue
- Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law
- Publisher
- ACM
- Published
- 2021-06-21
- Pages
- 109-118
- Cited by
- 21
- References
- 6
- Funders
- German Federal Ministry of Education and Research (BMBF) (03WKDA1A)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluating document representations for content-based legal literature recommendations. https://app.cochat.ai/papers/10.1145/3462757.3466073
- The paper
- Ostendorff, M., Ash, E., Ruas, T., Gipp, B., Moreno-Schneider, J., & Rehm, G. (2021). Evaluating document representations for content-based legal literature recommendations. Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law. https://doi.org/10.1145/3462757.3466073
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.