CoChat

Evaluating document representations for content-based legal literature recommendations

Ostendorff, Malte, Ash, Elliott, Ruas, Terry, Gipp, Bela, Moreno-Schneider, Julian, Rehm, Georg · 2021 · Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law
Cited by 21

The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.

Abstract

Recommender systems assist legal professionals in finding relevant literature for supporting their case. Despite its importance for the profession, legal applications do not reflect the latest advances in recommender systems and representation learning research. Simultaneously, legal recommender systems are typically evaluated in small-scale user study without any public available benchmark datasets. Thus, these studies have limited reproducibility. To address the gap between research and practice, we explore a set of state-of-the-art document representation methods for the task of retrieving semantically related US case law. We evaluate text-based (e.g., fast-Text, Transformers), citation-based (e.g., DeepWalk, Poincaré), and hybrid methods. We compare in total 27 methods using two silver standards with annotations for 2,964 documents. The silver standards are newly created from Open Case Book and Wikisource and can be reused under an open license facilitating reproducibility. Our experiments show that document representations from averaged fastText word vectors (trained on legal corpora) yield the best results, closely followed by Poincaré citation embeddings. Combining fastText and Poincaré in a hybrid manner further improves the overall result. Besides the overall performance, we analyze the methods depending on document length, citation count, and the coverage of their recommendations.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3462757.3466073

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 43

  1. Sanjeev Arora , Yingyu Liang , and Tengyu Ma . 2017 . A simple but though Baseline for Sentence Embeddings . In 5th International Conference on Learning Representations (ICLR 2017), Vol. 15 . 416--424. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but though Baseline for Sentence Embeddings. In 5th International Conference on Learning Representations (ICLR 2017), Vol. 15. 416--424.
  2. 10.2139/ssrn.3204926 · DOI
  3. 10.1109/ACCESS.2018.2890388 · DOI · Paper page
  4. Joeran Beel , Corinna Breitinger , Stefan Langer , Andreas Lommatzsch , and Bela Gipp . 2016. Towards reproducibility in recommender-systems research. User Modeling and User-Adapted Interaction (UMAI) 26 ( 2016 ). Joeran Beel, Corinna Breitinger, Stefan Langer, Andreas Lommatzsch, and Bela Gipp. 2016. Towards reproducibility in recommender-systems research. User Modeling and User-Adapted Interaction (UMAI) 26 (2016).
  5. Iz Beltagy , Matthew E. Peters , and Arman Cohan . 2020 . Longformer: The Long-Document Transformer. (2020). arXiv:2004.05150 Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long-Document Transformer. (2020). arXiv:2004.05150 · arXiv
  6. Paheli Bhattacharya Kripabandhu Ghosh Arindam Pal and Saptarshi Ghosh. 2020. Methods for Computing Legal Document Similarity: A Comparative Study. (2020). arXiv:2004.12307 Paheli Bhattacharya Kripabandhu Ghosh Arindam Pal and Saptarshi Ghosh. 2020. Methods for Computing Legal Document Similarity: A Comparative Study. (2020). arXiv:2004.12307 · arXiv
  7. David M Blei , Andrew Y Ng , and Michael I Jordan . 2003 . Latent dirichlet allocation . Journal of machine Learning research 3 , Jan (2003), 993 -- 1022 . David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of machine Learning research 3, Jan (2003), 993--1022.
  8. 10.1007/s10506-016-9184-3 · DOI
  9. 10.3233/978-1-61499-726-9-131 · doi:10.3233/978-1-61499-726-9-131
  10. 10.1162/tacl_a_00051 · DOI · Paper page
  11. Samuel R. Bowman , Gabor Angeli , Christopher Potts , and Christopher D. Manning . 2015. A large annotated corpus for learning natural language inference . Proc. of EMNLP ( 2015 ), 632--642. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. Proc. of EMNLP (2015), 632--642.
  12. 10.18653/v1/2020.findings-emnlp.261 · DOI · Paper page
  13. Mouzhi Ge , Carla Delgado-Battenfeld , and Dietmar Jannach . 2010. Beyond accuracy: evaluating recommender systems by coverage and serendipity . In Proc. of RecSys '10 . ACM Press , New York , New York, USA, 257. Mouzhi Ge, Carla Delgado-Battenfeld, and Dietmar Jannach. 2010. Beyond accuracy: evaluating recommender systems by coverage and serendipity. In Proc. of RecSys '10. ACM Press, New York, New York, USA, 257.
  14. 10.1145/2939672.2939754 · DOI · Paper page
  15. Nils Holzenberger , Andrew Blair-Stanek , and Benjamin Van Durme . 2020 . A dataset for statutory reasoning in tax law entailment and question answering . In Proc. of the 2020 Natural Legal Language Processing Workshop. 31--38 . Nils Holzenberger, Andrew Blair-Stanek, and Benjamin Van Durme. 2020. A dataset for statutory reasoning in tax law entailment and question answering. In Proc. of the 2020 Natural Legal Language Processing Workshop. 31--38.
  16. 10.1111/j.1469-8137.1912.tb05611.x · DOI · Paper page
  17. 10.1145/1980422.1980439 · DOI
  18. 10.18653/v1/w16-1609 · DOI · Paper page
  19. Quoc V. Le and Tomas Mikolov. 2014 . Distributed Representations of Sentences and Documents. Int. Conf. on Machine Learning 32 ( 2014 ), 1188--1196. Quoc V. Le and Tomas Mikolov. 2014. Distributed Representations of Sentences and Documents. Int. Conf. on Machine Learning 32 (2014), 1188--1196.
  20. 10.1145/3341161.3342864 · DOI
  21. Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. (2019). arXiv:1907.11692 Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. (2019). arXiv:1907.11692 · arXiv
  22. 10.1145/3140107.3140119 · doi:10.1145/3140107.3140119
  23. 10.1017/CBO9780511809071 · DOI · Paper page
  24. David Mellinkoff . 1963. The language of the law . Boston : Little Brown and Company ( 1963 ). David Mellinkoff. 1963. The language of the law. Boston: Little Brown and Company (1963).
  25. Tomas Mikolov Kai Chen Greg Corrado and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. (2013) 1--12. arXiv:1301.3781 Tomas Mikolov Kai Chen Greg Corrado and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. (2013) 1--12. arXiv:1301.3781 · arXiv
  26. Akshay Minocha , Navjyoti Singh , and Arjit Srivastava . 2015. Finding Relevant Indian Judgments using Dispersion of Citation Network . In Proc. of WWW '15 . ACM Press , New York , New York, USA, 1085--1088. Akshay Minocha, Navjyoti Singh, and Arjit Srivastava. 2015. Finding Relevant Indian Judgments using Dispersion of Citation Network. In Proc. of WWW '15. ACM Press, New York, New York, USA, 1085--1088.
  27. 10.1007/s10506-018-9236-y · DOI
  28. 10.1145/3383583.3398616 · doi:10.1145/3383583.3398616
  29. Malte Ostendorff , Peter Bourgonje , Maria Berger , Julian Moreno-Schneider , Georg Rehm , and Bela Gipp . 2019 . Enriching BERT with Knowledge Graph Embeddings for Document Classification . In Proc. of the 15th Conference on Natural Language Processing (KONVENS 2019). GSCL, Erlangen, Germany, 305--312. Malte Ostendorff, Peter Bourgonje, Maria Berger, Julian Moreno-Schneider, Georg Rehm, and Bela Gipp. 2019. Enriching BERT with Knowledge Graph Embeddings for Document Classification. In Proc. of the 15th Conference on Natural Language Processing (KONVENS 2019). GSCL, Erlangen, Germany, 305--312.
  30. 10.5555/1953048.2078195 · doi:10.5555/1953048.2078195
  31. 10.3115/v1/D14-1162 · DOI · Paper page
  32. Bryan Perozzi , Rami Al-Rfou , and Steven Skiena . 2014. DeepWalk: online learning of social representations . In Proc. of KDD '14 . ACM Press , New York , New York, USA, 701--710. Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. DeepWalk: online learning of social representations. In Proc. of KDD '14. ACM Press, New York, New York, USA, 701--710.
  33. 10.1145/3110025.3110086 · DOI · Paper page
  34. 10.18653/v1/D19-1410 · DOI · Paper page
  35. Benedek Rozemberczki Oliver Kiss and Rik Sarkar. 2020. An API Oriented Open-source Python Framework for Unsupervised Learning on Graphs. (2020). arXiv:2003.04819 Benedek Rozemberczki Oliver Kiss and Rik Sarkar. 2020. An API Oriented Open-source Python Framework for Unsupervised Learning on Graphs. (2020). arXiv:2003.04819 · arXiv
  36. 10.1145/361219.361220 · DOI · Paper page
  37. 10.1145/2910896.2910908 · DOI
  38. 10.1007/s10506-017-9195-8 · DOI
  39. A. Vaswani N. Shazeer N. Parmar J. Uszkoreit L. Jones A. N. Gomez L. Kaiser and I. Polosukhin. 2017. Attention Is All You Need. Advances in Neural Information Processing Systems 30 (Jun 2017) 5998--6008. A. Vaswani N. Shazeer N. Parmar J. Uszkoreit L. Jones A. N. Gomez L. Kaiser and I. Polosukhin. 2017. Attention Is All You Need. Advances in Neural Information Processing Systems 30 (Jun 2017) 5998--6008.
  40. 10.7717/peerj-cs.262 · DOI
  41. Lidan Wang , Ming Tan , and Jiawei Han . 2016 . FastHybrid: A hybrid model for efficient answer selection . Proc. of the 26th International Conference on Computational Linguistics (2016), 2378--2388. Lidan Wang, Ming Tan, and Jiawei Han. 2016. FastHybrid: A hybrid model for efficient answer selection. Proc. of the 26th International Conference on Computational Linguistics (2016), 2378--2388.
  42. 10.18653/v1/n18-1101 · DOI · Paper page
  43. Radboud Winkels , Alexander Boer , Bart Vredebregt , and Alexander Van Someren . 2014 . Towards a Legal Recommender System . In Frontiers in Artificial Intelligence and Applications , Vol. 271. 169 -- 178 . Radboud Winkels, Alexander Boer, Bart Vredebregt, and Alexander Van Someren. 2014. Towards a Legal Recommender System. In Frontiers in Artificial Intelligence and Applications, Vol. 271. 169--178.

Cited by

See all 21

Metadata

Authors
Ostendorff, M., Ash, E., Ruas, T., Gipp, B., Moreno-Schneider, J., & Rehm, G.
Affiliations
  • Open Legal Data, Germany
  • ETH Zurich, Switzerland
  • University of Wuppertal, Germany
  • DFKI GmbH, Germany
Venue
Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law
Publisher
ACM
Published
2021-06-21
Pages
109-118
Cited by
21
References
6
Funders
German Federal Ministry of Education and Research (BMBF) (03WKDA1A)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.