Jointly Minimizing the Expected Costs of Review for Responsiveness and Privilege in E-Discovery
The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
Abstract
Discovery is an important aspect of the civil litigation process in the United States of America, in which all parties to a lawsuit are permitted to request relevant evidence from other parties. With the rapid growth of digital content, the emerging need for “e-discovery” has created a strong demand for techniques that can be used to review massive collections both for “responsiveness” (i.e., relevance) to the request and for “privilege” (i.e., presence of legally protected content that the party performing the review may have a right to withhold). In this process, the party performing the review may incur costs of two types, namely, annotation costs (deriving from the fact that human reviewers need to be paid for their work) and misclassification costs (deriving from the fact that failing to correctly determine the responsiveness or privilege of a document may adversely affect the interests of the parties in various ways). Relying exclusively on automatic classification would minimize annotation costs but could result in substantial misclassification costs, while relying exclusively on manual classification could generate the opposite consequences. This article proposes a risk minimization framework (called MINECORE, for “min imizing the e xpected co sts of re view”) that seeks to strike an optimal balance between these two extreme stands. In MINECORE (a) the documents are first automatically classified for both responsiveness and privilege, and then (b) some of the automatically classified documents are annotated by human reviewers for responsiveness (typically by junior reviewers) and/or, in cascade, for privilege (typically by senior reviewers), with the overall goal of minimizing the expected cost (i.e., the risk ) of the entire process. Risk minimization is achieved by optimizing, for both responsiveness and privilege, the choice of which documents to manually review. We present a simulation study in which classes from a standard text classification test collection (RCV1-v2) are used as surrogates for responsiveness and privilege. The results indicate that MINECORE can yield substantially lower total cost than any of a set of strong baselines.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Current systems are costly because they used supervised learning and active learning to label the documents as responsive [Grossman and Cormack 2010; Oard et al. 2018; Yang et al. 2021].”
From On the Opportunities and Risks of Foundation Models (2021) · cites this paper as Oard et al. 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Prospect Theory: An Analysis of Decision under Risk
Kahneman, D., & Tversky, A. (1979) · Econometrica · doi:10.2307/1914185 · Cited by 48,524
Transforming classifier scores into accurate multiclass probability estimates
Zadrozny, B., & Elkan, C. (2002) · Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/775047.775151 · Cited by 1,280
The Comparison and Evaluation of Forecasters
DeGroot, M. H., & Fienberg, S. E. (1983) · The Statistician · doi:10.2307/2987588 · Cited by 604 · cited 1×
“Posterior probabilities Pr(c |d ) are said to be well calibrated when, given a sample S drawn from some population, lim|S |→∞ |{d∈c | Pr(c |d )=x }| |{d∈S | Pr(c |d )=x }| = x [10].”
BackgroundEvaluation of machine-learning protocols for technology-assisted review in electronic discovery
Cormack, G. V., & Grossman, M. R. (2014) · Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval · doi:10.1145/2600428.2609601 · Cited by 149
Multi-Faceted Recall of Continuous Active Learning for Technology-Assisted Review
Cormack, G. V., & Grossman, M. R. (2015) · Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/2766462.2767771 · Cited by 38
Research works printed but not linked: 35
- Paul Anand . 1993. Foundations of Rational Choice Under Risk . Oxford University Press , Oxford, UK . Paul Anand. 1993. Foundations of Rational Choice Under Risk. Oxford University Press, Oxford, UK.
- 10.1145/2505515.2505708 · DOI
- Jason R. Baron Michael D. Berman and Ralph C. Losey (Eds.). 2016. Perspectives on Predictive Coding and Other Advanced Search and Review Technologies for the Legal Practitioner. ABA Book Publishing Washington. Jason R. Baron Michael D. Berman and Ralph C. Losey (Eds.). 2016. Perspectives on Predictive Coding and Other Advanced Search and Review Technologies for the Legal Practitioner. ABA Book Publishing Washington.
- 10.1145/2806416.2806597 · doi:10.1145/2806416.2806597
- 10.1145/2742548 · DOI
- Gordon V. Cormack and Maura R . Grossman . 2015 . Autonomy and reliability of continuous active learning for technology-assisted review. CoRR abs/1504.06868. Gordon V. Cormack and Maura R. Grossman. 2015. Autonomy and reliability of continuous active learning for technology-assisted review. CoRR abs/1504.06868. · arXiv
- Gordon V. Cormack and Mona Mojdeh. 2009. Machine learning for information retrieval: TREC 2009 web, relevance feedback and legal tracks . In Proceedings of the 18th Text Retrieval Conference (TREC’09) . Gordon V. Cormack and Mona Mojdeh. 2009. Machine learning for information retrieval: TREC 2009 web, relevance feedback and legal tracks. In Proceedings of the 18th Text Retrieval Conference (TREC’09).
- 10.1023/A:1007413511361 · DOI · Paper page
- 10.1145/1148170.1148216 · DOI
- Manfred Gabriel , Chris Paskach , and David Sharpe . 2013 . The challenge and promise of predictive coding for privilege . In Proceedings of the ICAIL 2013 Workshop on Standards for Using Predictive Coding (DESI’13) . Manfred Gabriel, Chris Paskach, and David Sharpe. 2013. The challenge and promise of predictive coding for privilege. In Proceedings of the ICAIL 2013 Workshop on Standards for Using Predictive Coding (DESI’13).
- 10.1109/ICDM.2015.137 · DOI
- Maura R. Grossman and Gordon V. Cormack . 2011 . Technology-assisted review in e-discovery can be more effective and more efficient than exhaustive manual review . Richmond J. Law Technol. 17 , 3 (2011), Article 5. Maura R. Grossman and Gordon V. Cormack. 2011. Technology-assisted review in e-discovery can be more effective and more efficient than exhaustive manual review. Richmond J. Law Technol. 17, 3 (2011), Article 5.
- Maura R. Grossman and Gordon V. Cormack . 2013 . The Grossman-Cormack glossary of technology-assisted review, with foreword by John M. Facciola, U.S. magistrate judge . Fed. Courts Law Rev. 7 , 1 (2013), 1 -- 34 . Maura R. Grossman and Gordon V. Cormack. 2013. The Grossman-Cormack glossary of technology-assisted review, with foreword by John M. Facciola, U.S. magistrate judge. Fed. Courts Law Rev. 7, 1 (2013), 1--34.
- Sherry B. Harris and Paul H. McVoy (Eds.). 2014 . The Sedona Conference Glossary: E-Discovery and Digital Information Management (4th ed.). Retrieved from http://bit.ly/2Bhz0TB. Sherry B. Harris and Paul H. McVoy (Eds.). 2014. The Sedona Conference Glossary: E-Discovery and Digital Information Management (4th ed.). Retrieved from http://bit.ly/2Bhz0TB.
- Thorsten Joachims . 1999. Making large-scale SVM learning practical . In Advances in Kernel Methods—Support Vector Learning , Bernhard Schölkopf , Christopher J. Burges, and Alexander J. Smola (Eds.). The MIT Press , Cambridge, MA, Chapter 11, 169--184. Thorsten Joachims. 1999. Making large-scale SVM learning practical. In Advances in Kernel Methods—Support Vector Learning, Bernhard Schölkopf, Christopher J. Burges, and Alexander J. Smola (Eds.). The MIT Press, Cambridge, MA, Chapter 11, 169--184.
- Thorsten Joachims . 2000 . Estimating the generalization performance of a SVM efficiently . In Proceedings of the 17th International Conference on Machine Learning (ICML’00) . 431--438. Thorsten Joachims. 2000. Estimating the generalization performance of a SVM efficiently. In Proceedings of the 17th International Conference on Machine Learning (ICML’00). 431--438.
- Thorsten Joachims . 2006. Transductive support vector machines . In Semi-Supervised Learning, Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien (Eds.). The MIT Press , Cambridge , MA , 105--117. Thorsten Joachims. 2006. Transductive support vector machines. In Semi-Supervised Learning, Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien (Eds.). The MIT Press, Cambridge, MA, 105--117.
- Ashish Kapoor , Eric Horvitz , and Sumit Basu . 2007 . Selective supervision: Guiding supervised learning with decision-theoretic active learning . In Proceedings of the 20th International Joint Conference on Artifical Intelligence (IJCAI’07) . 877--882. Ashish Kapoor, Eric Horvitz, and Sumit Basu. 2007. Selective supervision: Guiding supervised learning with decision-theoretic active learning. In Proceedings of the 20th International Joint Conference on Artifical Intelligence (IJCAI’07). 877--882.
- Daphne Koller and Mehran Sahami . 1997 . Hierarchically classifying documents using very few words . In Proceedings of the 14th International Conference on Machine Learning (ICML’97) . 170--178. Daphne Koller and Mehran Sahami. 1997. Hierarchically classifying documents using very few words. In Proceedings of the 14th International Conference on Machine Learning (ICML’97). 170--178.
- David D. Lewis and William A. Gale. 1994. A sequential algorithm for training text classifiers . In Proceedings of the 17th ACM International Conference on Research and Development in Information Retrieval (SIGIR’94) . 3--12. David D. Lewis and William A. Gale. 1994. A sequential algorithm for training text classifiers. In Proceedings of the 17th ACM International Conference on Research and Development in Information Retrieval (SIGIR’94). 3--12.
- 10.5555/1005332.1005345 · doi:10.5555/1005332.1005345
- Douglas Oard , Jyothi Vinjumur , and Fabrizio Sebastiani . 2017 . When is it rational to review for privilege? In Proceedings of the Workshop on Using Advanced Data Analysis in eDiscovery 8 Related Disciplines to Identify and Protect Sensitive Information in Large Collections (DESI’17) . Douglas Oard, Jyothi Vinjumur, and Fabrizio Sebastiani. 2017. When is it rational to review for privilege? In Proceedings of the Workshop on Using Advanced Data Analysis in eDiscovery 8 Related Disciplines to Identify and Protect Sensitive Information in Large Collections (DESI’17).
- 10.1561/1500000025 · DOI
- John C. Platt . 2000. Probabilistic outputs for support vector machines and comparison to regularized likelihood methods . In Advances in Large Margin Classifiers, Alexander Smola, Peter Bartlett , Bernard Schölkopf , and Dale Schuurmans (Eds.). The MIT Press , Cambridge, MA, 61--74. John C. Platt. 2000. Probabilistic outputs for support vector machines and comparison to regularized likelihood methods. In Advances in Large Margin Classifiers, Alexander Smola, Peter Bartlett, Bernard Schölkopf, and Dale Schuurmans (Eds.). The MIT Press, Cambridge, MA, 61--74.
- Stephen E. Robertson and Ian Soboroff. 2002 . The TREC 2002 filtering track report. In Proceedings of the 11th Text REtrieval Conference (TREC’02) . Stephen E. Robertson and Ian Soboroff. 2002. The TREC 2002 filtering track report. In Proceedings of the 11th Text REtrieval Conference (TREC’02).
- 10.5555/1672957.1672981 · doi:10.5555/1672957.1672981
- 10.1109/BigData.2015.7363867 · DOI
- 10.1109/ICDM.2005.102 · doi:10.1109/icdm.2005.102
- Burr Settles. 2012. Active Learning. Morgan 8 Claypool Publishers San Rafael. Burr Settles. 2012. Active Learning. Morgan 8 Claypool Publishers San Rafael. · DOI · Paper page
- Burr Settles , Mark Craven , and Lewis Friedland . 2008 . Active learning with real annotation costs . In Proceedings of the NIPS Workshop on Cost-Sensitive Learning. Burr Settles, Mark Craven, and Lewis Friedland. 2008. Active learning with real annotation costs. In Proceedings of the NIPS Workshop on Cost-Sensitive Learning.
- Katrin Tomanek and Udo Hahn . 2010 . A comparison of models for cost-sensitive active learning . In Proceedings of the 23rd International Conference on Computational Linguistics (COLING’10) . 1247--1255. Katrin Tomanek and Udo Hahn. 2010. A comparison of models for cost-sensitive active learning. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING’10). 1247--1255.
- 10.1145/2746090.2746101 · doi:10.1145/2746090.2746101
- 10.1145/2854946.2854964 · doi:10.1145/2854946.2854964
- 10.1145/2600428.2609506 · DOI
- Erik D. Wiener , Jan O. Pedersen , and Andreas S. Weigend . 1995. A neural network approach to topic spotting . In Proceedings of the 4th Annual Symposium on Document Analysis and Information Retrieval (SDAIR’95) . 317--332. Erik D. Wiener, Jan O. Pedersen, and Andreas S. Weigend. 1995. A neural network approach to topic spotting. In Proceedings of the 4th Annual Symposium on Document Analysis and Information Retrieval (SDAIR’95). 317--332.
Cited by
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Metadata
- Authors
- Oard, D. W., Sebastiani, F., & Vinjumur, J. K.
- Affiliations
- University of Maryland, MD, USA
- Consiglio Nazionale delle Ricerche, Pisa, Italy
- ORCID
- Sebastiani, Fabrizio
- Venue
- ACM Transactions on Information Systems
- Published
- 2018-11-19
- Volume
- 37
- Issue
- 1
- Pages
- 1-35
- ISSN
- 1046-8188
- 1558-2868
- Cited by
- 18
- References
- 5
- Language
- en
- Funders
- National Science Foundation (1065250 and 1618695)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.