The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Evaluation of machine-learning protocols for technology-assisted review in electronic discovery
Open accessAbstract
Abstract Using a novel evaluation toolkit that simulates a human reviewer in the loop, we compare the effectiveness of three machine-learning protocols for technology-assisted review as used in document review for discovery in legal proceedings. Our comparison addresses a central question in the deployment of technology-assisted review: Should training documents be selected at random, or should they be selected using one or more non-random methods, such as keyword search or active learning? On eight review tasks -- four derived from the TREC 2009 Legal Track and four derived from actual legal matters -- recall was measured as a function of human review effort. The results show that entirely non-random training methods, in which the initial training documents are selected using a simple keyword search, and subsequent training documents are selected by active learning, require substantially and significantly less human review effort (P<0.01) to achieve any given level of recall, than passive learning, in which the machine-learning algorithm plays no role in the selection of training documents. Among passive-learning methods, significantly less human review effort (P<0.01) is required when keywords are used instead of random sampling to select the initial training documents. Among active-learning methods, continuous active learning with relevance feedback yields generally superior results to simple active learning with uncertainty sampling, while avoiding the vexing issue of "stabilization" -- determining when training is adequate, and therefore may stop.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The application of active learning models in systematic reviews has been extensively studied in simulation studies.
“The application of active learning models in systematic reviews has been extensively studied in simulation studies [11-21].”
From Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records (2023) · cites this paper as [13]
The Abstract and §3, “Simulating Review,” describe a simulated evaluation of active-learning protocols across eight document-review tasks. Those tasks concern electronic discovery in legal proceedings, not systematic reviews; this paper therefore supports the simulation aspect but not the claimed setting. (LLM, conf=0.86)
Experimental — accuracy not yet measuredRecall plots offer insight in model performance throughout the entire screening process.
“Such recall plots offer insight in model performance throughout the entire screening process [11, 13].”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Machine learning in automated text categorization
Sebastiani, F. (2002) · ACM Computing Surveys · doi:10.1145/505282.505283 · Cited by 7,927
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
A generalization of sampling without replacement from a finite universe
Horvitz, D. G., & Thompson, D. J. (1952) · Journal of the American Statistical Association · doi:10.1080/01621459.1952.10483446 · Cited by 4,293
A Generalization of Sampling Without Replacement From a Finite Universe
Horvitz, D. G., & Thompson, D. J. (1952) · Journal of the American Statistical Association · doi:10.2307/2280784 · Cited by 3,011
A sequential algorithm for training text classifiersPreprint
Lewis, D., & Gale, W. A. (1994) · arXiv (Cornell University) · doi:10.5555/188490.188495 · Cited by 1,838
Variations in relevance judgments and the measurement of retrieval effectiveness
Voorhees, E. M. (2000) · Information Processing & Management · doi:10.1016/s0306-4573(00)00010-8 · Cited by 508
Information Retrieval: Implementing and Evaluating Search Engines
Büttcher, S., Clarke, C. L. A., & Cormack, G. V. (2010) · Cited by 384
Self-citationBailey, P., Craswell, N., Soboroff, I. M., Thomas, P., de Vries, A. P., & Yılmaz, E. (2008) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1390334.1390447 · Cited by 180
Information Retrieval for E-Discovery
Oard, D. W., & Webber, W. (2013) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000025 · Cited by 90
Grossman, M. R., & Cormack, G. V. (2011) · Richmond journal of law & technology · Cited by 85
Self-citationOverview of the TREC 2009 Legal Track
Hedin, B., Tomlinson, S., Baron, J. R., et al. (2009) · Cited by 58
Human performance and retrieval precision revisited
Smucker, M. D., & Jethani, C. P. (2010) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1835449.1835549 · Cited by 39
Machine Learning for Information Retrieval: TREC 2009 Web, Relevance Feedback and Legal Tracks.
Cormack, G. V., & Mojdeh, M. (2009) · Cited by 27
Self-citationInconsistent Responsiveness Determination in Document Review: Difference of Opinion or Human Error?
Grossman, M. R., & Cormack, G. V. (2012) · Pace law review · doi:10.58948/2331-3528.1800 · Cited by 17
Self-citationAssessor error in stratified evaluation
Webber, W., Oard, D. W., Scholer, F., & Hedin, B. (2010) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/1871437.1871508 · Cited by 17
Information Retrieval on the Blogosphere
Santos, R. L. T., Macdonald, C., McCreadie, R., et al. (2012) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000026 · Cited by 16
Towards minimizing the annotation cost of certified text classification
Bagdouri, M., Webber, W., Lewis, D. D., & Oard, D. W. (2013) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2505515.2505708 · Cited by 14
Assessor disagreement and text classifier accuracy
Webber, W., & Pickens, J. (2013) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2484028.2484156 · Cited by 13
H5 at TREC 2008 Legal Interactive: User Modeling, Assessment & Measurement
Hogan, C. J., Brassil, D., Reinhart, J., Gerber, M., & Jade, T. (2008) · Text REtrieval Conference · Cited by 10
Learning Task Experiments in the TREC 2010 Legal Track.
Tomlinson, S. (2010) · Cited by 9
Introduction: contains Cover, Table of Contents, Letter from the Editor, and Masthead
Oroszlan, F. C. (2011) · Richmond journal of law & technology · Cited by 6
The Impact of Incorrect Training Sets and Rolling Collections on Technology-Assisted Review.
Scholtes, J., & van Cann, T. H. W. (2013) · Cited by 5
The seventeenth text retrieval conference TREC 2008
Voorhees, E. M., & Buckland, L. P. (2009) · doi:10.6028/nist.sp.500-277 · Cited by 3
The nineteenth text retrieval conference TREC 2010
Voorhees, E. M., & Buckland, L. P. (2011) · doi:10.6028/nist.sp.500-294 · Cited by 2
Flagged first, then by citations
Research works printed but not linked: 13
- Da Silva, 2012, Publicis Groupe, 287, 182
- Order Case Management, 2012, La.
- Büttcher S., 2010, MIT Press
- Cheng J., 2013, ICAIL 2013 DESI V Workshop
- Cormack G. V., 2009, Relevance Feedback and Legal Tracks. The Eighteenth Text REtrieval Conference (TREC 2009)
- Grossman M. R., 2013, The Grossman-Cormack glossary of technology-assisted review with foreword by John M. Facciola, U.S. Magistrate Judge. Federal Courts Law Review, 7, 1
- Grossman M. R., Comments on "The Implications of Rule 26(g) on the Use of Technology-Assisted Review, Federal Courts Law Review, 1, 2014
- Ravid Y., 2013, System for Enhancing Expert-Based Computerized Analysis of a Set of Digital Documents and Methods Useful, Conjunction Therewith. United States Patent, 8527523
- 10.1002/asi.v61:1 · DOI
- Schieneman K., 2013, The implications of Rule 26(g) on the use of technology-assisted review, Federal Courts Law Review, 7, 239
- Scholtes J. C., 2013, ICAIL 2013 DESI V Workshop
- Settles B., 2010, University of Wisconsin
- Yablon C., 2013, Emerging questions and concerns, South Carolina Law Review, 64, 633
Cited by
Sorted by citations
Engineering Quality and Reliability in Technology-Assisted Review
Cormack, G. V., & Grossman, M. R. (2016) · Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval · doi:10.1145/2911451.2911510 · Cited by 98
Self-citationFerdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., Teijema, J. J., & van de Schoot, R. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02257-7 · Cited by 96
“Such recall plots offer insight in model performance throughout the entire screening process [11, 13].”
BackgroundFinding better active learners for faster literature reviews
Yu, Z., Kraft, N. A., & Menzies, T. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9587-0 · Cited by 83
FAST2: An intelligent assistant for finding relevant papers
Yu, Z., & Menzies, T. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.11.021 · Cited by 80
Active learning for screening prioritization in systematic reviews - A simulation study
Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., & van de Schoot, R. (2020) · doi:10.31219/osf.io/w6qbg · Cited by 51
Multi-Faceted Recall of Continuous Active Learning for Technology-Assisted Review
Cormack, G. V., & Grossman, M. R. (2015) · Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/2766462.2767771 · Cited by 38
Self-citation
Metadata
- Authors
- Cormack, G. V., & Grossman, M. R.
- Affiliations
- University of Waterloo (CA)
- Wachtell, Lipton, Rosen & Katz, New York, NY, USA
- Venue
- Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval
- Publisher
- ACM
- Published
- 2014-07-03
- Pages
- 153-162
- OpenAlex
- W2029075138
- Cited by
- 149
- References
- 24
- Citations by year
- 2026: 6 · 2025: 3 · 2024: 30 · 2023: 10 · 2022: 9 · 2021: 11 · 2020: 13 · 2019: 16 · 2018: 20 · 2017: 10 · 2016: 10 · 2015: 11
- FWCI
- 94.92
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
- Funders
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Social Sciences › Law › Artificial Intelligence in Law
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Keywords
- technology-assisted review, electronic discovery, document review, active learning (machine learning), passive learning, training data selection, keyword search, relevance feedback, uncertainty sampling, memory retrieval, legal proceedings
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
