Multi-Faceted Recall of Continuous Active Learning for Technology-Assisted Review
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Continuous active learning achieves high recall for technology-assisted review, not only for an overall information need, but also for various facets of that information need, whether explicit or implicit. Through simulations using Cormack and Grossman's TAR Evaluation Toolkit (SIGIR 2014), we show that continuous active learning, applied to a multi-faceted topic, efficiently achieves high recall for each facet of the topic. Our results assuage the concern that continuous active learning may achieve high overall recall at the expense of excluding identifiable categories of relevant information.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The application of active learning models in systematic reviews has been extensively studied in simulation studies [11-21].”
Citations
Cites
RCV1: A New Benchmark Collection for Text Categorization Research
Lewis, D., Yang, Y., Rose, T. G., & Li, F. (2004) · Goldsmiths (University of London) · doi:10.5555/1005332.1005345 · Cited by 2,503
Novelty and diversity in information retrieval evaluation
Clarke, C. L. A., Kolla, M., Cormack, G. V., Vechtomova, O., Ashkan, A., Büttcher, S., & MacKinnon, I. (2008) · Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1390334.1390446 · Cited by 962
Self-citationVariations in relevance judgments and the measurement of retrieval effectiveness
Voorhees, E. M. (2000) · Information Processing & Management · doi:10.1016/s0306-4573(00)00010-8 · Cited by 504
Stemming algorithms: A case study for detailed evaluation
Hull, D. A. (1996) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199601)47:1<70::aid-asi7>3.3.co;2-q · Cited by 369
Evaluation of machine-learning protocols for technology-assisted review in electronic discovery
Cormack, G. V., & Grossman, M. R. (2014) · Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval · doi:10.1145/2600428.2609601 · Cited by 149
Self-citationGrossman, M. R., & Cormack, G. V. (2011) · Richmond journal of law & technology · Cited by 83
Self-citationSTAIRS redux: Thoughts on the STAIRS evaluation, ten years after
Blair, D. C. (1996) · Journal of the American Society for Information Science · doi:10.1002/(sici)1097-4571(199601)47:1<4::aid-asi2>3.0.co;2-3 · Cited by 64
Overview of the TREC 2009 Legal Track
Hedin, B., Tomlinson, S., Baron, J. R., & Oard, D. W. (2009) · Cited by 58
Machine Learning for Information Retrieval: TREC 2009 Web, Relevance Feedback and Legal Tracks.
Cormack, G. V., & Mojdeh, M. (2009) · Cited by 26
Self-citationTowards minimizing the annotation cost of certified text classification
Bagdouri, M., Webber, W., Lewis, D. D., & Oard, D. W. (2013) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2505515.2505708 · Cited by 14
Sequential testing in classifier evaluation yields biased estimates of effectiveness
Webber, W., Bagdouri, M., Lewis, D. D., & Oard, D. W. (2013) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/2484028.2484159 · Cited by 8
Introduction: contains Cover, Table of Contents, Letter from the Editor, and Masthead
Oroszlan, F. C. (2011) · Richmond journal of law & technology · Cited by 7
Research works printed but not linked: 4
- 10.1002/(sici)1097-4571(199601)47:1%3c4::aid-asi2%3e3.3.co;2-5 · doi:10.1002/(sici)1097-4571(199601)47:1<4::aid-asi2>3.3.co;2-5
- Cormack G., 2009, Relevance Feedback and Legal Tracks. In Eighteenth Text REtrieval Conference
- Grossman M. R., 2014, Comments on "The implications of rule 26 (g) on the use of technology-assisted review, Federal Courts Law Review, 7, 285
- Schieneman K., 2013, The implications of Rule 26(g) on the use of technology-assisted review, Federal Courts Law Review, 7, 239
Cited by
Scalability of Continuous Active Learning for Reliable High-Recall Text Classification
Cormack, G. V., & Grossman, M. R. (2016) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2983323.2983776 · Cited by 100
Self-citationTREC 2016 Total Recall Track Overview.
Grossman, M. R., Cormack, G. V., & Roegiest, A. (2016) · Text REtrieval Conference · Cited by 99
Self-citationEngineering Quality and Reliability in Technology-Assisted Review
Cormack, G. V., & Grossman, M. R. (2016) · Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval · doi:10.1145/2911451.2911510 · Cited by 98
Self-citationFerdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., Teijema, J. J., & van de Schoot, R. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02257-7 · Cited by 96
TREC 2015 Total Recall Track Overview.
Roegiest, A., Cormack, G. V., Clarke, C. L. A., & Grossman, M. R. (2015) · Text REtrieval Conference · Cited by 58
Self-citation
Metadata
- Authors
- Cormack, G. V., & Grossman, M. R.
- Affiliations
- University of Waterloo (CA)
- Wachtell, Lipton, Rosen & Katz, New York, NY, USA
- Venue
- Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval
- Publisher
- ACM
- Published
- 2015-08-09
- Pages
- 763-766
- OpenAlex
- W2122053905
- Cited by
- 38
- References
- 12
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 3 · 2023: 8 · 2022: 2 · 2021: 1 · 2020: 4 · 2019: 3 · 2018: 3 · 2017: 4 · 2016: 5 · 2015: 1
- FWCI
- 3.77
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Keywords
- technology-assisted review, information retrieval, query refinement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.