Active learning for biomedical citation screening
The DOI matches its Crossref record; 32 of the 34 references it cites were found in a registry or an index.
Abstract
Active learning (AL) is an increasingly popular strategy for mitigating the amount of labeled data required to train classifiers, thereby reducing annotator effort. We describe a real-world, deployed application of AL to the problem of biomedical citation screening for systematic reviews at the Tufts Medical Center's Evidence-based Practice Center. We propose a novel active learning strategy that exploits a priori domain knowledge provided by the expert (specifically, labeled features)and extend this model via a Linear Programming algorithm for situations where the expert can provide ranked labeled features. Our methods outperform existing AL strategies on three real-world systematic review datasets. We argue that evaluation must be specific to the scenario under consideration. To this end, we propose a new evaluation framework for finite-pool scenarios, wherein the primary aim is to label a fixed set of examples rather than to simply induce a good predictive model. We use a method from medical decision theory for eliciting the relative costs of false positives and false negatives from the domain expert, constructing a utility measure of classification performance that integrates the expert preferences. Our findings suggest that the expert can, and should, provide more information than instance labels alone. In addition to achieving strong empirical results on the citation screening problem, this work outlines many important steps for moving away from simulated active learning and toward deploying AL for real-world applications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Active learning for systematic review screening promises to reduce the human effort required to identify relevant documents for a systematic review.”
From Statistical stopping criteria for automated screening in systematic reviews (2020) · cites this paper as [5]
Citations
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918 · cited 1×
“We use a modified version of LibSVM [3] and its Python interface.”
MethodThe Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Libsvm : A library for support vector machines
Ferrari, V. (2008) · Medical Entomology and Zoology · Cited by 10,122
Text categorization with Support Vector Machines: Learning with many relevant features
Joachims, T. (1998) · Lecture Notes in Computer Science · doi:10.1007/bfb0026683 · Cited by 7,999 · cited 1×
“We used Support Vector Machines (SVMs) as the base classier [27] due to their good empirical performance over text data [ 12 ] and the Simple [26] method for uncertainty sampling, which selects for labeling those instances closest to the current separating hyperplane.”
MethodDecision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341 · cited 3×
“We borrow a method from medical decision making developed for diagnostic test assessment to infer this weight by means of a thought experiment [28].”
MethodInfluentialCombining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
A Practical Guide to Support Vector Classication
Hsu, C., Chang, C.-C., & Lin, C. (2008) · Cited by 5,083
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
Learning in the Presence of Concept Drift and Hidden Contexts
Widmer, G., & Kubat, M. (1996) · Machine Learning · doi:10.1023/a:1018046501280 · Cited by 1,857 · cited 3×
“[30] G. Widmer and M. Kubat.”
BackgroundInfluentialA sequential algorithm for training text classifiers
Lewis, D., & Gale, W. A. (1994) · arXiv (Cornell University) · doi:10.5555/188490.188495 · Cited by 1,731
Get another label? improving data quality and data mining using multiple, noisy labelers
Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008) · Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1401890.1401965 · Cited by 1,165 · cited 1×
“[23] addressed this ‘crowd-sourced’ scenario, i.”
Selective Sampling Using the Query by Committee Algorithm
Freund, Y., Seung, H. S., Shamir, E., & Tishby, N. (1997) · Machine Learning · doi:10.1023/a:1007330508534 · Cited by 1,097 · cited 1×
“Note that Co-Testing is a specific case of Query by Committee [8].”
BackgroundBreast Cancer Diagnosis and Prognosis Via Linear Programming
Mangasarian, O. L., Street, W. N., & Wolberg, W. H. (1995) · Operations Research · doi:10.1287/opre.43.4.570 · Cited by 843 · cited 2×
“Similar to existing LP methods [17], we begin by assuming that we have a set of positive instances (relevant citations) P and a set of negative instances N .”
MethodEmploying EM and Pool-Based Active Learning for Text Classification
McCallum, A., & Nigam, K. (1998) · Cited by 689
Formulating Questions and Locating Primary Studies for Inclusion in Systematic Reviews
Counsell, C. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-5-199709010-00008 · Cited by 505 · cited 1×
“To minimize bias, a systematic review develops and follows a protocol of well defined steps, including: formulating answerable research questions, specifying literature review criteria, conducting a comprehensive literature search, screening of the abstracts obtained from the search to select potentially relevant stud…”
MethodLearning from Imbalanced Data Sets: A Comparison of Various Strategies *
Japkowicz, N. (2000) · Cited by 455
Support Vector Machine Active Learning with Application sto Text Classification
Tong, S., & Koller, D. (2000) · International Conference on Machine Learning · Cited by 369
Online Choice of Active Learning Algorithms
Baram, Y., El‐Yaniv, R., & Luz, K. (2003) · International Conference on Machine Learning · Cited by 369
Semi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · cited 2×
“To formalize the above intuition, we define two metrics we have proposed elsewhere [29]: yield and burden.”
BackgroundMethodSelf-citationBiomedical Language Processing: What's Beyond PubMed?
Hunter, L., & Cohen, K. B. (2006) · Molecular Cell · doi:10.1016/j.molcel.2006.02.012 · Cited by 308 · cited 1×
“Furthermore, this review effort will only continue to grow due to the exponential growth of biomedical literature [10], motivating methods to semi-automate this process.”
BackgroundEvaluating and optimizing autonomous text classification systems
Lewis, D. D. (1995) · Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '95 · doi:10.1145/215206.215366 · Cited by 308 · cited 1×
“Most work in information retrieval on metrics for the evaluation of text classifiers has focused on the weighted F measure [13], i.”
BackgroundLearning from labeled features using generalized expectation criteria
Druck, G., Mann, G., & McCallum, A. (2008) · Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1390334.1390436 · Cited by 249 · cited 1×
“[7] propose an active learning framework in which the expert labels features rather than instances and uses the generalized expectation (GE) criteria to build a predictive model from labeled features [6].”
BackgroundActive Learning with Multiple Views
Muslea, I., Minton, S., & Knoblock, C. A. (2006) · Journal of Artificial Intelligence Research · doi:10.1613/jair.2005 · Cited by 236
Proactive learning: cost-sensitive active learning with multiple imperfect oracles
Donmez, P., & Carbonell, J. G. (2008) · Proceedings of the 17th ACM conference on Information and knowledge management · doi:10.1145/1458082.1458165 · Cited by 228 · cited 2×
“[5] P. Donmez and J. G. Carbonell.”
MethodOnline Choice of Active Learning Algorithms
BaramYoram, El-YanivRan, & LuzKobi (2004) · Journal of Machine Learning Research · doi:10.5555/1005332.1005342 · Cited by 192
Active learning by labeling features
Druck, G., Settles, B., & McCallum, A. (2009) · Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing Volume 1 - EMNLP '09 · doi:10.3115/1699510.1699522 · Cited by 162 · cited 2×
“[7] propose an active learning framework in which the expert labels features rather than instances and uses the generalized expectation (GE) criteria to build a predictive model from labeled features [6].”
BackgroundBalancing Exploration and Exploitation: A New Algorithm for Active Machine Learning
Osugi, T. T., Deng, K., & Scott, S. (2006) · Proceedings · doi:10.1109/icdm.2005.33 · Cited by 132 · cited 2×
“Other work [18, 1] has also addressed this problem more generally as a tradeoff between exploration (random sampling) and exploitation (uncertainty sampling).”
BackgroundAn interactive algorithm for asking and incorporating feature feedback into support vector machines
Raghavan, H., & Allan, J. (2007) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1277741.1277758 · Cited by 72 · cited 2×
“[20, 19] present a method for interleaving feature and instance labels.”
BackgroundMethodActive Learning with Feedback on Features and Instances
RaghavanHema, MadaniOmid, & JonesRosie (2006) · Journal of Machine Learning Research · doi:10.5555/1248547.1248608 · Cited by 71
A web survey on the use of active learning to support annotation of text data
Tomanek, K., & Olsson, F. (2009) · Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing - HLT '09 · doi:10.3115/1564131.1564140 · Cited by 66 · cited 1×
“sociated with deploying AL in “real-world” scenarios [25].”
BackgroundUncertainty sampling and transductive experimental design for active dual supervision
Sindhwani, V., Melville, P., & Lawrence, R. D. (2009) · Proceedings of the 26th Annual International Conference on Machine Learning · doi:10.1145/1553374.1553496 · Cited by 56 · cited 1×
“[24] presented a method for querying an oracle for labels on both features and instances.”
MethodPerformance thresholding in practical text classification
Schütze, H., Velipasaoglu, E., & Pedersen, J. O. (2006) · Proceedings of the 15th ACM international conference on Information and knowledge management - CIKM '06 · doi:10.1145/1183614.1183709 · Cited by 54 · cited 3×
“This assumption is violated in the case of XOR-like or multiple input distribution concept clusters [1, 21], as uncertainty sampling may continue to request labels along the initially discovered boundary, ignoring as-yet undiscovered partitions.”
Background
Research works printed but not linked: 3
- 10.5555/1699510.1699522 · doi:10.5555/1699510.1699522
- 10.5555/1564131.1564140 · doi:10.5555/1564131.1564140
- O. F. Zaidan , J. Eisner , and C. Piatko . Machine learning with annotator rationales to reduce annotation cost . In NIPS Workshop on Cost Sensitive Learning , December 2008 O. F. Zaidan, J. Eisner, and C. Piatko. Machine learning with annotator rationales to reduce annotation cost. In NIPS Workshop on Cost Sensitive Learning, December 2008
Cited by
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodInfluentialSettles, B. (2012) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.2200/s00429ed1v01y201207aim018 · Cited by 605
ERASER: A Benchmark to Evaluate Rationalized NLP Models
DeYoung, J., Jain, S., Rajani, N. F., Lehman, E., Xiong, C., Socher, R., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.408 · Cited by 561
Self-citationAutomation of systematic literature reviews: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106589 · Cited by 432
“…Freire, Gual ´ an and Saquicela [40] describe an algorithm that automates metadata extraction from articles Cost assignment [66] Data resampling [41,42,58,64] SMOTE [43,53] Feature enrichment [43,63] Cost-sensitive classifiers [43,53, 61,66] Precision@95% recall [ published by IEEE, ACM, Springer,…”
MethodSearching for studies: a guide to information retrieval for Campbell systematic reviews
Kugley, S., Wade, A., Thomas, J., Mahood, Q., Jørgensen, A. K., Hammerstrøm, K., & Sathe, N. (2017) · Campbell Systematic Reviews · doi:10.4073/cmg.2016.1 · Cited by 371
Metadata
- Authors
- Wallace, B. C., Small, K., Brodley, C. E., & Trikalinos, T. A.
- Affiliations
- Tufts University (US)
- Tufts Medical Center (US)
- Venue
- Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining
- Publisher
- ACM
- Published
- 2010-07-25
- Pages
- 173-182
- OpenAlex
- W2037771830
- Cited by
- 123
- References
- 34
- Citations by year
- 2026: 5 · 2025: 7 · 2024: 4 · 2023: 10 · 2022: 4 · 2021: 8 · 2020: 13 · 2019: 9 · 2018: 10 · 2017: 10 · 2016: 8 · 2015: 10 · 2014: 7 · 2013: 7 · 2012: 7
- FWCI
- 4.65
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- Funders
- National Science Foundation
- U.S. Social Security Administration
- National Institutes of Health
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- active learning (machine learning), systematic reviews, linear programming, evidence-based practice centers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.