Active learning for biomedical citation screening
The DOI matches its Crossref record; 32 of the 34 references it cites were found in a registry or an index.
Abstract
Active learning (AL) is an increasingly popular strategy for mitigating the amount of labeled data required to train classifiers, thereby reducing annotator effort. We describe a real-world, deployed application of AL to the problem of biomedical citation screening for systematic reviews at the Tufts Medical Center's Evidence-based Practice Center. We propose a novel active learning strategy that exploits a priori domain knowledge provided by the expert (specifically, labeled features)and extend this model via a Linear Programming algorithm for situations where the expert can provide ranked labeled features. Our methods outperform existing AL strategies on three real-world systematic review datasets. We argue that evaluation must be specific to the scenario under consideration. To this end, we propose a new evaluation framework for finite-pool scenarios, wherein the primary aim is to label a fixed set of examples rather than to simply induce a good predictive model. We use a method from medical decision theory for eliciting the relative costs of false positives and false negatives from the domain expert, constructing a utility measure of classification performance that integrates the expert preferences. Our findings suggest that the expert can, and should, provide more information than instance labels alone. In addition to achieving strong empirical results on the citation screening problem, this work outlines many important steps for moving away from simulated active learning and toward deploying AL for real-world applications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Active learning for systematic review screening promises to reduce the human effort required to identify relevant documents for a systematic review.”
From Statistical stopping criteria for automated screening in systematic reviews (2020) · cites this paper as [5]
Citations
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918 · cited 1×
“We use a modified version of LibSVM [3] and its Python interface.”
MethodThe Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Libsvm : A library for support vector machines
Ferrari, V. (2008) · Medical Entomology and Zoology · Cited by 10,123
Text categorization with Support Vector Machines: Learning with many relevant features
Joachims, T. (1998) · Lecture Notes in Computer Science · doi:10.1007/bfb0026683 · Cited by 7,999 · cited 1×
“We used Support Vector Machines (SVMs) as the base classier [27] due to their good empirical performance over text data [ 12 ] and the Simple [26] method for uncertainty sampling, which selects for labeling those instances closest to the current separating hyperplane.”
MethodDecision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341 · cited 3×
“We borrow a method from medical decision making developed for diagnostic test assessment to infer this weight by means of a thought experiment [28].”
MethodInfluentialCombining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
A Practical Guide to Support Vector Classication
Hsu, C., Chang, C.-C., & Lin, C. (2008) · Cited by 5,083
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
Learning in the Presence of Concept Drift and Hidden Contexts
Widmer, G., & Kubat, M. (1996) · Machine Learning · doi:10.1023/a:1018046501280 · Cited by 1,857 · cited 3×
“[30] G. Widmer and M. Kubat.”
BackgroundInfluentialA sequential algorithm for training text classifiers
Lewis, D., & Gale, W. A. (1994) · arXiv (Cornell University) · doi:10.5555/188490.188495 · Cited by 1,731
Research works printed but not linked: 3
- 10.5555/1699510.1699522 · doi:10.5555/1699510.1699522
- 10.5555/1564131.1564140 · doi:10.5555/1564131.1564140
- O. F. Zaidan , J. Eisner , and C. Piatko . Machine learning with annotator rationales to reduce annotation cost . In NIPS Workshop on Cost Sensitive Learning , December 2008 O. F. Zaidan, J. Eisner, and C. Piatko. Machine learning with annotator rationales to reduce annotation cost. In NIPS Workshop on Cost Sensitive Learning, December 2008
Cited by
Items 101–149 of 149
Natural Language Processing Approaches in Bioinformatics
Han, X., & Kwoh, C. K. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-809633-8.20463-9 · Cited by 3
Simulation-based Active Learning for Systematic Reviews: A Scoping Review of LiteraturePreprint
Teijema, J. J., Ribeiro, G., Seuren, S., Anadria, D., Bagheri, A., & van de Schoot, R. (2025) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/67zmt_v2 · Cited by 3
A large-scale study of SVM-based methods for abstract screening in systematic reviewsPreprint
Saha, T. K., Ouzzani, M., Hammady, H. M., & Elmagarmid, A. K. (2017) · arXiv (Cornell University) · doi:10.6084/m9.figshare.4522916 · Cited by 3
Sampling Strategies to Evaluate the Performance of Unknown Predictors
Valizadegan, H., Amizadeh, S., & Hauskrecht, M. (2012) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972825.43 · Cited by 2
Crowdsourcing a Comprehensive Clinical Trial Repository
van Valkenhoef, G., & Kuiper, J. (2014) · University of Groningen research database (University of Groningen / Centre for Information Technology) · Cited by 2
Active Learning for Survival Analysis with Incrementally Disclosed Label Information
Dedja, K., Nakano, F. K., & Vens, C. (2023) · Lirias · Cited by 2
Active Learning for Named Entity Recognition in Clinical Text
Chen, Y. (2015) · Cited by 2
Testing Reading Tactics for Automated Reading Assistance: Is it Useful to Apply Old Knowledge?Preprint
Yu, Z., & Menzies, T. (2017) · arXiv (Cornell University) · Cited by 2
Understanding in vivo modelling of depression
Bannach‐Brown, A. (2019) · ERA · Cited by 2
Active Visual Analytics: Assisted Data Discovery in Interactive Visualizations via Active Search.Preprint
Monadjemi, S., Nguyen, Q., Chai, H., Garnett, R., & Ottley, A. (2020) · arXiv (Cornell University) · Cited by 2
Active Learning : an unbiased approach
Mello, C. E. (2013) · theses.fr (ABES) · doi:10.70675/353a733cz1933z4164zb5a4z8df0e271147d · Cited by 2
“As this procedure introduces an inherent bias in the training set, it may even perform worse than a simple random sampling(WALLACE et al., 2010).”
BackgroundKhokhar, M. J. (2019) · theses.fr (ABES) · doi:10.70675/ebbef3bcz99e6z4794z927dz1e05879191df · Cited by 2
“This phenomenon is referred to as hasty generalization in literature [91] where the learner tends to quickly converge to a final model based on an underlying inaccurate decision boundary.”
BackgroundSimulation-based active learning for systematic reviews: A scoping review of literature
Teijema, J. J., Ribeiro, G., Seuren, S., Anadria, D., Bagheri, A., & van de Schoot, R. (2025) · Journal of Information Science · doi:10.1177/01655515251379058 · Cited by 1
Total Recall and Software Engineering.
Yu, Z. (2020) · NCSU Libraries Repository (North Carolina State University Libraries) · Cited by 1
NLP Applications—Biomedical Literature
Zhang, R., Kastrin, A., Hristovski, D., Fiszman, M., & Kilicoglu, H. (2024) · Cognitive informatics in biomedicine and healthcare · doi:10.1007/978-3-031-55865-8_13 · Cited by 1
Machine learning in health informatics: making better use of domain experts
Brodley, C. E., & Wallace, B. (2012) · Cited by 1
Self-citationCAL-Log: Calibration-Aware Logarithmic Cost Modeling for Active Learning in Low-Resource NLP
Kariyakaranage, V. S., & Athuraliya, B. (2026) · International Research Conference on Smart Computing and Systems Engineering (SCSE) · doi:10.1109/scse70081.2026.11499970 · Cited by 1
“While annotation cost is well-recognized [20], [21], most AL research assumes uniform cost per example.”
A Principled Methodology: A Dozen Principles of Software Effort Estimation
Kocagüneli, E. (2012) · The Research Repository @ WVU (West Virginia University) · doi:10.33915/etd.3601 · Cited by 1
Petersen, H. (2016) · The Sydney eScholarship Repository (The University of Sydney) · Cited by 1
The Intelligent Management of Crowd-Powered Machine Learning
Lin, C. H. (2017) · ResearchWorks at the University of Washington (University of Washington) · Cited by 1
Alluri, S., Komatineni, K., Goli, R., Boyce, R. D., Hubig, N., Min, H., Gong, Y., Sittig, D. F., Robinson, D., Biondich, P., et al. (2025) · medRxiv · doi:10.1101/2025.04.15.25325868 · Cited by 1
“The use of KPAvg in reverse order provides a more uniform document selection process, though the later-stage fluctuations suggest that the diversity of available documents might become a limiting factor over time.”
Menzies, T., Kocagüneli, E., Minku, L. L., Peters, F., & Turhan, B. (2015) · Elsevier eBooks · doi:10.1016/b978-0-12-417295-1.00018-7
EpistAid: Interactive Interface for Document Filtering in Evidence-based Health CarePreprint
Donoso-Guzmán, I., & Parra, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.02119
Discovering Play Store Reviews Related to Specific Android App Issues
Ghosh, A. (2018) · UWSpace (University of Waterloo)
Uma plataforma para revisão automática de literatura via técnicas de Text Mining
Gonçalves, S. G. P. (2018) · Portuguese National Funding Agency for Science, Research and Technology (RCAAP Project by FCT)
Searching for Test Case Prioritization Schemes: An Expert System-Assisted Literature Review.Preprint
Yu, Z., Carver, J. C., Rothermel, G., & Menzies, T. (2019) · arXiv (Cornell University)
Vision based Analysis in Fishery Applications
Wang, G. (2019) · ResearchWorks at the University of Washington (University of Washington)
Applying Active Learning to Biomedical Text Processing
Chen, Y. (2013)
Wisener, L. V. (2021) · The Atrium (University of Guelph) · doi:10.1017/s1466252319000252
Interface Design and Task Difficulty Impact ML-Assisted Visual Data Foraging.Preprint
Monadjemi, S., Ha, S., Nguyen, Q., Chai, H., Garnett, R., & Ottley, A. (2021) · arXiv (Cornell University)
Lee, G. E., & Sun, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.14318
“active learning setting through relevance feedback [8, 33].”
(2015) · Elsevier eBooks · doi:10.1016/b978-0-12-417295-1.09988-4
Seed-driven document ranking for systematic reviews in evidence-based medicine
Lee, E. (2020) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/146709
The model thinks what?! Interpreting deep NLP models with rationales and influence
Jain, S. (2022) · Digital Repository Service (Northeastern University) · doi:10.17760/d20467212
Natural Language Processing Approaches in Bioinformatics
Martinis, M. C., & Chiara, Z. (2024) · Elsevier eBooks · doi:10.1016/b978-0-323-95502-7.00179-2
Abdelkader, W., Xie, D., Lokker, C., Chu, L., Schandelmaier, S., Saha, A., Afzal, M., & Iorio, A. (2025) · medRxiv · doi:10.1101/2025.10.13.25337935
“Active learning is a specialized form of supervised machine learning that allows an algorithm to interactively query a user or another information source to label new data points with desired outputs 17,18 .”
Han, H. W., Kim, J. H., Chio, J., Shim, S. R., Lee, J., & Lee, M. (2025) · doi:10.2196/preprints.87065
Computationally Efficient Active Learning for Large Imbalanced Datasets
Lesci, P., & Vlachos, A. (2024) · Apollo (University of Cambridge) · doi:10.17863/cam.106929
Learning Collections of Functions
Platanios, E. A. (2021) · Figshare · doi:10.1184/r1/13574678
Automated Eligibility Screening and its Evaluation in the Medical Domain
Kusa, W. (2024) · reposiTUm (TU Wien) · doi:10.34726/hss.2024.124620
Big data in medical research: From understanding to improving automation
van Altena, A. J. (2022) · Pure Amsterdam UMC
Active learning aided systematic reviews within highly inclusive datasets
Klopper, T. S. (2021) · Utrecht University Repository (Utrecht University)
Ahn, S., Kim, S., & Valdes, J. (2026) · Springer international handbooks of education · doi:10.1007/978-3-031-88898-4_24-1
A Tri-Level Semantic and LLM-Assisted Methodology for Systematic Literature Reviews (TriSem-LLM)
Carvalho, T., Garcês, S., & Jardim, S. (2026) · IEEE Access · doi:10.1109/access.2026.3700013
“Previous studies have demonstrated that active learning strategies can identify the majority relevant publications while screening only a fraction of the dataset [19].”
Verduin, M. D., Lodi, V. K., & Reis, J. C. S. (2026) · doi:10.5753/sbcas.2026.21687
Lesci, P. (2025) · Apollo (University of Cambridge) · doi:10.17863/cam.131697
LLM-Enhanced Rapid Review (LLMEn-RR): a methodology proposal
Irace, M. B., Simões, J. L., Argôlo, M., Barbosa, C. E., Nóbrega, L., Martinez, L. F., & de Souza, J. M. (2026) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd68734.2026.11582469
“Early AI-assisted review approaches relied on machine learning classifiers and active learning to prioritize relevant articles for human screening, showing reductions in manual effort while maintaining reliability [20]–[22].”
Sánchez‐Piedra, C., Heyerdahl-Viau, I., Garcia-Carpintero, E.-E., Martínez-Núñez, J. M., & Prado-Galbarro, F. J. (2026) · Medical Sciences · doi:10.3390/medsci14050533
“Active learning algorithms trained on human decisions can substantially reduce screening workload while maintaining high sensitivity, enabling review teams to prioritise clinically relevant evidence more efficiently [11,12].”
Fariborzi, H., & Steel, P. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.7388958
Metadata
- Authors
- Wallace, B. C., Small, K., Brodley, C. E., & Trikalinos, T. A.
- Affiliations
- Tufts University (US)
- Tufts Medical Center (US)
- Venue
- Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining
- Publisher
- ACM
- Published
- 2010-07-25
- Pages
- 173-182
- OpenAlex
- W2037771830
- Cited by
- 123
- References
- 34
- Citations by year
- 2026: 5 · 2025: 7 · 2024: 4 · 2023: 10 · 2022: 4 · 2021: 8 · 2020: 13 · 2019: 9 · 2018: 10 · 2017: 10 · 2016: 8 · 2015: 10 · 2014: 7 · 2013: 7 · 2012: 7
- FWCI
- 4.65
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- Funders
- National Science Foundation
- U.S. Social Security Administration
- National Institutes of Health
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- active learning (machine learning), systematic reviews, linear programming, evidence-based practice centers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.