Active learning for biomedical citation screening
The DOI matches its Crossref record; 32 of the 34 references it cites were found in a registry or an index.
Abstract
Active learning (AL) is an increasingly popular strategy for mitigating the amount of labeled data required to train classifiers, thereby reducing annotator effort. We describe a real-world, deployed application of AL to the problem of biomedical citation screening for systematic reviews at the Tufts Medical Center's Evidence-based Practice Center. We propose a novel active learning strategy that exploits a priori domain knowledge provided by the expert (specifically, labeled features)and extend this model via a Linear Programming algorithm for situations where the expert can provide ranked labeled features. Our methods outperform existing AL strategies on three real-world systematic review datasets. We argue that evaluation must be specific to the scenario under consideration. To this end, we propose a new evaluation framework for finite-pool scenarios, wherein the primary aim is to label a fixed set of examples rather than to simply induce a good predictive model. We use a method from medical decision theory for eliciting the relative costs of false positives and false negatives from the domain expert, constructing a utility measure of classification performance that integrates the expert preferences. Our findings suggest that the expert can, and should, provide more information than instance labels alone. In addition to achieving strong empirical results on the citation screening problem, this work outlines many important steps for moving away from simulated active learning and toward deploying AL for real-world applications.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Active learning for systematic review screening promises to reduce the human effort required to identify relevant documents for a systematic review.”
From Statistical stopping criteria for automated screening in systematic reviews (2020) · cites this paper as [5]
Citations
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918 · cited 1×
“We use a modified version of LibSVM [3] and its Python interface.”
MethodThe Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Libsvm : A library for support vector machines
Ferrari, V. (2008) · Medical Entomology and Zoology · Cited by 10,122
Text categorization with Support Vector Machines: Learning with many relevant features
Joachims, T. (1998) · Lecture Notes in Computer Science · doi:10.1007/bfb0026683 · Cited by 7,999 · cited 1×
“We used Support Vector Machines (SVMs) as the base classier [27] due to their good empirical performance over text data [ 12 ] and the Simple [26] method for uncertainty sampling, which selects for labeling those instances closest to the current separating hyperplane.”
MethodDecision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341 · cited 3×
“We borrow a method from medical decision making developed for diagnostic test assessment to infer this weight by means of a thought experiment [28].”
MethodInfluentialCombining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
A Practical Guide to Support Vector Classication
Hsu, C., Chang, C.-C., & Lin, C. (2008) · Cited by 5,083
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
Learning in the Presence of Concept Drift and Hidden Contexts
Widmer, G., & Kubat, M. (1996) · Machine Learning · doi:10.1023/a:1018046501280 · Cited by 1,857 · cited 3×
“[30] G. Widmer and M. Kubat.”
BackgroundInfluentialA sequential algorithm for training text classifiers
Lewis, D., & Gale, W. A. (1994) · arXiv (Cornell University) · doi:10.5555/188490.188495 · Cited by 1,731
Research works printed but not linked: 3
- 10.5555/1699510.1699522 · doi:10.5555/1699510.1699522
- 10.5555/1564131.1564140 · doi:10.5555/1564131.1564140
- O. F. Zaidan , J. Eisner , and C. Piatko . Machine learning with annotator rationales to reduce annotation cost . In NIPS Workshop on Cost Sensitive Learning , December 2008 O. F. Zaidan, J. Eisner, and C. Piatko. Machine learning with annotator rationales to reduce annotation cost. In NIPS Workshop on Cost Sensitive Learning, December 2008
Cited by
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
“How has the workload reduction issue been evaluated? The following section addresses research question 4: How has the workload reduction issue been evaluated? There are three aspects that we explore: what has been compared and through what research design; and what metrics were used to evaluate the performance of the…”
BackgroundMethodInfluentialSettles, B. (2012) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.2200/s00429ed1v01y201207aim018 · Cited by 605
ERASER: A Benchmark to Evaluate Rationalized NLP Models
DeYoung, J., Jain, S., Rajani, N. F., Lehman, E., Xiong, C., Socher, R., & Wallace, B. C. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.408 · Cited by 561
Self-citationAutomation of systematic literature reviews: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106589 · Cited by 432
“…Freire, Gual ´ an and Saquicela [40] describe an algorithm that automates metadata extraction from articles Cost assignment [66] Data resampling [41,42,58,64] SMOTE [43,53] Feature enrichment [43,63] Cost-sensitive classifiers [43,53, 61,66] Precision@95% recall [ published by IEEE, ACM, Springer,…”
MethodSearching for studies: a guide to information retrieval for Campbell systematic reviews
Kugley, S., Wade, A., Thomas, J., Mahood, Q., Jørgensen, A. K., Hammerstrøm, K., & Sathe, N. (2017) · Campbell Systematic Reviews · doi:10.4073/cmg.2016.1 · Cited by 371
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Automating data extraction in systematic reviews: a systematic review
Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0066-7 · Cited by 287
SWIFT-Review: a text-mining workbench for systematic review
Howard, B. E., Phillips, J., Miller, K., Tandon, A., Mav, D., Shah, M. R., Holmgren, S., Pelch, K. E., Walker, V., Rooney, A. A., et al. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0263-z · Cited by 196
“To address these issues, we and others [23, 25 – 27] are moving toward active learning and models that can be initialized without seed studies and then continuously updated during the screening process.”
Active Learning Query Strategies for Classification, Regression, and Clustering: A Survey
Kumar, P., & Gupta, A. (2020) · Journal of Computer Science and Technology · doi:10.1007/s11390-020-9487-4 · Cited by 195
“…tasks Remote sensing image Multi-spectral and hyper-spectral image classification [3,101,102] Medical imaging Classification of medical images [46,164,165] Image segmentation Extract the object of interest and avoid homogeneous regions [166,167] Image classification Multi-label image classification…”
BackgroundReducing systematic review workload through certainty-based screening
Miwa, M., Thomas, J., O’Mara-Eves, A., & Ananiadou, S. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.06.005 · Cited by 190
“Here, b is a constant that represents the relative importance of Yield, in comparison to Burden, and it is set to 19, following [24].”
BackgroundBannach-Brown, A., Przybyła, P., Thomas, J., Rice, A. S. C., Ananiadou, S., Liao, J., & Macleod, M. R. (2019) · Systematic Reviews · doi:10.1186/s13643-019-0942-7 · Cited by 185
“screened, chosen either through random sampling or, using active learning [10], on the basis either of those with highest uncertainty of predictions [11, 12] or alter-”
MethodShemilt, I., Simon, A., Hollands, G. J., Marteau, T. M., Ogilvie, D., O'Mara‐Eves, A., Kelly, M. P., & Thomas, J. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1093 · Cited by 170
“The output from AC differs from that obtained using ATR because rather than an ordered list of title-abstract records, AC automatically classifies title-abstract records as being relevant or not relevant (Wallace et al., 2010b).”
BackgroundMethodHoward, B. E., Phillips, J., Tandon, A., Maharana, A., Elmore, R., Mav, D., Sedykh, A., Thayer, K., Merrick, B. A., Walker, V., et al. (2020) · Environment International · doi:10.1016/j.envint.2020.105623 · Cited by 157
Active Discriminative Text Representation Learning
Zhang, Y., Lease, M., & Wallace, B. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10962 · Cited by 156
“There has been a wealth of work on AL approaches for traditional machine learning methods in general (Settles 2010), and for text classification in particular (Tong and Koller 2002; McCallumzy and Nigamy 1998; Wallace et al. 2010).”
BackgroundSelf-citationBarratt, H., Rojas-García, A., Clarke, K., Moore, A., Whittington, C., Stockton, S., Thomas, J., Pilling, S., & Raine, R. (2016) · PLOS ONE · doi:10.1371/journal.pone.0154449 · Cited by 117
“We employed a text mining and machine learning method, known as ‘active learning’,[8] using the systematic review software EPPI-Reviewer 4 to screen titles and abstracts.”
MethodLarge-scale live active learning: Training object detectors with crawled data and crowds
Vijayanarasimhan, S., & Grauman, K. (2011) · CVPR 2011 · doi:10.1109/cvpr.2011.5995430 · Cited by 111
Statistical stopping criteria for automated screening in systematic reviews
Callaghan, M. W., & Müller-Hansen, F. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01521-4 · Cited by 121
Kocaguneli, E., Menzies, T., Keung, J., Cok, D., & Madachy, R. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.88 · Cited by 97
“Wallace et al. [15] used active learning for a practical application example.”
BackgroundSupporting systematic reviews using LDA-based document representations
Mo, Y., Kontonatsios, G., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0117-0 · Cited by 83
“To overcome these limitations, methods such as machine learning, text mining [9, 10], text classification [11] and active learning [6, 12] have been used to partially automate this process, in order to reduce the workload, without sacrificing the quality of the reviews.”
MethodScalability of Continuous Active Learning for Reliable High-Recall Text Classification
Cormack, G. V., & Grossman, M. R. (2016) · Proceedings of the 25th ACM International on Conference on Information and Knowledge Management · doi:10.1145/2983323.2983776 · Cited by 78
“The same problem has been observed within the context of systematic review in evidence-based medicine [22], but the literature has limited discussion of technological solutions to this problem [6, 40].”
BackgroundKhabsa, M., Elmagarmid, A., Ilyas, I., Hammady, H., & Ouzzani, M. (2016) · Machine Learning · doi:10.1007/s10994-015-5535-7 · Cited by 98
“…active learning to select which articles to tag reported needing up to 30–40 % of the entire dataset to be able to achieve recall as high as 1 (Wallace et al. 2010b) (Albeit working with a different dataset that contains three SRs only, their results about the required percentage of training…”
BackgroundFinding better active learners for faster literature reviews
Yu, Z., Kraft, N. A., & Menzies, T. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9587-0 · Cited by 83
“– Review those active learning methods [10, 11, 45, 58, 60] and find that in evidence-based medicine and legal electronic discovery, there are three widely recognized state-of-the-art active learning methods [10, 38, 60].”
BackgroundMethodCombining Crowd and Expert Labels Using Decision Theoretic Active Learning
Nguyen, A., Wallace, B., & Lease, M. (2015) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v3i1.13225 · Cited by 73
“We investigate finite-pool data categorization (Wallace et al. 2010a), in which the objective is to exhaustively and accurately categorize a set of examples while minimizing cost.”
BackgroundMethodSelf-citationFAST2: An intelligent assistant for finding relevant papers
Yu, Z., & Menzies, T. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.11.021 · Cited by 80
“Yu et al. [41] tested various fast-reading text miners from medicine [22, 36, 38] and legal reasoning [6–8 , 13 ].”
BackgroundWho Should Label What? Instance Allocation in Multiple Expert Active Learning
Wallace, B. C., Small, K., Brodley, C. E., & Trikalinos, T. A. (2011) · Proceedings of the 2011 SIAM International Conference on Data Mining · doi:10.1137/1.9781611972818.16 · Cited by 63
“Indeed, this is in line with our previous observations that uncertainty-sampling AL can perform poorly when there is significant class imbalance [20].”
BackgroundMethodResultInfluentialSelf-citationImproving Vulnerability Inspection Efficiency Using Active Learning
Yu, Z., Theisen, C., Williams, L. A., & Menzies, T. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2949275 · Cited by 69
“The experience in other domains is that such active learners can significantly reduce the amount of effort required to achieve high recall [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24].”
BackgroundToward modernizing the systematic review pipeline in genetics: efficient updating via data mining
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., Schmid, C. H., Bertram, L., Lill, C. M., Cohen, J. T., & Trikalinos, T. A. (2012) · Genetics in Medicine · doi:10.1038/gim.2012.7 · Cited by 62
Self-citationBetter Data Labelling With EMBLEM (and how that Impacts Defect Prediction)
Tu, H., Yu, Z., & Menzies, T. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2986415 · Cited by 62
“achieve high recall in electronic discovery, evidence-based medicine, and reading research papers [21], [22], [23], [24], [25], [99], [100], [101], [102], [103], [114], [116].”
BackgroundAutomating Risk of Bias Assessment for Clinical Trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2015) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2015.2431314 · Cited by 59
Self-citationTERMINATOR: better automated UI test case prioritization
Yu, Z., Fahid, F. M., Menzies, T., Rothermel, G., Patrick, K., & Cherian, S. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3340448 · Cited by 58
“supervised learners and can significantly reduce the effort required to achieve high recall [7–11, 19, 50–54, 56, 57].”
BackgroundModernizing the systematic review process to inform comparative effectiveness: tools and methods
Wallace, B. C., Dahabreh, I. J., Schmid, C. H., Lau, J., & Trikalinos, T. A. (2013) · Journal of Comparative Effectiveness Research · doi:10.2217/cer.13.17 · Cited by 51
Self-citationOlorisade, B. K., Brereton, P., & Andras, P. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.07.010 · Cited by 54
Seed-driven Document Ranking for Systematic Reviews in Evidence-Based Medicine
Lee, G. E., & Sun, A. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3209994 · Cited by 48
“To train classifiers, many machine learning algorithms were examined including SVM and LDA [8, 25].”
MethodSharing Data and Models in Software Engineering
Menzies, T., Kocagüneli, E., Turhan, B., Minku, L. L., & Peters, F. (2014) · Cited by 47
A semi-supervised approach using label propagation to support citation screening
Kontonatsios, G., Brockmeier, A. J., Przybyła, P., McNaught, J., Mu, T., Goulermas, J. Y., & Ananiadou, S. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.06.018 · Cited by 42
“[43] noted that, according to experts, yield”
Carvallo, A., Parra, D., Löbel, H., & Soto, Á. (2020) · Scientometrics · doi:10.1007/s11192-020-03648-6 · Cited by 44
The Constrained Weight Space SVM: Learning with Ranked Features
Small, K., Wallace, B., Trikalinos, T. A., & Brodley, C. E. (2011) · International Conference on Machine Learning · Cited by 44
Self-citationOlorisade, B. K., de Quincey, E., Brereton, P., & Andras, P. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2915982 · Cited by 42
“These facts might have accounted for the performance recorded and substantial use of SVM in the studies.”
InfluentialUncertainty-Based Active Learning via Sparse Modeling for Image Classification
Wang, G., Hwang, J., Rose, C. S., & Wallace, F. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2867913 · Cited by 42
“Sometimes uncertainty sampling even works worse than random sampling in scenarios when very limited labeled data are used to train the initial classifier [20]–[22].”
BackgroundAppraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews
Yun, H., Marshall, I. J., Trikalinos, T. A., & Wallace, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.626 · Cited by 41
Self-citationFlores, C. A., Figueroa, R. L., & Pezoa, J. E. E. (2021) · IEEE Access · doi:10.1109/access.2021.3064000 · Cited by 39
“To address this problem, the AL provides an alternative to reduce the number of examples needed to train the classification algorithms [25].”
MethodA Multi-Channel Convolutional Neural Network approach to automate the citation screening process
van Dinter, R., Catal, C., & Tekinerdoğan, B. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107765 · Cited by 36
Semi-Automated evidence synthesis in health psychology: current methods and future prospects
Marshall, I. J., Johnson, B. T., Wang, Z., Rajasekaran, S., & Wallace, B. (2020) · Health Psychology Review · doi:10.1080/17437199.2020.1716198 · Cited by 34
“For evaluating classi fi er performance in the context of citation screening (as described above), custom metrics have been de fi ned that attempt to better capture the speci fi c trade-o ff s involved (Cohen et al., 2006; Wallace, Small, Brodley, & Trikalinos, 2010).”
MethodSelf-citationActive Learning with Unbalanced Classes and Example-Generation Queries
Lin, C., Mausam, M., & Weld, D. S. (2018) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v6i1.13334 · Cited by 32
“Other approaches use multiple clas-sifiers to choose the next examples to label (Wallace et al. 2010; Li et al. 2012).”
BackgroundHerde, M., Huseljic, D., Sick, B., & Calma, A. (2021) · IEEE Access · doi:10.1109/access.2021.3135514 · Cited by 33
“[112] To which class does instance xn ∈ X not belong? distinct annotations: ΩZ = P(ΩY )”
BackgroundChallenges and opportunities in applied machine learning
Brodley, C. E., Rebbapragada, U., Small, K., & Wallace, B. C. (2012) · AI Magazine · doi:10.1609/aimag.v33i1.2367 · Cited by 32
“Second, we developed a novel active learning algorithm that works well under class imbalance by exploiting domain knowledge in the form of labeled terms to guide the active learning process (Wallace et al. 2010).”
BackgroundSelf-citationAutomating risk of bias assessment for clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2014) · Proceedings of the 5th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics · doi:10.1145/2649387.2649406 · Cited by 28
Self-citationEvaluating sentence-level relevance feedback for high-recall information retrieval
Zhang, H., Cormack, G. V., Grossman, M. R., & Smucker, M. D. (2019) · Information Retrieval · doi:10.1007/s10791-019-09361-0 · Cited by 31
“Prior to CLEF 2017, the systematic review literature described primarily text categorization efforts Track, in which the effective-ness evaluated ranking of the test set (Hersh and Bhupatiraju, 2003; Wallace et al, 2010, 2013).”
MethodKlemme, F., & Amrouch, H. (2022) · IEEE Transactions on Circuits and Systems I Regular Papers · doi:10.1109/tcsi.2022.3201431 · Cited by 30
“1. outperforming active learning strategies [11], [15], [16], [17].”
BackgroundClustering based active learning for biomedical Named Entity Recognition
Han, X., Kwoh, C. K., & Kim, J. (2016) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2016.7727341 · Cited by 30
“It has been shown, however, that the uncertainty-based approach may have worse performance than random selection [22], [23], [24].”
BackgroundA decision support system for automating document retrieval and citation screening
van Dinter, R., Catal, C., & Tekinerdoğan, B. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115261 · Cited by 26
Lechtenberg, F., Farreres, J., Galvan-Cara, A.-L., Somoza-Tornos, A., Espuña, A., & Graells, M. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.116967 · Cited by 26
Modeling annotation time to reduce workload in comparative effectiveness reviews
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2010) · Proceedings of the 1st ACM International Health Informatics Symposium · doi:10.1145/1882992.1882999 · Cited by 25
Self-citationLu, J., & Namee, B. M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.13138 · Cited by 25
“This measures the performance of the full active learning system including human annotators which is prevalent in active learning community (Wallace et al., 2010; Hashimoto et al., 2016).”
BackgroundInfluentialBannach‐Brown, A., Przybyła, P., Thomas, J., Rice, A. S. C., Ananiadou, S., Liao, J., & Macleod, M. R. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/255760 · Cited by 24
Yun, H. S., Pogrebitskiy, D., Marshall, I. J., & Wallace, B. (2024) · PubMed · doi:10.48550/arxiv.2405.01686 · Cited by 23
Self-citationCan Active Learning Preemptively Mitigate Fairness Issues?Preprint
Branchaud-Charron, F., Atighehchian, P., Rodríguez, P., Abuhamad, G., & Lacoste, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.06879 · Cited by 22
“, 2020), high-skew AL methods (Vijayanarasimhan & Grauman, 2014; Wallace et al., 2010; Aggarwal et al., 2020), and resampling methods (Amini et al.”
MethodKusa, W., Lipani, A., Knoth, P., & Hanbury, A. (2023) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2023.200193 · Cited by 21
“Burden , utility and coverage were introduced for evaluating active learning models in the context of citation screening (Wallace et al., 2010, Miwa et al., 2014).”
CSMeD: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviews
Kusa, W., Mendoza, Ó. E., Samwald, M., Knoth, P., & Hanbury, A. (2023) · neural information processing systems · doi:10.52202/075280-1017 · Cited by 21
AnchorAL: Computationally Efficient Active Learning for Large and Imbalanced Datasets
Lesci, P., & Vlachos, A. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.467 · Cited by 20
“…makes assumptions that cannot be true in real-world settings (Margatina and Ale-tras, 2023); for example, a perfect oracle, uniform annotation difficulty, and the possibility to monitor the performance of the AL strategy on a test set while training (Wallace et al., 2010; Levonian et al., 2022).”
BackgroundASReview LAB Software Documentation
Bruin, J. D., Ferdinands, G., Harkema, A., de Boer, J., van den Brand, S. A. G. E., Ma, Y., & van de Schoot, R. (2020) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.4287119 · Cited by 20
Semi-automated Tools for Systematic Searches
Adam, G. P., Wallace, B. C., & Trikalinos, T. A. (2022) · Methods in Molecular Biology · doi:10.1007/978-1-0716-1566-9_2 · Cited by 19
Self-citationLearning-to-Rank and Relevance Feedback for Literature Appraisal in Empirical Medicine
Lagopoulos, A., Anagnostou, A., Minas, A., & Tsoumakas, G. (2018) · Lecture notes in computer science · doi:10.1007/978-3-319-98932-7_5 · Cited by 18
“These approaches are based on text mining [31,20,11] along with active learning [8,30] or learning-to-rank [22].”
BackgroundOutcome-based Evaluation of Systematic Review Automation
Kusa, W., Zuccon, G., Knoth, P., & Hanbury, A. (2023) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3578337.3605135 · Cited by 17
Active Literature Discovery for Scoping Evidence Reviews How Many Needles are There
Wallace, B., Dahabreh, I. J., Moran, K. H., Brodley, C. E., & Trikalinos, T. A. (2013) · Cited by 17
Self-citationSercombe, J., Bryant, Z., & Wilson, J. (2025) · JMIR Formative Research · doi:10.2196/68666 · Cited by 16
“Machine learning classifiers, such as support vector machines or complement naive Bayes have been trained to replicate human inclusion/exclusion decisions based on label training data [10,11].”
AI-Based Literature Reviews: A Topic Modeling Approach
Verma, M. K., & Yuvaraj, M. (2023) · Journal of Information and Knowledge · doi:10.17821/srels/2023/v60i2/170967 · Cited by 16
“It has been found that a seasoned reviewer can screen about two abstracts per minute on average, but more complex abstracts can take much longer (Wallace et al., 2010).”
BackgroundReducing the user labeling effort in effective high recall tasks by fine-tuning active learning
Bianco, G. D., Duarte, D., & Gonçalves, M. A. (2023) · Journal of Intelligent Information Systems · doi:10.1007/s10844-022-00772-y · Cited by 15
“Active learning fits perfectly into the HIRE problem, as ideally, only the most informative documents would be presented to the user for labeling, drastically reducing the manual cost of the process (Cormack & Grossman, 2014; Wallace et al. 2010).”
MethodPetersen, H., Poon, J., Poon, S. K.-S., & Loy, C. T. (2014) · JMIR Medical Informatics · doi:10.2196/medinform.3037 · Cited by 15
“In contrast, Wallace et al [7, 27, 28] describe an active learning approach, where the classifier is built in an iterative process.”
Akinseloyin, O., Jiang, X., & Palade, V. (2026) · Biology Methods and Protocols · doi:10.1093/biomethods/bpag006 · Cited by 14
“Active learning sought to alleviate these issues through iterative human feedback [16–19], but faced challenges in achieving a high degree of automation due to the zero-shot nature of this problem [20, 21].”
Modernizing Evidence Synthesis for Evidence-Based Medicine
Wallace, B., Dahabreh, I. J., Schmid, C. H., Lau, J., & Trikalinos, T. A. (2014) · Elsevier eBooks · doi:10.1016/b978-0-12-398476-0.00012-9 · Cited by 14
Self-citationHan, X., Kim, J., & Kwoh, C. K. (2016) · Journal of Biomedical Semantics · doi:10.1186/s13326-016-0059-z · Cited by 13
“It has been shown, however, that the uncertainty-based approach may have worse performance than random selection [13–15].”
BackgroundData Balancing for Technologically Assisted Reviews: Undersampling or Reweighting.
Yu, Z., & Menzies, T. (2017) · CLEF (Working Notes) · Cited by 13
Predicting more from less: Synergies of learning
Kocagüneli, E., Čukić, B., & Lu, H. (2013) · International Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE) · doi:10.1109/raise.2013.6615203 · Cited by 12
Paul, N. R., Balabantaray, R. C., & Sahoo, D. (2023) · SN Computer Science · doi:10.1007/s42979-023-02061-z · Cited by 11
On Active Sampling of Controlled Experiments for QoE Modeling
Khokhar, M. J., Saber, N. A., Spetebroot, T., & Barakat, C. (2017) · doi:10.1145/3098603.3098609 · Cited by 11
Simulation-based Active Learning for Systematic Reviews: A Scoping Review of Literature
Teijema, J. J., Ribeiro, G., Seuren, S., Anadria, D., Bagheri, A., & van de Schoot, R. (2023) · doi:10.31234/osf.io/67zmt · Cited by 10
An intelligent sampling framework for controlled experimentation and QoE modeling
Khokhar, M. J., Saber, N. A., Spetebroot, T., & Barakat, C. (2018) · Computer Networks · doi:10.1016/j.comnet.2018.10.011 · Cited by 10
“This phenomenon is referred to as hasty generalization in literature [10] where the learner tends to quickly converge to a final model based on an underlying inaccurate decision boundary.”
Anagnostou, A., Lagopoulos, A., Tsoumakas, G., & Vlahavas, I. P. (2017) · CLEF (Working Notes) · Cited by 10
Lu, J., Henchion, M. M., & Namee, B. M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.03505 · Cited by 10
“Bag-of-words (BOW) is the most basic representation scheme for documents, and has been widely used in many active learning applications [8,10,11,12,22,23].”
Backgroundde Andrade, P. H. M. A. (2015) · RIUnB Institutional Repository (University of Brasília) · doi:10.26512/2015.09.d.21004 · Cited by 9
Large-Scale Simulation Study of Active Learning Models for Systematic Reviews
Teijema, J. J., de Bruin, J., Bagheri, A., & van de Schoot, R. (2023) · doi:10.31234/osf.io/2w3rm · Cited by 8
Epistemological aspect of topic modelling in the social sciences: Latent Dirichlet Allocation
Baranowski, M. (2022) · Przegląd Krytyczny · doi:10.14746/pk.2022.4.1.1 · Cited by 8
Systematic review automation methods
Norman, C. R. (2020) · theses.fr (ABES) · doi:10.70675/044ffa34z5c9bz433az99c8z64697dcd28a2 · Cited by 8
MMiDaS-AE: multi-modal missing data aware stacked autoencoder for biomedical abstract screening
Lee, E. W., Wallace, B., Galaviz, K. I., & Ho, J. C. (2020) · ACM Conference on Health, Inference, and Learning (CHIL) · doi:10.1145/3368555.3384463 · Cited by 7
“Because supervision is expensive for this task, and a new model must be trained for each new review, a common strategy explored is active learning [14, 28, 34, 49, 50] in which the learner starts with a small subset of manually labeled records, which are used to train the initial classifier.”
BackgroundSelf-citationGraph-Sparse LDA: A Topic Model with Structured SparsityPreprint
Doshi‐Velez, F., Wallace, B., & Adams, R. P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.4510 · Cited by 7
“for reducing the labor involved in this process have therefo re been investigated [18, 19].”
BackgroundSelf-citationLLM-Guided Weak Supervision Improves Active Learning for Abstract ScreeningPreprint
Akinseloyin, O., Jiang, X., & Palade, V. (2025) · medRxiv · doi:10.1101/2025.08.24.25334314 · Cited by 7
“Wallace et al. (2010) query unlabeled documents using a uncertainty-based method, which prioritise instances closest to the classifier’s decision boundary, assuming that solving these hard samples improves model performance.”
InfluentialAssessing expert system-assisted literature reviews with a case study
Yu, Z., Carver, J. C., Rothermel, G., & Menzies, T. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.116958 · Cited by 6
VoMBaT: A Tool for Visualising Evaluation Measure Behaviour in High-Recall Search Tasks
Kusa, W., Lipani, A., Knoth, P., & Hanbury, A. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591802 · Cited by 6
“, Fβ–score, Yield, Burden [23], Utilityβ [22], sensitivity-maximising thresholds [7], and AUC [2].”
Normalised Precision at Fixed Recall for Evaluating TAR
Kusa, W., Peikos, G., Staudinger, M., Lipani, A., & Hanbury, A. (2024) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3664190.3672532 · Cited by 6
An Online Sampling Approach for Controlled Experimentation and QoE Modeling
Khokhar, M. J., Spetebroot, T., & Barakat, C. (2018) · International Conference on Communications · doi:10.1109/icc.2018.8422954 · Cited by 6
“This phenomenon is referred to as hasty generalization in literature [10] where the learner tends to quickly converge to a final model based on an underlying inaccurate decision boundary.”
BackgroundTang, F.-S. K.-B., Bukowski, M., Schmitz‐Rode, T., & Farkas, R. (2023) · Applied Sciences · doi:10.3390/app13137639 · Cited by 5
A large scale study of SVM based methods for abstract screening in systematic reviews
Saha, T. K., Ouzzani, M., Hammady, H. M., Elmagarmid, A. K., Dhifli, W., & Hasan, M. A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.00192 · Cited by 5
“We use β = 19 in our experimental evaluations following the suggestion from [18].”
MethodYu, Z., Carver, J. C., Rothermel, G., & Menzies, T. (2019) · Cited by 4
Modernizing evidence synthesis for evidence-based medicine
Saldanha, I. J., Adam, G. P., Schmid, C. H., Trikalinos, T. A., & Konnyu, K. J. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-91200-6.00006-1 · Cited by 4
Self-citationBoolean and Ranked Information Retrieval for Biomedical Systematic Reviewing
Pohl, S. (2012) · Minerva Access (University of Melbourne) · Cited by 4
What Does the Evidence Say? Models to Help Make Sense of the Biomedical Literature
Wallace, B. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/899 · Cited by 4
“…clinical question, are a particularly natural fit here; empirical results suggest that methods can substantially reduce the workload involved in comprehensive retrieval of literature relevant to a particular clinical question [Wallace et al., 2010a; Przybyła et al., 2018; O’Mara-Eves et al., 2015].”
Self-citationLagopoulos, A., & Tsoumakas, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.09752 · Cited by 4
“Later studies evaluated active learning techniques to deal with class imbalance (Wallace et al., 2010a,b; Miwa et al., 2014) and several studies exploited the advantages of using the Naive Bayes algorithm (Bekhuis and Demner-Fushman, 2012; Matwin et al., 2010; Frunza et al., 2010, 2011).”
SyROCCo: enhancing systematic reviews using machine learning
Fang, Z., Arana‐Catania, M., van Lier, F.-A., Velarde, J. O., Bregazzi, H., Airoldi, M., Carter, E., & Procter, R. (2024) · Data & Policy · doi:10.1017/dap.2024.33 · Cited by 3
“Several strategies have been proposed to improve the accuracy of text classification models in systematic reviews (Wallace et al., 2010; Wallace et al., 2012; Miwa et al., 2014; Kontonatsios et al., 2017; Liu et al., 2018), all of which focus on screening more records to supplement the training set.”
BackgroundActive Learning amidst Logical KnowledgePreprint
Platanios, E. A., Kapoor, A., & Horvitz, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.08850 · Cited by 3
“Other methods that use “side” information in active learning include those of Kapoor and Baker (2009), Wallace et al. (2010), and Angeli et al. (2014).”
Method
Metadata
- Authors
- Wallace, B. C., Small, K., Brodley, C. E., & Trikalinos, T. A.
- Affiliations
- Tufts University (US)
- Tufts Medical Center (US)
- Venue
- Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining
- Publisher
- ACM
- Published
- 2010-07-25
- Pages
- 173-182
- OpenAlex
- W2037771830
- Cited by
- 123
- References
- 34
- Citations by year
- 2026: 5 · 2025: 7 · 2024: 4 · 2023: 10 · 2022: 4 · 2021: 8 · 2020: 13 · 2019: 9 · 2018: 10 · 2017: 10 · 2016: 8 · 2015: 10 · 2014: 7 · 2013: 7 · 2012: 7
- FWCI
- 4.65
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (publishedVersion)
- Funders
- National Science Foundation
- U.S. Social Security Administration
- National Institutes of Health
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- active learning (machine learning), systematic reviews, linear programming, evidence-based practice centers
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.