Semi-automated screening of biomedical citations for systematic reviews
Open accessThe DOI matches its Crossref record; 44 of the 45 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Systematic reviews address a specific clinical question by unbiasedly assessing and analyzing the pertinent literature. Citation screening is a time-consuming and critical step in systematic reviews. Typically, reviewers must evaluate thousands of citations to identify articles eligible for a given review. We explore the application of machine learning techniques to semi-automate citation screening, thereby reducing the reviewers' workload. RESULTS: We present a novel online classification strategy for citation screening to automatically discriminate "relevant" from "irrelevant" citations. We use an ensemble of Support Vector Machines (SVMs) built over different feature-spaces (e.g., abstract and title text), and trained interactively by the reviewer(s). Semi-automating the citation screening process is difficult because any such strategy must identify all citations eligible for the systematic review. This requirement is made harder still due to class imbalance; there are far fewer "relevant" than "irrelevant" citations for any given systematic review. To address these challenges we employ a custom active-learning strategy developed specifically for imbalanced datasets. Further, we introduce a novel undersampling technique. We provide experimental results over three real-world systematic review datasets, and demonstrate that our algorithm is able to reduce the number of citations that must be screened manually by nearly half in two of these, and by around 40% in the third, without excluding any of the citations eligible for the systematic review. CONCLUSIONS: We have developed a semi-automated citation screening algorithm for systematic reviews that has the potential to substantially reduce the number of citations reviewers have to manually screen, without compromising the quality and comprehensiveness of the review.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


![Figure 3 Construction of confusion matrices for the semi-automated abstract screening strategy. The leftmost matrix represents citations that are labeled by the reviewer while training the classification model. The middle matrix displays the predictions of the trained model over the remaining unlabeled set of citations U. The rightmost matrix shows the corresponding crosstabulation at the end of "Level 1a" (see Figure 1). The quantities mentioned in this figure are used in the definition of Yield and Burden, the chosen evaluation metrics (see Equations 1 and 2). Superscripts T and U refer to model training and applying the model to yet unlabeled citations, respectively. tp[T|U]: "true positives", tn[T|U]: "true negatives", fp[T|U]: "false positives", fn [T|U]: "false negatives". We assume that reviewers will never erroneously exclude a citation that is eligible for systematic review, i.e. fnT = 0.](https://figures.cochat.ai/paper-figures/dddd0ade76024cfbbd2b1f727fda5c6c/semi-automated-screening-of-biomedical-citations-for-systematic-reviews-figure-3.jpg)



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Several statistical stopping metrics have been proposed in the literature [15, 21, 22, 35, 38, 49, 50, 53, 55]).”
From The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses (2024) · cites this paper as [50]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These include computing an inflection point [16, 21, 35, 38, 49, 50], estimating recall for the sequential screening of a ranked list of references [22], or computing the lengths of consecutive spans of excluded documents that occur between each relevant document during screening [38, 53, 55].”
From The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses (2024) · cites this paper as [50]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For example, Wallace et al. (2010) found that using a time-based approach in which reviewers classified 50% of all records in a database resulted in a 100% recall.”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Wallace et al. (2010)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Lastly, researchers have an ongoing uncertainty regarding the optimal number of articles to screen using these tools, and the existing simulation studies have yielded limited and conflicting findings (Callaghan & MüllerHansen, 2020; Wallace et al., 2010).”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Wallace et al., 2010
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, classifiers can achieve high accuracy but still fail to identify any of the relevant publications [15].”
From Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records (2023) · cites this paper as [15]
Experimental — accuracy not yet measured
10 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918 · cited 1×
“Our code is written in Python and makes use of a modified version of LibSVM [39].”
The Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325 · cited 1×
“One obvious technique that we plan on implementing is bagging [36], in which multiple subsets of each feature-space would be constructed at random from the labeled data, and used to build ensemble classifiers over each feature-space.”
MethodProceedings of the 24th international conference on Machine learning
(2007) · doi:10.1145/1273496 · Cited by 11,630
Libsvm : A library for support vector machines
Ferrari, V. (2008) · Medical Entomology and Zoology · Cited by 10,122
Text categorization with Support Vector Machines: Learning with many relevant features
Joachims, T. (1998) · Lecture Notes in Computer Science · doi:10.1007/bfb0026683 · Cited by 7,999 · cited 1×
“We use SVMs because they have empirically performed well over high-dimensional text data in general [21], and in the context of biomedical text classification in particular [9,10].”
Combining labeled and unlabeled data with co-training
Blum, A., & Mitchell, T. (1998) · Proceedings of the eleventh annual conference on Computational learning theory · doi:10.1145/279943.279962 · Cited by 5,495
A Practical Guide to Support Vector Classication
Hsu, C., Chang, C.-C., & Lin, C. (2008) · Cited by 5,083
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
A STATISTICAL INTERPRETATION OF TERM SPECIFICITY AND ITS APPLICATION IN RETRIEVAL
SPARCK JONES, K. (1972) · Journal of Documentation · doi:10.1108/eb026526 · Cited by 4,505 · cited 1×
“use standard Term Frequency/Inverse Document Frequency (TF-IDF) encoding [25] to generate “bag-ofwords” representations of each citation in the respective feature-spaces.”
A sequential algorithm for training text classifiers
Lewis, D., & Gale, W. A. (1994) · arXiv (Cornell University) · doi:10.5555/188490.188495 · Cited by 1,731
Improving generalization with active learning
Cohn, D., Atlas, L., & Ladner, R. (1994) · Machine Learning · doi:10.1007/bf00993277 · Cited by 1,591 · cited 2×
“This method has been shown to work well empirically [18], and we use it as the foundation of our approach.”
Handling imbalanced datasets: A review
Kotsiantis, S. B., Kanellopoulos, D. N., & Pintelas, P. (2006) · Cited by 1,030
Proceedings of the Seventeenth International Conference on Machine Learning
Langley, P. (2000) · International Conference on Machine Learning · Cited by 550
Formulating Questions and Locating Primary Studies for Inclusion in Systematic Reviews
Counsell, C. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-5-199709010-00008 · Cited by 505 · cited 1×
“Like all scientific approaches, a systematic review tries to address a well-formulated research question by following a protocol of well-defined steps [1,2].”
BackgroundIncorporating diversity in active learning with support vector machines
Brinker, K. (2003) · Cited by 458
Learning from Imbalanced Data Sets: A Comparison of Various Strategies *
Japkowicz, N. (2000) · Cited by 455
Hierarchical sampling for active learning
Dasgupta, S., & Hsu, D. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390183 · Cited by 444 · cited 1×
“Finally, exploiting latent hierarchical structure in the UMLS ontologies might also be helpful, particularly because active learning techniques for hierarchical data have recently been developed [38].”
BackgroundReducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 1×
“demonstrated that machine learning techniques can reduce the labor required to update systematic reviews [16].”
BackgroundSupport Vector Machine Active Learning with Application sto Text Classification
Tong, S., & Koller, D. (2000) · International Conference on Machine Learning · Cited by 369
Blanke, T., & Lalmas, M. (2006) · Cited by 312
Biomedical Language Processing: What's Beyond PubMed?
Hunter, L., & Cohen, K. B. (2006) · Molecular Cell · doi:10.1016/j.molcel.2006.02.012 · Cited by 308 · cited 1×
“Due to the exponential growth of available biomedical literature [6], much work has been done in automatically mining and learning from published manuscripts.”
BackgroundFrontiers of biomedical text mining: current progress
Zweigenbaum, P., Demner-Fushman, D., Yu, H., & Cohen, K. B. (2007) · Briefings in Bioinformatics · doi:10.1093/bib/bbm045 · Cited by 282 · cited 2×
“(In any case, the choice of classification algorithm seems to matter less than the choice of features when working with biomedical texts [22].”
BackgroundText Categorization Models for High-Quality Article Retrieval in Internal Medicine
Aphinyanaphongs, Y. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1641 · Cited by 177 · cited 3×
“( An alternative to this approach would be to join the feature-spaces into a single feature-space, concatenating the the respective points for a given document to form a single point in the combined space [10].”
BackgroundMethodActive learning for class imbalance problem
Ertekin, S., Huang, J., & Giles, C. L. (2007) · Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1277741.1277927 · Cited by 166 · cited 1×
“The idea is to explicitly push the decision boundary away from the minority class, as it has been observed that when there is class imbalance, SVMs are prone to discovering hyperplanes that are closer to the minority class than the ideal separating boundary, resulting in false negatives [31].”
BackgroundActive learning by labeling features
Druck, G., Settles, B., & McCallum, A. (2009) · Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing Volume 1 - EMNLP '09 · doi:10.3115/1699510.1699522 · Cited by 162 · cited 1×
“We are also interested in the emerging work on active learning over features rather than instances [37], which may be helpful in identifying phrases or UMLS concepts particularly characteristic of relevant citations.”
BackgroundSystematic Review: Charged-Particle Radiation Therapy for Cancer
Terasawa, T., Dvorak, T., Ip, S., Raman, G., Lau, J., & Trikalinos, T. A. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-8-200910200-00145 · Cited by 104
Self-citationTowards Automatic Recognition of Scientifically Rigorous Clinical Research Evidence
Kilicoglu, H., Demner-Fushman, D., Rindflesch, T. C., Wilczynski, N. L., & Haynes, R. B. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2996 · Cited by 101 · cited 6×
“and employ different classification algorithms over the different featurespaces [8] (here we used only SVMs).”
BackgroundMethodInfluentialAMIA 2007 Symposium Proceedings
Leegon, J., Hoot, N. R., Aronsky, D., & Storkey, A. (2006) · Cited by 85
Multimodal concept-dependent active learning for image retrieval
Goh, K.-S., Chang, E. Y., & Lai, W.-C. (2004) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1027527.1027664 · Cited by 81 · cited 1×
“However, this combined representation can be problematic for active learning [26].”
The effect of feature representation on MEDLINE document classification.
Yetisgen-Yildiz, M., & Pratt, W. (2005) · PubMed · Cited by 68
Better rules, fewer features: a semantic approach to selecting features from text
Blake, C. L., & Pratt, W. (2002) · Proceedings · doi:10.1109/icdm.2001.989501 · Cited by 61 · cited 2×
“The representation of biomedical documents can have a large effect on classification performance [8,13-15] and is arguably more important than the choice of the classification algorithm itself [22].”
BackgroundYu, W., Clyne, M., Dolan, S. M., Yesupriya, A., Wulf, A., Liu, T., Khoury, M. J., & Gwinn, M. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-205 · Cited by 52 · cited 1×
“Here we briefly review the emerging body of promising research on applications of machine learning methods to biomedical text classification [7-11], particularly those works focused on automatic classification of biomedical abstracts into clinically relevant categories.”
BackgroundReporting of systematic reviews of micronutrients and health: a critical appraisal
Chung, M., Balk, E. M., Ip, S., Raman, G., Yu, W. W., Trikalinos, T. A., Lichtenstein, A. H., Yetley, E. A., & Lau, J. (2009) · American Journal of Clinical Nutrition · doi:10.3945/ajcn.2008.26821 · Cited by 47 · cited 2×
“The Micronutrients dataset is from a systematic empirical appraisal of reporting of systematic reviews on associations of micronutrients and disease [33].”
MethodSelf-citationAutomatic document classification of biological literature
Chen, D. J., Müller, H., & Sternberg, P. W. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-370 · Cited by 41 · cited 2×
“Here we briefly review the emerging body of promising research on applications of machine learning methods to biomedical text classification [7-11], particularly those works focused on automatic classification of biomedical abstracts into clinically relevant categories.”
BackgroundMethodCriteria for determining disability in infants and children: low birth weight.
Cole, C. H., Binney, G., Casey, P., Fiascone, J. M., Hagadorn, J. I., Kim, C., Lau, J. T. F., Wang, C.-H. J., Devine, D. K., & Miller, K. (2002) · PubMed Central · Cited by 32
Self-citationStatistical Considerations in Meta-analysis
Barza, M. J., Trikalinos, T. A., & Lau, J. (2009) · Infectious Disease Clinics of North America · doi:10.1016/j.idc.2009.01.003 · Cited by 31 · cited 1×
“Like all scientific approaches, a systematic review tries to address a well-formulated research question by following a protocol of well-defined steps [1,2].”
BackgroundSelf-citationOntology-Based MEDLINE Document Classification
Camous, F., Blott, S. M., & Smeaton, A. F. (2007) · Springer eBooks · doi:10.1007/978-3-540-71233-6_34 · Cited by 27 · cited 1×
“Other encoded ontologies might also provide a source of enrichment [ 35 ].”
Criteria for determining disability in infants and children: failure to thrive.
Perrin, E. C., Cole, C. H., Frank, D. A., Glicken, S. R., Guerina, N. G., Petit, K., Sege, R. D., Volpe, M. V., Lau, J., McFadden, C. A., et al. (2003) · PubMed Central · Cited by 27
Self-citationMedinfo 2007. Proceedings of the 12th World Congress on Health (Medical) Informatics
Nøhr, C. G., Andersen, S. K., Bernstein, K., Bruun-Rasmussen, M., & Vingtoft, S. (2007) · IOS Press eBooks · Cited by 20
Text categorization models for identifying unproven cancer treatments on the web.
Aphinyanaphongs, Y., & Aliferis, C. (2007) · PubMed · Cited by 16
Criteria for determining disability in infants and children: short stature.
Wheeler, P. G., Bresnahan, K., Shephard, B., Lau, J. T. F., Balk, E. M., Devine, D. K., Chung, M. C., & Miller, K. (2003) · PubMed Central · Cited by 11
Self-citation(2001) · doi:10.1002/pds.4850 · Cited by 7
Research works printed but not linked: 3
- Wilcox A Hripcsak G Friedman C Using Knowledge Sources to Improve Classification of Medical Text Reports KDD-2000 Workshop on Text Mining (poster) 2000 Boston, MA
- Wallace B Trikalinos T Lau J Schmid C Brodley C SRSM (Presentation), Seattle, WA 2009
- PubMed Stopword List 2009 http://www.ncbi.nlm.nih.gov/entrez/query/static/help/pmhelp.html#Stopwords
Cited by
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
“Interest in the automation of systematic reviews has been driven by a necessity to expedite the availability of current best evidence for policy and clinical decision-making as much as engaging with technology to allow review authors to redirect their focus on aspects where they are best at [4].”
BackgroundTraumatic brain injury: integrated approaches to improve prevention, clinical care, and research
Maas, A. I. R., Menon, D. K., Adelson, P. D., Andelic, N., Bell, M. J., Belli, A., Bragge, P., Brazinova, A., Büki, A., Chesnut, R. M., et al. (2017) · The Lancet Neurology · doi:10.1016/s1474-4422(17)30371-x · Cited by 2,609
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
“as a means of overcoming hasty generalisation using an active learning approach [11].”
BackgroundMethodInfluentialLiving Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap
Elliott, J. H., Turner, T., Clavisi, O., Thomas, J., Higgins, J. P. T., Mavergames, C., & Gruen, R. L. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001603 · Cited by 596
“Here, machine learning technologies have the potential to reduce the manual ‘‘screening’’ of titles and abstracts by up to 50% in new reviews [27] and more than 90% in review updates [28], greatly enhancing the efficiency of review production.”
Background
Metadata
- Authors
- Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H.
- Affiliations
- Tufts University (US)
- Tufts Medical Center (US)
- Corresponding author
- Wallace, Byron C
- Venue
- BMC Bioinformatics
- Published
- 2010-01-26
- Volume
- 11
- Issue
- 1
- Pages
- 55-55
- ISSN
- 1471-2105 (ISSN-L)
- PMC
- PMC2824679
- OpenAlex
- W2043566294
- Cited by
- 351
- References
- 45
- Citations by year
- 2026: 23 · 2025: 27 · 2024: 34 · 2023: 33 · 2022: 23 · 2021: 28 · 2020: 36 · 2019: 29 · 2018: 22 · 2017: 15 · 2016: 12 · 2015: 20 · 2014: 15 · 2013: 21 · 2012: 7
- FWCI
- 3.01
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/2.0 (tdm)
- Locations
- BMC Bioinformatics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- U.S. Social Security Administration
- Tufts Medical Center
- National Institutes of Health (5r33rr017109-03)
- Agency for Healthcare Research and Quality (R01HS018494)
- National Center for Research Resources (R33 RR17109)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- citation screening, systematic reviews, semi-automated screening, active learning (machine learning), class imbalance, undersampling, support vector machine, ensemble classifiers, text classification, biomedical literature
- MeSH terms
- Periodicals as Topic, Publications, Review Literature as Topic, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.