Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 44 of the 45 references it cites were found in a registry or an index.

Semi-automated screening of biomedical citations for systematic reviews

Open access

Abstract

BACKGROUND: Systematic reviews address a specific clinical question by unbiasedly assessing and analyzing the pertinent literature. Citation screening is a time-consuming and critical step in systematic reviews. Typically, reviewers must evaluate thousands of citations to identify articles eligible for a given review. We explore the application of machine learning techniques to semi-automate citation screening, thereby reducing the reviewers' workload. RESULTS: We present a novel online classification strategy for citation screening to automatically discriminate "relevant" from "irrelevant" citations. We use an ensemble of Support Vector Machines (SVMs) built over different feature-spaces (e.g., abstract and title text), and trained interactively by the reviewer(s). Semi-automating the citation screening process is difficult because any such strategy must identify all citations eligible for the systematic review. This requirement is made harder still due to class imbalance; there are far fewer "relevant" than "irrelevant" citations for any given systematic review. To address these challenges we employ a custom active-learning strategy developed specifically for imbalanced datasets. Further, we introduce a novel undersampling technique. We provide experimental results over three real-world systematic review datasets, and demonstrate that our algorithm is able to reduce the number of citations that must be screened manually by nearly half in two of these, and by around 40% in the third, without excluding any of the citations eligible for the systematic review. CONCLUSIONS: We have developed a semi-automated citation screening algorithm for systematic reviews that has the potential to substantially reduce the number of citations reviewers have to manually screen, without compromising the quality and comprehensiveness of the review.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/1471-2105-11-55

Cites

44 of 44 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 Shown are the typical approach and our modified approach that includes semi-automated abstract screening on the left and right-hand, respectively (see text for details). In the modified approach the reviewers train and use a classification model to exclude completely "irrelevant" citations ("Level 1a"). They will trust the model's exclusions, and will review only the citations suggested by the classification model.
Figure 1 Shown are the typical approach and our modified approach that includes semi-automated abstract screening on the left and right-hand, respectively (see text for details). In the modified approach the reviewers train and use a classification model to exclude completely "irrelevant" citations ("Level 1a"). They will trust the model's exclusions, and will review only the citations suggested by the classification model.CC BY · PubMed Central
Figure 2 An article is broken down into its component parts (title, abstract text, and keywords), and these are in turn represented as either bag-of-words or bag-of-UMLS-biomedical concepts vectors.
Figure 2 An article is broken down into its component parts (title, abstract text, and keywords), and these are in turn represented as either bag-of-words or bag-of-UMLS-biomedical concepts vectors.CC BY · PubMed Central
Figure 3 Construction of confusion matrices for the semi-automated abstract screening strategy. The leftmost matrix represents citations that are labeled by the reviewer while training the classification model. The middle matrix displays the predictions of the trained model over the remaining unlabeled set of citations U. The rightmost matrix shows the corresponding crosstabulation at the end of "Level 1a" (see Figure 1). The quantities mentioned in this figure are used in the definition of Yield and Burden, the chosen evaluation metrics (see Equations 1 and 2). Superscripts T and U refer to model training and applying the model to yet unlabeled citations, respectively. tp[T|U]: "true positives", tn[T|U]: "true negatives", fp[T|U]: "false positives", fn [T|U]: "false negatives". We assume that reviewers will never erroneously exclude a citation that is eligible for systematic review, i.e. fnT = 0.
Figure 3 Construction of confusion matrices for the semi-automated abstract screening strategy. The leftmost matrix represents citations that are labeled by the reviewer while training the classification model. The middle matrix displays the predictions of the trained model over the remaining unlabeled set of citations U. The rightmost matrix shows the corresponding crosstabulation at the end of "Level 1a" (see Figure 1). The quantities mentioned in this figure are used in the definition of Yield and Burden, the chosen evaluation metrics (see Equations 1 and 2). Superscripts T and U refer to model training and applying the model to yet unlabeled citations, respectively. tp[T|U]: "true positives", tn[T|U]: "true negatives", fp[T|U]: "false positives", fn [T|U]: "false negatives". We assume that reviewers will never erroneously exclude a citation that is eligible for systematic review, i.e. fnT = 0.CC BY · PubMed Central
Figure 4 Yield (blue) and burden (red) curves for four learning strategies over the proton beam dataset as a function of the size of thetraining set. The thick lines are averages over 10 runs. Thin lines denote individual runs. Clockwise from the upper left, the strategies shown are: random sampling, SIMPLE, PAL, and PAL with aggressive undersampling. It is desirable to achieve maximum Yield while minimizing Burden. The upper right-corner (100% yield and 100% burden) corresponds to the manual approach of citation screening. Every point where Yield (the blue line) is at 1.0 and Burden (the red line) is less than 1.0 is thus progress. Note that Burden curves are U-shaped because classifiers trained on very small training sets tend to classify the majority of the unlabeled citations as "relevant" (due to our undersampling and cautious aggregation technique), and all citations classified as "relevant" must be subsequently screened by a human. When the training set is very large, the reviewers manually screen the majority of the citations during training.
Figure 4 Yield (blue) and burden (red) curves for four learning strategies over the proton beam dataset as a function of the size of thetraining set. The thick lines are averages over 10 runs. Thin lines denote individual runs. Clockwise from the upper left, the strategies shown are: random sampling, SIMPLE, PAL, and PAL with aggressive undersampling. It is desirable to achieve maximum Yield while minimizing Burden. The upper right-corner (100% yield and 100% burden) corresponds to the manual approach of citation screening. Every point where Yield (the blue line) is at 1.0 and Burden (the red line) is less than 1.0 is thus progress. Note that Burden curves are U-shaped because classifiers trained on very small training sets tend to classify the majority of the unlabeled citations as "relevant" (due to our undersampling and cautious aggregation technique), and all citations classified as "relevant" must be subsequently screened by a human. When the training set is very large, the reviewers manually screen the majority of the citations during training.CC BY · PubMed Central
Figure 5 Results over the COPD dataset.
Figure 5 Results over the COPD dataset.CC BY · PubMed Central
Figure 6 Results over the micro-nutrients dataset.
Figure 6 Results over the micro-nutrients dataset.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

10 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 45 references

Research works printed but not linked: 3

  1. Wilcox A, Hripcsak G, Friedman C: Using Knowledge Sources to Improve Classification of Medical Text Reports. In KDD-2000 Workshop on Text Mining (poster). Boston, MA; 2000.
  2. Wallace B, Trikalinos T, Lau J, Schmid C, Brodley C: SRSM (Presentation), Seattle, WA. 2009.
  3. PubMed Stopword List2009. [ http://www.ncbi.nlm.nih.gov/entrez/query/static/help/pmhelp.html#Stopwords ]

Cited by

See all 352

Metadata

Authors
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H.
Affiliations
  • Tufts University (US)
  • Tufts Medical Center (US)
Corresponding author
Wallace, Byron C
Venue
BMC Bioinformatics
Published
2010-01-26
Volume
11
Issue
1
Pages
55-55
ISSN
1471-2105 (ISSN-L)
OpenAlex
W2043566294
Cited by
351
References
45
Citations by year
2026: 23 · 2025: 27 · 2024: 34 · 2023: 33 · 2022: 23 · 2021: 28 · 2020: 36 · 2019: 29 · 2018: 22 · 2017: 15 · 2016: 12 · 2015: 20 · 2014: 15 · 2013: 21 · 2012: 7
FWCI
3.01
Type
journal article
Language
en
Open access
Licence
  • cc-by
  • creativecommons.org/licenses/by/2.0 (tdm)
Locations
  • BMC Bioinformatics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • U.S. Social Security Administration
  • Tufts Medical Center
  • National Institutes of Health (5r33rr017109-03)
  • Agency for Healthcare Research and Quality (R01HS018494)
  • National Center for Research Resources (R33 RR17109)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
citation screening, systematic reviews, semi-automated screening, active learning (machine learning), class imbalance, undersampling, support vector machine, ensemble classifiers, text classification, biomedical literature
MeSH terms
Periodicals as Topic, Publications, Review Literature as Topic, Information Storage and Retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Semi-automated screening of biomedical citations for systematic reviews. https://app.cochat.ai/papers/10.1186/1471-2105-11-55
The paper
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010). Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics. https://doi.org/10.1186/1471-2105-11-55

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.