SWIFT-Review: a text-mining workbench for systematic review
Open accessThe DOI matches its Crossref record; 23 of the 24 references it cites were found in a registry or an index.
Abstract
BACKGROUND: There is growing interest in using machine learning approaches to priority rank studies and reduce human burden in screening literature when conducting systematic reviews. In addition, identifying addressable questions during the problem formulation phase of systematic review can be challenging, especially for topics having a large literature base. Here, we assess the performance of the SWIFT-Review priority ranking algorithm for identifying studies relevant to a given research question. We also explore the use of SWIFT-Review during problem formulation to identify, categorize, and visualize research areas that are data rich/data poor within a large literature corpus. METHODS: Twenty case studies, including 15 public data sets, representing a range of complexity and size, were used to assess the priority ranking performance of SWIFT-Review. For each study, seed sets of manually annotated included and excluded titles and abstracts were used for machine training. The remaining references were then ranked for relevance using an algorithm that considers term frequency and latent Dirichlet allocation (LDA) topic modeling. This ranking was evaluated with respect to (1) the number of studies screened in order to identify 95 % of known relevant studies and (2) the "Work Saved over Sampling" (WSS) performance metric. To assess SWIFT-Review for use in problem formulation, PubMed literature search results for 171 chemicals implicated as EDCs were uploaded into SWIFT-Review (264,588 studies) and categorized based on evidence stream and health outcome. Patterns of search results were surveyed and visualized using a variety of interactive graphics. RESULTS: Compared with the reported performance of other tools using the same datasets, the SWIFT-Review ranking procedure obtained the highest scores on 11 out of 15 of the public datasets. Overall, these results suggest that using machine learning to triage documents for screening has the potential to save, on average, more than 50 % of the screening effort ordinarily required when using un-ordered document lists. In addition, the tagging and annotation capabilities of SWIFT-Review can be useful during the activities of scoping and problem formulation. CONCLUSIONS: Text-mining and machine learning software such as SWIFT-Review can be valuable tools to reduce the human screening burden and assist in problem formulation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



![Fig. 4 Performance of ranking algorithm on five datasets: Transgenerational, BPA, PFOS/PFOA, Neuropain: N = 100 [50 included; 50 excluded.]; Fluoride: N = 60 [30 Included; 30 Excluded.]) In all cases, the ranking algorithm results in a substantial potential reduction in screening effort compared to random ordering, with WSS@95 scores ranging from about 60 % (neuropain) to 90 % (fluoride)](https://figures.cochat.ai/paper-figures/151bd9c326414fb68963e03cfc380a91/swift-review-a-text-mining-workbench-for-systematic-review-figure-4.jpg)






Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The underlying machine learning model used by SWIFT-Active Screener is equivalent to the one previously described in Howard et al. (2016).”
From SWIFT-Active Screener: Accelerated document screening through active learning and integrated recall estimation (2020) · cites this paper as Howard et al. (2016)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In by far the most common approach, researchers use a classifier to produce a ranked list and then report WSS or area under a receiver operating characteristic (ROC) curve to show that high recall can be achieved at the beginning of the list, that is, if you were to somehow know when to stop.”
From SWIFT-Active Screener: Accelerated document screening through active learning and integrated recall estimation (2020) · cites this paper as Howard et al., 2016
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We have shown that by using SWIFT-Review, users can potentially reduce the total number of articles screened by 50% or more, provided that they are willing to trade a small reduction in recall for the reduced effort.”
From SWIFT-Active Screener: Accelerated document screening through active learning and integrated recall estimation (2020) · cites this paper as Howard et al., 2016
The cited source says what the claim attributes to it.
In the citing paper
“SWIFT-Active Screener (or simply “Active Screener,” see Fig. A.1) is an improvement of our previous method for article prioritization which we have earlier shown to theoretically reduce by more than 50% the human effort required to screen articles for inclusion in a systematic review (Howard et al., 2016)”
From SWIFT-Active Screener: Accelerated document screening through active learning and integrated recall estimation (2020) · cites this paper as Howard et al., 2016
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“This number (95%) was selected as it is a common recall number used when measuring the reduction in workload; it also approximates the level of human error in screening [3, 16, 26].”
From An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes (2020) · cites this paper as [16]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
A Limited Memory Algorithm for Bound Constrained Optimization
Byrd, R. H., Lu, P., Nocedal, J., & Zhu, C. (1995) · SIAM Journal on Scientific Computing · doi:10.1137/0916069 · Cited by 6,221 · cited 1×
“The likelihood equation is maximized using the limited memory Broyden-Fletcher-Goldfarb-Shanno (LBFGS) algorithm, a gradient-based optimization procedure designed for high-dimensional parameter spaces [12].”
MethodBlei, D. M. (2012) · Communications of the ACM · doi:10.1145/2133806.2133826 · Cited by 5,516 · cited 1×
“The latent Dirichlet allocation (LDA) topic modeling approach [9, 10] was used to probabilistically assign documents to topics.”
MethodScoping reviews: Time for clarity in definition, methods, and reporting
Colquhoun, H. L., Levac, D., O'Brien, K. K., Straus, S., Tricco, A. C., Perrier, L., Kastner, M., & Moher, D. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.03.013 · Cited by 2,783 · cited 1×
“These methods have much potential to reduce the human burden in screening studies for relevance and to produce “scoping reports” or “scoping studies,” a type of knowledge synthesis undertaken to guide the direction of future research priorities or to help with problem formulation when conducting a systematic review [5…”
Understanding inverse document frequency: on theoretical arguments for IDF
Robertson, S. (2004) · Journal of Documentation · doi:10.1108/00220410410560582 · Cited by 1,541 · cited 1×
“Raw word counts in each document are converted to length-normalized term frequency-inverse document frequency (TF-IDF) scores [8].”
MethodState-of-the-science of endocrine disrupting chemicals, 2012
Bergman, Å., Heindel, J., Jobling, S., Kidd, K., & Zoeller, R. T. (2012) · Toxicology Letters · doi:10.1016/j.toxlet.2012.03.020 · Cited by 1,388
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188 · cited 1×
“It has been estimated that every year, more than 4000 systematic reviews are conducted and published, each with the goal of summarizing the current state of knowledge relevant to a specific research question [1].”
BackgroundExpediting systematic reviews: methods and implications of rapid reviews
Ganann, R., Ciliska, D., & Thomas, H. (2010) · Implementation Science · doi:10.1186/1748-5908-5-56 · Cited by 752 · cited 1×
“On average, the amount of time required to conduct a single systematic review is at least 6 months to a year [2], and a considerable portion of this time is often spent on formulating the problem and identifying the relevant literature.”
BackgroundO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 1×
“However, to date, few software systems have been deployed that automate these methodologies, and usage of many of the reported methods requires the assistance of a data scientist having a detailed understanding of statistics and/or the skills necessary to program in statistical programming languages such as R, Python,…”
MethodReducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 10×
“Antihistamines [6] PubMed (PMIDs) 310 16 (5.”
BackgroundMethodInfluential
Research works printed but not linked: 3
- Robertson SE, Porter MF. New Models in Probabilistic Information Retrieval. Issue 5587 of British Library research & development report. London: Publisher Computer Laboratory, University of Cambridge; 1980. p. 123.
- SIL. English wordlists. 2015 [Online]. Available: http://www-01.sil.org/linguistics/wordlists/english/ . [Accessed: 01 Jan 2015].
- Wallace BC, Dahabreh IJ, Moran KH, Brodley CE, Trikalinos TA. Active literature discovery for scoping evidence reviews. In: 1th ACM SIGKDD Conference onf Knowledge Discovery and Data Mining (KDD). 2013.
Cited by
Kohl, C., McIntosh, E. J., Unger, S., Haddaway, N. R., Kecke, S., Schiemann, J., & Wilhelm, R. (2018) · Environmental Evidence · doi:10.1186/s13750-018-0115-5 · Cited by 471
Toxicity testing in the 21st century: progress in the past decade and future perspectives
Krewski, D., Andersen, M. E., Tyshenko, M. G., Krishnan, K., Hartung, T., Boekelheide, K., Wambaugh, J. F., Jones, D., Whelan, M., Thomas, R., et al. (2020) · Archives of Toxicology · doi:10.1007/s00204-019-02613-4 · Cited by 383
Olker, J. H., Elonen, C. M., Pilli, A., Anderson, A., Kinziger, B., Erickson, S., Skopinski, M., Pomplun, A., LaLone, C. A., Russom, C. L., et al. (2022) · Environmental Toxicology and Chemistry · doi:10.1002/etc.5324 · Cited by 301
Burkhard, L. P. (2021) · Environmental Toxicology and Chemistry · doi:10.1002/etc.5010 · Cited by 244
“Using Swift‐Review (Howard et al. 2016), relevant references were identified using the search function and the terms (i.”
MethodInfluentialArtificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
Metadata
- Authors
- Howard, B. E., Phillips, J., Miller, K., Tandon, A., Mav, D., Shah, M. R., Holmgren, S., Pelch, K. E., Walker, V., Rooney, A. A., Macleod, M., Shah, R. R., & Thayer, K.
- Affiliations
- National Institutes of Health (US)
- University of Edinburgh (GB)
- Corresponding author
- Howard, Brian E.
- ORCID
- Venue
- Systematic Reviews
- Published
- 2016-05-23
- Volume
- 5
- Issue
- 1
- Pages
- 87-87
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC4877757
- OpenAlex
- W2397182203
- Cited by
- 196
- References
- 24
- Citations by year
- 2026: 15 · 2025: 18 · 2024: 23 · 2023: 25 · 2022: 33 · 2021: 24 · 2020: 24 · 2019: 12 · 2018: 14 · 2017: 7 · 2016: 1
- FWCI
- 3.89
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- Funders
- National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC/L000970/1)
- Medical Research Council (MR/N015665/1)
- National Institute of Environmental Health Sciences
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- systematic reviews, text mining, literature screening, relevance ranking, latent Dirichlet allocation, topic modeling, problem formulation, endocrine disruptors, interactive visualization
- MeSH terms
- Machine Learning, Systematic Reviews as Topic, Algorithms, Software, Linear Models, Databases, Factual, Information Storage and Retrieval, Data Mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: SWIFT-Review: a text-mining workbench for systematic review. https://app.cochat.ai/papers/10.1186/s13643-016-0263-z
- The paper
- Howard, B. E., Phillips, J., Miller, K., Tandon, A., Mav, D., Shah, M. R., Holmgren, S., Pelch, K. E., Walker, V., Rooney, A. A., Macleod, M., Shah, R. R., & Thayer, K. (2016). SWIFT-Review: a text-mining workbench for systematic review. Systematic Reviews. https://doi.org/10.1186/s13643-016-0263-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.