CoChat

SWIFT-Review: a text-mining workbench for systematic review

Open access

The DOI matches its Crossref record; 23 of the 24 references it cites were found in a registry or an index.

Abstract

BACKGROUND: There is growing interest in using machine learning approaches to priority rank studies and reduce human burden in screening literature when conducting systematic reviews. In addition, identifying addressable questions during the problem formulation phase of systematic review can be challenging, especially for topics having a large literature base. Here, we assess the performance of the SWIFT-Review priority ranking algorithm for identifying studies relevant to a given research question. We also explore the use of SWIFT-Review during problem formulation to identify, categorize, and visualize research areas that are data rich/data poor within a large literature corpus. METHODS: Twenty case studies, including 15 public data sets, representing a range of complexity and size, were used to assess the priority ranking performance of SWIFT-Review. For each study, seed sets of manually annotated included and excluded titles and abstracts were used for machine training. The remaining references were then ranked for relevance using an algorithm that considers term frequency and latent Dirichlet allocation (LDA) topic modeling. This ranking was evaluated with respect to (1) the number of studies screened in order to identify 95 % of known relevant studies and (2) the "Work Saved over Sampling" (WSS) performance metric. To assess SWIFT-Review for use in problem formulation, PubMed literature search results for 171 chemicals implicated as EDCs were uploaded into SWIFT-Review (264,588 studies) and categorized based on evidence stream and health outcome. Patterns of search results were surveyed and visualized using a variety of interactive graphics. RESULTS: Compared with the reported performance of other tools using the same datasets, the SWIFT-Review ranking procedure obtained the highest scores on 11 out of 15 of the public datasets. Overall, these results suggest that using machine learning to triage documents for screening has the potential to save, on average, more than 50 % of the screening effort ordinarily required when using un-ordered document lists. In addition, the tagging and annotation capabilities of SWIFT-Review can be useful during the activities of scoping and problem formulation. CONCLUSIONS: Text-mining and machine learning software such as SWIFT-Review can be valuable tools to reduce the human screening burden and assist in problem formulation.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13643-016-0263-z

Cites

23 of 23 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 “Work Saved over Sampling” (WSS) performance metric. The dotted black line illustrates the expected recall achieved when traversing a randomly ordered list. Similarly, the blue line shows the recall obtained when traversing a (hypothetical) ranked list. The length of the dotted red line indicates the percent reduction in effort achieved by ranking and corresponds to the WSS at 95 % recall, in this case, approximately 15 % (95–80 %)
Fig. 1 “Work Saved over Sampling” (WSS) performance metric. The dotted black line illustrates the expected recall achieved when traversing a randomly ordered list. Similarly, the blue line shows the recall obtained when traversing a (hypothetical) ranked list. The length of the dotted red line indicates the percent reduction in effort achieved by ranking and corresponds to the WSS at 95 % recall, in this case, approximately 15 % (95–80 %)CC BY · PubMed Central
Fig. 2 SWIFT-Review user interface and tag browser. The SWIFT-Review Tag Browser allows users to interactively filter a literature set by selecting various combinations of “tags” that have been automatically and/or manually applied to the corpus. In this case, the user has selected for investigation research articles in the “Neoplasms” health outcome category; terms in each abstract that are related to these tags are highlighted automatically in the Document Preview panel
Fig. 2 SWIFT-Review user interface and tag browser. The SWIFT-Review Tag Browser allows users to interactively filter a literature set by selecting various combinations of “tags” that have been automatically and/or manually applied to the corpus. In this case, the user has selected for investigation research articles in the “Neoplasms” health outcome category; terms in each abstract that are related to these tags are highlighted automatically in the Document Preview panelCC BY · PubMed Central
Fig. 3 Learning curves. The graphs above show that, as expected, performance of the prioritization method on each dataset is an increasing function of training set size. Since the total number of available positive instances varies significantly between datasets, not all sizes could be tested for each dataset
Fig. 3 Learning curves. The graphs above show that, as expected, performance of the prioritization method on each dataset is an increasing function of training set size. Since the total number of available positive instances varies significantly between datasets, not all sizes could be tested for each datasetCC BY · PubMed Central
Fig. 4 Performance of ranking algorithm on five datasets: Transgenerational, BPA, PFOS/PFOA, Neuropain: N = 100 [50 included; 50 excluded.]; Fluoride: N = 60 [30 Included; 30 Excluded.]) In all cases, the ranking algorithm results in a substantial potential reduction in screening effort compared to random ordering, with WSS@95 scores ranging from about 60 % (neuropain) to 90 % (fluoride)
Fig. 4 Performance of ranking algorithm on five datasets: Transgenerational, BPA, PFOS/PFOA, Neuropain: N = 100 [50 included; 50 excluded.]; Fluoride: N = 60 [30 Included; 30 Excluded.]) In all cases, the ranking algorithm results in a substantial potential reduction in screening effort compared to random ordering, with WSS@95 scores ranging from about 60 % (neuropain) to 90 % (fluoride)CC BY · PubMed Central
Fig. 5 Observed changes in WSS@95 attributable to three feature types. a LDA, b MeSH terms, and c N-grams on 20 SR datasets. Mean changes in WSS@95 were 4.4 % (LDA), 1 % (MeSH), and −0. 4 % (NGrams). In each case, performance was measured on each of the 20 datasets both with and without the specified feature type. The resulting WSS@95 differences for each dataset were averaged over 25 trials. As shown in a, adding LDA features to the ranking algorithm can result in significant performance increases, whereas inclusion of the MeSH and NGram features (b and c) were not found to result in large additional benefits when the remaining feature types were also included
Fig. 5 Observed changes in WSS@95 attributable to three feature types. a LDA, b MeSH terms, and c N-grams on 20 SR datasets. Mean changes in WSS@95 were 4.4 % (LDA), 1 % (MeSH), and −0. 4 % (NGrams). In each case, performance was measured on each of the 20 datasets both with and without the specified feature type. The resulting WSS@95 differences for each dataset were averaged over 25 trials. As shown in a, adding LDA features to the ranking algorithm can result in significant performance increases, whereas inclusion of the MeSH and NGram features (b and c) were not found to result in large additional benefits when the remaining feature types were also includedCC BY · PubMed Central
Fig. 6 Study Flow diagram for the analysis of 171 UNEP EDC chemicals. The literature search identified 221,898 recent research articles out of the total 709,573 EDC articles retrieved from searching PubMed
Fig. 6 Study Flow diagram for the analysis of 171 UNEP EDC chemicals. The literature search identified 221,898 recent research articles out of the total 709,573 EDC articles retrieved from searching PubMedCC BY · PubMed Central
Fig. 7 Interactively exploring arsenic in the EDC scoping report in SWIFT-Review. This example shows a pie chart survey of health outcomes represented among the 2400 studies on arsenic with a MeSH disease code. In this pie chart, studies lacking a MeSH disease code are not displayed (9553 of the 11,953 documents retrieved for arsenic) and documents may appear in multiple health outcome categories. Below the pie-chart is a list of 1342 documents relevant to “arsenic and neoplasms”. Inset Using the interactive browser, users can “drill down” to further explore documents in a specific area, e.g., arsenic and neoplasms, based on other tags such as evidence stream (human, animal, in vitro)
Fig. 7 Interactively exploring arsenic in the EDC scoping report in SWIFT-Review. This example shows a pie chart survey of health outcomes represented among the 2400 studies on arsenic with a MeSH disease code. In this pie chart, studies lacking a MeSH disease code are not displayed (9553 of the 11,953 documents retrieved for arsenic) and documents may appear in multiple health outcome categories. Below the pie-chart is a list of 1342 documents relevant to “arsenic and neoplasms”. Inset Using the interactive browser, users can “drill down” to further explore documents in a specific area, e.g., arsenic and neoplasms, based on other tags such as evidence stream (human, animal, in vitro)CC BY · PubMed Central
Fig. 8 Survey of types of chemicals associated with female urogenital disease and pregnancy. The current example uses a pie chart graphic to survey the types of stressors (e.g., pesticides, drugs of abuse, diet and nutrition) associated with the health outcome of female urogenital disease and pregnancy. Below the pie chart is a list of 611 documents retrieved as part of the “pesticides” filter within SWIFT-Review and a bar chart of the most common Tox21 chemicals referenced in the pesticides cluster. Note that bisphenol A is not a pesticide but appears on this list because it was frequently mentioned in the pesticide studies
Fig. 8 Survey of types of chemicals associated with female urogenital disease and pregnancy. The current example uses a pie chart graphic to survey the types of stressors (e.g., pesticides, drugs of abuse, diet and nutrition) associated with the health outcome of female urogenital disease and pregnancy. Below the pie chart is a list of 611 documents retrieved as part of the “pesticides” filter within SWIFT-Review and a bar chart of the most common Tox21 chemicals referenced in the pesticides cluster. Note that bisphenol A is not a pesticide but appears on this list because it was frequently mentioned in the pesticide studiesCC BY · PubMed Central
Fig. 9 Excerpt of a heat map displaying search results for the 171 EDC chemicals categorized by health outcomes. The numbers displayed indicate the number of SWIFT-Review records matching each combination of chemical (rows) and health outcomes (columns). “Pockets” with larger numbers of matching records are displayed in red color
Fig. 9 Excerpt of a heat map displaying search results for the 171 EDC chemicals categorized by health outcomes. The numbers displayed indicate the number of SWIFT-Review records matching each combination of chemical (rows) and health outcomes (columns). “Pockets” with larger numbers of matching records are displayed in red colorCC BY · PubMed Central
Fig. 10 Topic models bar chart from the EDC scoping report. Topic modeling is an unsupervised clustering technique that can often automatically “discover” the main themes in an unlabeled literature corpus. For example, in the case of the EDC literature set, several interesting topics are shown above including topics related to BPA exposure during pregnancy (topic 13), analytical methods used to measure levels of EDCs (topic 23), estrogen, expression, and receptors (topic 7), lead and arsenic exposure (topic 31), breast and prostate cancer (topic 26), and thyroid disease (topic 15). Within SWIFT-Review, users can select any of these topics to interactively browse the associated documents
Fig. 10 Topic models bar chart from the EDC scoping report. Topic modeling is an unsupervised clustering technique that can often automatically “discover” the main themes in an unlabeled literature corpus. For example, in the case of the EDC literature set, several interesting topics are shown above including topics related to BPA exposure during pregnancy (topic 13), analytical methods used to measure levels of EDCs (topic 23), estrogen, expression, and receptors (topic 7), lead and arsenic exposure (topic 31), breast and prostate cancer (topic 26), and thyroid disease (topic 15). Within SWIFT-Review, users can select any of these topics to interactively browse the associated documentsCC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 23 references

Research works printed but not linked: 3

  1. Robertson SE, Porter MF. New Models in Probabilistic Information Retrieval. Issue 5587 of British Library research & development report. London: Publisher Computer Laboratory, University of Cambridge; 1980. p. 123.
  2. SIL. English wordlists. 2015 [Online]. Available: http://www-01.sil.org/linguistics/wordlists/english/ . [Accessed: 01 Jan 2015].
  3. Wallace BC, Dahabreh IJ, Moran KH, Brodley CE, Trikalinos TA. Active literature discovery for scoping evidence reviews. In: 1th ACM SIGKDD Conference onf Knowledge Discovery and Data Mining (KDD). 2013.

Cited by

See all 196

Metadata

Authors
Howard, B. E., Phillips, J., Miller, K., Tandon, A., Mav, D., Shah, M. R., Holmgren, S., Pelch, K. E., Walker, V., Rooney, A. A., Macleod, M., Shah, R. R., & Thayer, K.
Affiliations
  • National Institutes of Health (US)
  • University of Edinburgh (GB)
Corresponding author
Howard, Brian E.
Venue
Systematic Reviews
Published
2016-05-23
Volume
5
Issue
1
Pages
87-87
ISSN
2046-4053 (ISSN-L)
OpenAlex
W2397182203
Cited by
196
References
24
Citations by year
2026: 15 · 2025: 18 · 2024: 23 · 2023: 25 · 2022: 33 · 2021: 24 · 2020: 24 · 2019: 12 · 2018: 14 · 2017: 7 · 2016: 1
FWCI
3.89
Type
journal article
Language
en
Open access
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
Funders
  • National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC/L000970/1)
  • Medical Research Council (MR/N015665/1)
  • National Institute of Environmental Health Sciences
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
systematic reviews, text mining, literature screening, relevance ranking, latent Dirichlet allocation, topic modeling, problem formulation, endocrine disruptors, interactive visualization
MeSH terms
Machine Learning, Systematic Reviews as Topic, Algorithms, Software, Linear Models, Databases, Factual, Information Storage and Retrieval, Data Mining

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: SWIFT-Review: a text-mining workbench for systematic review. https://app.cochat.ai/papers/10.1186/s13643-016-0263-z
The paper
Howard, B. E., Phillips, J., Miller, K., Tandon, A., Mav, D., Shah, M. R., Holmgren, S., Pelch, K. E., Walker, V., Rooney, A. A., Macleod, M., Shah, R. R., & Thayer, K. (2016). SWIFT-Review: a text-mining workbench for systematic review. Systematic Reviews. https://doi.org/10.1186/s13643-016-0263-z

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.