Automated screening of research studies for systematic reviews using study characteristics
Open accessThe DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Screening candidate studies for inclusion in a systematic review is time-consuming when conducted manually. Automation tools could reduce the human effort devoted to screening. Existing methods use supervised machine learning which train classifiers to identify relevant words in the abstracts of candidate articles that have previously been labelled by a human reviewer for inclusion or exclusion. Such classifiers typically reduce the number of abstracts requiring manual screening by about 50%. METHODS: We extracted four key characteristics of observational studies (population, exposure, confounders and outcomes) from the text of titles and abstracts for all articles retrieved using search strategies from systematic reviews. Our screening method excluded studies if they did not meet a predefined set of characteristics. The method was evaluated using three systematic reviews. Screening results were compared to the actual inclusion list of the reviews. RESULTS: The best screening threshold rule identified studies that mentioned both exposure (E) and outcome (O) in the study abstract. This screening rule excluded 93.7% of retrieved studies with a recall of 98%. CONCLUSIONS: Filtering studies for inclusion in a systematic review based on the detection of key study characteristics in abstracts significantly outperformed standard approaches to automated screening and appears worthy of further development and evaluation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“as researchers agree that this is one of the most time-consuming steps [58, 67,71].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [71]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, these tools, sometimes referred to as discriminative artificial intelligence (AI) technologies, generally require human researchers providing large amount of sufficiently diverse data for training and active learning [1, 3, 5, 6].”
From Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis (2025) · cites this paper as [6]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092 · cited 1×
“For example, the systematic review tool Rayyan updates a machine learning classifier that promotes abstracts to the top of the screening queue when they have more similar words to previously included abstracts [6].”
MethodOutdoor Particulate Matter Exposure and Lung Cancer: A Systematic Review and Meta-Analysis
Hamra, G. B., Guha, N., Cohen, A., Laden, F., Raaschou-Nielsen, O., Samet, J. M., Vineis, P., Forastiere, F., Saldiva, P., Yorifuji, T., et al. (2014) · Environmental Health Perspectives · doi:10.1289/ehp/1408092 · Cited by 1,023
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 2×
“A recent review found 44 such algorithms [5], and this functionality is already available in several commercial systems.”
BackgroundMethodLiving Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap
Elliott, J. H., Turner, T., Clavisi, O., Thomas, J., Higgins, J. P. T., Mavergames, C., & Gruen, R. L. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001603 · Cited by 596 · cited 1×
“The relentless growth in research and the greater reliance placed on evidence for clinical and policy decisions have created a need for automation to support the process of systematic reviews [1, 2].”
BackgroundSystematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472 · cited 1×
“…task of appraising the full text is reserved for cases where decisions cannot be made based on the abstract alone, where reviewers disagree on whether to include or exclude, and for those papers that appear relevant at title/abstract screening, but may be disregarded at full-text screening [3].”
BackgroundSelf-citationJohnson, P. I., Sutton, P., Atchley, D. S., Koustas, E., Lam, J., Sen, S., Robinson, K. A., Axelrad, D. A., & Woodruff, T. J. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1307893 · Cited by 449
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Koustas, E., Lam, J., Sutton, P., Johnson, P. I., Atchley, D. S., Sen, S., Robinson, K. A., Axelrad, D. A., & Woodruff, T. J. (2014) · Environmental Health Perspectives · doi:10.1289/ehp.1307177 · Cited by 349
Pre- and Postnatal Exposure to Perfluorinated Compounds (PFCs)
Fromme, H., Mosch, C., Morovitz, M., Alba-Alejandre, I., Boehmer, S., Kiranoglu, M., Faber, F., Hannibal, I., Genzel-Boroviczény, O., Koletzko, B., et al. (2010) · Environmental Science & Technology · doi:10.1021/es101184f · Cited by 333 · cited 1×
“We have analysed the abstract of the one study that was missed by the system [13].”
BackgroundText mining and its potential applications in systems biology
Ananiadou, S., Kell, D. B., & Tsujii, J.-I. (2006) · Trends in Biotechnology · doi:10.1016/j.tibtech.2006.10.002 · Cited by 324 · cited 1×
“The algorithm was developed using the General Architecture for Text Engineering (GATE) [11].”
Method
Research works printed but not linked: 2
- Tsafnat G, Glasziou P, Dunn A, Coiera E. Systematic-review automation technologies. Systematic reviews. 2014;3:74. · DOI · Paper page
- Thayer K, Rooney A, Boyles A, Holmgren S, Walker V, Kissling G. Draft protocol for systematic review to evaluate the evidence for an association between bisphenol A (BPA) exposure and obesity. National Toxicology Program. 2013.
Cited by
Automation of systematic literature reviews: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106589 · Cited by 432
“The Selection of primary studies (SLR6) was automated most often, as researchers agree that this is one of the most time-consuming steps [58, 67,71].”
BackgroundChai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01635-3 · Cited by 190
“Machine learning utilises the mined text for training algorithms to create a classifier model based upon reviewers ’ inclusion and exclusion decisions [40].”
BackgroundMethodBaclic, O., Tunis, M., Young, K., Doan, C., & Swerdfeger, H. (2020) · Canada Communicable Disease Report · doi:10.14745/ccdr.v46i06a02 · Cited by 134
Cleo, G., Scott, A. M., Islam, F., Julien, B., & Beller, E. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1069-6 · Cited by 112
NLP-assisted software testing: A systematic mapping of the literature
Garousi, V., Bauer, S., & Felderer, M. (2020) · Information and Software Technology · doi:10.1016/j.infsof.2020.106321 · Cited by 100
“, [32, 33], and using our past experience in SLM/SLR studies, e.”
Method
Metadata
- Authors
- Tsafnat, G., Glasziou, P., Karystianis, G., & Coiera, E.
- Affiliations
- Macquarie University (AU)
- Bond University (AU)
- UNSW Sydney (AU)
- Corresponding author
- Tsafnat, Guy
- Venue
- Systematic Reviews
- Published
- 2018-04-25
- Volume
- 7
- Issue
- 1
- Pages
- 64-64
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC5918752
- OpenAlex
- W2801856446
- Cited by
- 44
- References
- 16
- Citations by year
- 2026: 3 · 2025: 5 · 2024: 6 · 2023: 9 · 2022: 3 · 2021: 7 · 2020: 7 · 2019: 4
- FWCI
- 0.89
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Bond University Research Portal (Bond University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Institute of Environmental Health Sciences
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- study characteristics, observational studies
- MeSH terms
- Machine Learning, Systematic Reviews as Topic, Automation, Humans, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Automated screening of research studies for systematic reviews using study characteristics. https://app.cochat.ai/papers/10.1186/s13643-018-0724-7
- The paper
- Tsafnat, G., Glasziou, P., Karystianis, G., & Coiera, E. (2018). Automated screening of research studies for systematic reviews using study characteristics. Systematic Reviews. https://doi.org/10.1186/s13643-018-0724-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.