The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index; 16 of the 21 sources its claims cite were checked for support.
An open source machine learning framework for efficient and transparent systematic reviews
Open accessAbstract
Abstract To help researchers conduct a systematic review or meta-analysis as efficiently and transparently as possible, we designed a tool to accelerate the step of screening titles and abstracts. For many tasks—including but not limited to systematic reviews and meta-analyses—the scientific literature needs to be checked systematically. Scholars and practitioners currently screen thousands of studies by hand to determine which studies to include in their review or meta-analysis. This is error prone and inefficient because of extremely imbalanced data: only a fraction of the screened studies is relevant. The future of systematic reviewing will be an interaction with machine learning algorithms to deal with the enormous increase of available text. We therefore developed an open source machine learning-aided pipeline applying active learning: ASReview. We demonstrate by means of simulation studies that active learning can yield far more efficient reviewing than manual reviewing while providing high quality. Furthermore, we describe the options of the free and open source research software and present the results from user experience tests. We invite the community to contribute to open source projects such as our own that provide measurable and reproducible improvements over current practice.
Overview
CoChat Summary
TL;DR
The paper presents ASReview, an open-source active-learning machine learning tool designed to speed up title-and-abstract screening in systematic reviews and meta-analyses while keeping the process transparent. In simulation studies and user experience tests, the authors report substantial screening-effort savings and iterative usability improvements.
Key findings
- Across four simulation datasets, ASReview’s average work saved over sampling at 95% recall was 83%, ranging from 67% to 92%; in practical terms, 95% of eligible studies could be found after screening only 8% to 33% of records.
- The number of relevant abstracts found after screening the first 10% of abstracts ranged from 70% to 100%.
- The paper reports dataset inclusion rates of 4.84% (120/2,481) for viral metagenomic sequencing in livestock, 1.2% (104/8,911) for fault prediction in software engineering, 0.66% (38/5,782) for longitudinal PTSD-related studies, and 1.6% (41/2,544) for ACE-inhibitor efficacy.
- User experience testing led to multiple releases and UI/documentation changes; participants rated the tool around 7.8–8.0/10 in the reported UX tests.
Methods
- Design: The paper combines a software description with simulation studies, real-world use cases, and user experience testing; ASReview uses active learning to iteratively prioritize records for screening.
- Data/sample: Four labeled datasets were used in simulation mode, drawn from published systematic reviews and searches in domains including livestock virology, software engineering, PTSD, and ACE-inhibitor efficacy.
- Main analysis: Performance was evaluated using work saved over sampling (WSS), WSS@100%, and RRF10%; UX feedback was analyzed with thematic analysis from moderated remote testing and unstructured interviews.
Limitations
- The authors state that active learning reduces screening burden but makes it hard to straightforwardly evaluate error rates without additional labeling.
- They also state there are no empirical benchmarks for how the approach performs in other non-review settings, and that the system automates only the screening step, not the full review workflow.
- Evident from the text: the simulation results are based on a limited set of four datasets and the software’s performance depends on the quality of the initial search and prior knowledge selection.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
14 Valid · 1 Not valid · 4 Not checked · 16 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For example, in systematic reviews, the optimal type of classifier will depend on variable parameters, such as the proportion of relevant publications in the initial search and the complexity of the inclusion criteria used by the researcher 23 .”
§ An open source machine learning framework for efficient and transparent systematic reviews > Not available.
What the sources say
- Partly supportedcites [23] Kilicoglu, H., Demner-Fushman, D., Rindflesch, T. C., Wilczynski, N. L. & Haynes, R. B. Towards automatic recognition of scientifically rigorous clinical research evidence. J. Am. Med. Inform. Assoc. 16, 25–31 (2009). sourceExperimental — accuracy not yet measured
The Scientific Rigor section says different classifiers serve different information needs, supporting the general point that the best choice depends on the task. The Data Collection section reports the proportion of rigorous articles in its collection, but our copy does not say that the optimal classifier depends on the proportion of relevant results in an initial systematic-review search or on the complexity of a reviewer's inclusion criteria. No page number was supplied for these passages. (LLM, conf=0.83)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“First, many are closed source applications with black box algorithms, which is problematic as transparency and data ownership are essential in the era of open science 22 .”
§ An open source machine learning framework for efficient and transparent systematic reviews > Not available.
What the sources say
- Partly supportedcites [22] Nosek, B. A. et al. Promoting an open research culture. Science 348, 1422–1425 (2015). sourceExperimental — accuracy not yet measured
The paper’s Policy section calls transparency and openness vital features of science, and its Standards discussion advocates sharing data and analytic code. It does not establish that many existing applications are closed source or use black-box algorithms, nor does it discuss data ownership as an essential principle. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“As such, results from ASReview depend on the quality of the initial search process, including selection of databases 24 and construction of comprehensive searches using keywords and controlled vocabulary.”
§ An open source machine learning framework for efficient and transparent systematic reviews > Real-world use cases and high-level function descriptions
What the sources say
- Supportedcites [24] Gusenbauer, M. & Haddaway, N. R. Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources. Res. Synth. Methods 11, 181–217 (2020). source
The citation is attached to the selection of databases. The Introduction says the choice of adequate search systems determines how many relevant articles are identified, and Method and Analysis says effective searching depends on choosing a system with suitable coverage. These passages support the cited point about the quality of the initial database search; they do not make an ASReview-specific claim. (LLM, conf=0.89)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Machine learning offers approaches to overcome the manual and time-consuming screening of large numbers of studies by prioritizing relevant studies via active learning 15 .”
§ An open source machine learning framework for efficient and transparent systematic reviews > W
What the sources say
- Partly supportedcites [15] Cohen, A. M., Hersh, W. R., Peterson, K. & Yen, P.-Y. Reducing workload in systematic review preparation using automated citation classification. J. Am. Med. Inform. Assoc. 13, 206–219 (2006). sourceExperimental — accuracy not yet measured
The Abstract says machine-learning classification could reduce manual review and help reviewers prioritize articles most likely to contain high-quality evidence. The Methods describe training a classifier on existing test collections, but our copy does not identify the approach as active learning; that specific mechanism is unsupported. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In fact, if the literature of a field is growing faster than the amount of time available for systematic reviews, adequate manual review of this field then becomes impossible 11 .”
§ An open source machine learning framework for efficient and transparent systematic reviews > W
What the sources say
- Supportedcites [11] Marshall, I. J. & Wallace, B. C. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Syst. Rev. 8, 163 (2019). source
The Background says that current review processes cannot efficiently produce reviews of current evidence, that reviews often become outdated quickly, and that rigorous manual methods do not scale to the volume of evidence. This supports the citing sentence’s point that available manual review capacity cannot keep pace with growing literature. (LLM, conf=0.88)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
10 more claims · 2 with their cited sources checked
Create a free account to see all 35 claims and every source verdict.
- It may state two separate claims as one. 12
- Probably not a substantial claim. 11
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Second, existing tools lack the flexibility to handle the wide range of classifiers and feature extraction techniques that can be implemented in ML screening tools effectively(van de Schoot et al., 2021).”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as van de Schoot et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Studies in health science have shown that using learning algorithms can save more than 90% of the time required for manual abstract screening (van de Schoot et al., 2021).”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as van de Schoot et al., 2021
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The application of active learning models in systematic reviews has been extensively studied in simulation studies [11-21].”
From Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records (2023) · cites this paper as [20]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the ML tools identified by this review have been created with different aims by a great variety of developers, ranging from individuals and small research groups to large organisations dedicated to evidence-based medicine and systematic reviewing at a large scale [31, 37, 44, 56, 57]”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [56]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, with the prevalence of active learning systematic review tools (Harrison et al., 2020), active learning is at this point the most approachable machine learning method for systematic reviews, with the clearest benefits coming in the title and abstract screening phase (van de Schoot et al., 2021).”
From SYMBALS: A Systematic Review Methodology Blending Active Learning and Snowballing (2021) · cites this paper as van de Schoot et al., 2021
Experimental — accuracy not yet measured
15 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
Distributed Representations of Sentences and Documents
Le, Q. V., & Mikolov, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.4053 · Cited by 4,934
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · cited 1×
“NATuRe MAchiNe iNTelligeNce comparison with some of the existing systems, as shown in the last column of Table 1.”
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708
Gusenbauer, M., & Haddaway, N. R. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1378 · Cited by 2,379
Research works printed but not linked: 4
- Cooper, H. Research Synthesis and Meta-analysis: A Step-by-Step Approach (SAGE Publications, 2015).
- Ramos, J. Using TF–IDF to determine word relevance in document queries. In Proc. 1st Instructional Conference on Machine Learning Vol. 242, 133–142 (ICML, 2003).
- Vasalou, A., Ng, B. D., Wiemer-Hastings, P. & Oshlyansky, L. Human-moderated remote user testing: orotocols and applications. In 8th ERCIM Workshop, User Interfaces for All Vol. 19 (ERCIM, 2004).
- Joffe, H. in Qualitative Research Methods in Mental Health and Psychotherapy: A Guide for Students and Practitioners (eds Harper, D. & Thompson, A. R.) Ch. 15 (Wiley, 2012).
Cited by
Explainable artificial intelligence (XAI) in deep learning-based medical image analysis
van der Velden, B. H. M., Kuijf, H. J., Gilhuijs, K. G. A., & Viergever, M. A. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102470 · Cited by 1,328
Sharkiya, S. H. (2023) · BMC Health Services Research · doi:10.1186/s12913-023-09869-8 · Cited by 416
Artificial intelligence and the conduct of literature reviews
Wagner, G., Lukyanenko, R., & Paré, G. (2022) · Journal of Information Technology · doi:10.1177/02683962211048201 · Cited by 296
“We note that ASReview is a promising exemplar in this regard since it offers a graphical user interface as well as command-line access to the underlying code (Van de Schoot et al., 2021).”
BackgroundMethodArtificial intelligence in systematic reviews: promising when appropriately used
van Dijk, S. H. B., Brusse-Keizer, M. G. J., Bucsán, C. C., van der Palen, J., Doggen, C. J. M., & Lenferink, A. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-072254 · Cited by 256
“After snowballing, one additional article from a study already included was added.”
BackgroundArtificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
“Likewise, several tools continue to employ BoW methods for text representation, although some of the most recent ones [42, 45, 71] have shifted towards adopting word and sentence embedding techniques, such as GloVe [95], ELMo [103], SciBERT [90], and Sentence-BERT [92].”
Metadata
- Authors
- van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., Harkema, A., Willemsen, J., Ma, Y., Fang, Q., Hindriks, S., Tummers, L., & Oberski, D. L.
- Affiliations
- Utrecht University (NL)
- ORCID
- Venue
- Nature Machine Intelligence
- Published
- 2021-02-01
- Volume
- 3
- Issue
- 2
- Pages
- 125-133
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W3111278950
- Cited by
- 1,157
- References
- 46
- Citations by year
- 2026: 355 · 2025: 329 · 2024: 253 · 2023: 134 · 2022: 60 · 2021: 25
- FWCI
- 4.94
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Nature Machine Intelligence (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- arXiv (Cornell University) (cc-by)
- Funders
- This project was funded by the Innovation Fund for IT in Research Projects, Utrecht University, The Netherlands.
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- systematic reviews, meta-analysis, title and abstract screening, active learning (machine learning), ASReview, open source software, literature search automation, class imbalance, reproducibility, user experience
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: An open source machine learning framework for efficient and transparent systematic reviews. https://app.cochat.ai/papers/10.1038/s42256-020-00287-7
- The paper
- van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., Harkema, A., Willemsen, J., Ma, Y., Fang, Q., Hindriks, S., Tummers, L., & Oberski, D. L. (2021). An open source machine learning framework for efficient and transparent systematic reviews. Nature Machine Intelligence. https://doi.org/10.1038/s42256-020-00287-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.