Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 63 of the 63 references it cites were found in a registry or an index; 16 of the 21 sources its claims cite were checked for support.

An open source machine learning framework for efficient and transparent systematic reviews

Open access
van de Schoot, Rens, de Bruin, Jonathan, Schram, Raoul, Zahedi, Parisa, de Boer, Jan, Weijdema, Felix, Kramer, Bianca, Huijts, Martijn, Hoogerwerf, Maarten, Ferdinands, Gerbrich, Harkema, Albert, Willemsen, Joukje, Ma, Yongchao, Fang, Qixiang, Hindriks, Sybren, Tummers, Lars, Oberski, Daniel L. · 2021 · Nature Machine Intelligence

Abstract

Abstract To help researchers conduct a systematic review or meta-analysis as efficiently and transparently as possible, we designed a tool to accelerate the step of screening titles and abstracts. For many tasks—including but not limited to systematic reviews and meta-analyses—the scientific literature needs to be checked systematically. Scholars and practitioners currently screen thousands of studies by hand to determine which studies to include in their review or meta-analysis. This is error prone and inefficient because of extremely imbalanced data: only a fraction of the screened studies is relevant. The future of systematic reviewing will be an interaction with machine learning algorithms to deal with the enormous increase of available text. We therefore developed an open source machine learning-aided pipeline applying active learning: ASReview. We demonstrate by means of simulation studies that active learning can yield far more efficient reviewing than manual reviewing while providing high quality. Furthermore, we describe the options of the free and open source research software and present the results from user experience tests. We invite the community to contribute to open source projects such as our own that provide measurable and reproducible improvements over current practice.

Overview

CoChat Summary

TL;DR

The paper presents ASReview, an open-source active-learning machine learning tool designed to speed up title-and-abstract screening in systematic reviews and meta-analyses while keeping the process transparent. In simulation studies and user experience tests, the authors report substantial screening-effort savings and iterative usability improvements.

Key findings

  • Across four simulation datasets, ASReview’s average work saved over sampling at 95% recall was 83%, ranging from 67% to 92%; in practical terms, 95% of eligible studies could be found after screening only 8% to 33% of records.
  • The number of relevant abstracts found after screening the first 10% of abstracts ranged from 70% to 100%.
  • The paper reports dataset inclusion rates of 4.84% (120/2,481) for viral metagenomic sequencing in livestock, 1.2% (104/8,911) for fault prediction in software engineering, 0.66% (38/5,782) for longitudinal PTSD-related studies, and 1.6% (41/2,544) for ACE-inhibitor efficacy.
  • User experience testing led to multiple releases and UI/documentation changes; participants rated the tool around 7.8–8.0/10 in the reported UX tests.

Methods

  • Design: The paper combines a software description with simulation studies, real-world use cases, and user experience testing; ASReview uses active learning to iteratively prioritize records for screening.
  • Data/sample: Four labeled datasets were used in simulation mode, drawn from published systematic reviews and searches in domains including livestock virology, software engineering, PTSD, and ACE-inhibitor efficacy.
  • Main analysis: Performance was evaluated using work saved over sampling (WSS), WSS@100%, and RRF10%; UX feedback was analyzed with thematic analysis from moderated remote testing and unstructured interviews.

Limitations

  • The authors state that active learning reduces screening burden but makes it hard to straightforwardly evaluate error rates without additional labeling.
  • They also state there are no empirical benchmarks for how the approach performs in other non-review settings, and that the system automates only the screening step, not the full review workflow.
  • Evident from the text: the simulation results are based on a limited set of four datasets and the software’s performance depends on the quality of the initial search and prior knowledge selection.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s42256-020-00287-7

Cites

63 of 63 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

14 Valid · 1 Not valid · 4 Not checked · 16 Own results and interpretations

10 more claims · 2 with their cited sources checked

Create a free account to see all 35 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 12
  • Probably not a substantial claim. 11

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

15 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 46 references

Research works printed but not linked: 4

  1. Cooper, H. Research Synthesis and Meta-analysis: A Step-by-Step Approach (SAGE Publications, 2015).
  2. Ramos, J. Using TF–IDF to determine word relevance in document queries. In Proc. 1st Instructional Conference on Machine Learning Vol. 242, 133–142 (ICML, 2003).
  3. Vasalou, A., Ng, B. D., Wiemer-Hastings, P. & Oshlyansky, L. Human-moderated remote user testing: orotocols and applications. In 8th ERCIM Workshop, User Interfaces for All Vol. 19 (ERCIM, 2004).
  4. Joffe, H. in Qualitative Research Methods in Mental Health and Psychotherapy: A Guide for Students and Practitioners (eds Harper, D. & Thompson, A. R.) Ch. 15 (Wiley, 2012).

Cited by

See all 1,157

Metadata

Authors
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., Harkema, A., Willemsen, J., Ma, Y., Fang, Q., Hindriks, S., Tummers, L., & Oberski, D. L.
Affiliations
Utrecht University (NL)
Venue
Nature Machine Intelligence
Published
2021-02-01
Volume
3
Issue
2
Pages
125-133
ISSN
2522-5839 (ISSN-L)
OpenAlex
W3111278950
Cited by
1,157
References
46
Citations by year
2026: 355 · 2025: 329 · 2024: 253 · 2023: 134 · 2022: 60 · 2021: 25
FWCI
4.94
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Machine Intelligence (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
  • arXiv (Cornell University) (cc-by)
Funders
This project was funded by the Innovation Fund for IT in Research Projects, Utrecht University, The Netherlands.
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
systematic reviews, meta-analysis, title and abstract screening, active learning (machine learning), ASReview, open source software, literature search automation, class imbalance, reproducibility, user experience

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: An open source machine learning framework for efficient and transparent systematic reviews. https://app.cochat.ai/papers/10.1038/s42256-020-00287-7
The paper
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., Harkema, A., Willemsen, J., Ma, Y., Fang, Q., Hindriks, S., Tummers, L., & Oberski, D. L. (2021). An open source machine learning framework for efficient and transparent systematic reviews. Nature Machine Intelligence. https://doi.org/10.1038/s42256-020-00287-7

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.