The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index; 18 of the 30 sources its claims cite were checked for support.
AIM review tool: artificial intelligence for smarter systematic review screening
Open accessAbstract
In this study, we present the AIM Review Tool , a modern web-based application that integrates active and supervised machine learning to accelerate the screening of publications for systematic reviews. AIM Review combines advanced text vectorization methods with machine learning models executed directly in the web browser, enabling rapid and privacy-preserving analysis. Unlike existing tools, AIM Review uniquely incorporates nested cross-validation and semi-automated screening strategies, enhancing both efficiency and precision in evidence synthesis. Using six real-world case studies across various topics, we demonstrate substantial workload reductions through active learning, with the percentage of publications not requiring screening while achieving ≥95% recall (WSS 95% ) ranging from 20% to 95%. Supervised learning pipelines trained on a subset of screened records predicted the relevance of unscreened publications with balanced accuracies between 75% and 87%. AIM Review provides a flexible, scalable, and accessible solution for large-scale literature screening and can be readily integrated into existing manual workflows.
Overview
CoChat Summary
## TL;DR
AIM Review is a client-side web application that combines active learning and nested cross-validated supervised machine learning to speed up title/abstract screening in systematic reviews. Across six case studies, it reduced screening workload substantially with active learning and achieved balanced accuracies of 75% to 87% for supervised prediction.
## Key findings
- Across six real-world case studies, active learning produced workload savings up to WSS95% of 95%.
- For the first three case studies, active learning reduced screening burden from 16,660 to 2,448 publications in case study 1, from 5,536 to 457 in case study 2, and from 729 to 561 in case study 3.
- The paper reports that supervised learning on a 20% training sample and 80% held-out sample achieved balanced accuracies between 75% and 87% across case studies and model configurations.
- Reported active-learning savings varied by case study: case study 1 improved from WSS95% 73.32% to 85.35%; case study 2 from 36.67% to 91.60%; case study 3 from 13.87% to 22.94%.
## Methods
- Built a client-side web application with three modules: labeling, inter-rater agreement, and NCV-based machine-learning prediction; all computations run in the browser.
- Benchmarked the system on six systematic review case studies: three from psychology/psychiatry and three additional studies from computer science, environmental health, and endocrinology.
- For supervised learning, used 20% of publications for training and 80% for OOCV, with nested 5×5 cross-validation and 5 outer permutations; evaluated classifiers including logistic regression, linear SVM, and sequential neural networks with multiple text vectorization methods and ensemble strategies.
## Limitations
- The paper states that browser/front-end computational resources are inherently constrained, although the system was optimized for efficiency.
- The benchmarking lacked a large systematic review (>10,000 entries) with a balanced percentage of relevant publications; the text says such studies are rarely available. This is a limitation stated in the paper.
- The authors did not benchmark across different devices and browsers, did not test every possible classifier/vectorizer/ensemble combination, and did not include deep learning models in active-learning benchmarking because of computational/time cost; these are stated in the paper.
- The evaluation was simulation-based and did not account for user learning time, browser memory usage, algorithmic biases, disagreement with traditional screening, or direct comparison with traditional manual screening; this is stated in the paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
8 Valid · 6 Not valid · 5 Not checked · 60 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“However, maintaining a high recall rate (i.e., identifying most relevant studies) when the proportion of relevant studies is larger often results in minimal time savings, as a substantial portion of literature still requires manual review 22 .”
What the sources say
- Disputedcites [22] Callaghan, M. W. & Müller-Hansen, F. Statistical stopping criteria for automated screening in systematic reviews. Syst. Rev. 9, 273 (2020). sourceExperimental — accuracy not yet measured
The Results (page unavailable) say that a low proportion of relevant documents can require extensive screening to establish high recall with confidence, leading to smaller work savings. The Discussion (page unavailable) says that for large datasets with a significant proportion of relevant documents, the additional effort is small compared with the work saved—the opposite of the cited claim about a larger relevant proportion. (LLM, conf=0.90)
Checked by a reasoning model — may be wrong.
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Some cited sources support this claim and some don’t — read each source’s verdict below.
In the paper
“Previous benchmark studies showed that substantial workload reductions with active learning mostly occur in large sample sizes ( > 2000 publications) where approximately less than 5% of publications are relevant 13,20 .”
What the sources say
- Disputedcites [13] Cohen, A. M., Hersh, W. R., Peterson, K. & Yen, P.-Y. Reducing workload in systematic review preparation using automated citation classification. J. Am. Med. Inform. Assoc. 13, 206–219 (2006). sourceExperimental — accuracy not yet measured
The Discussion reports substantial workload savings for several reviews, but explicitly says performance had essentially no correlation with sample size or the fraction of positive samples. It therefore contradicts the claim that savings mostly occurred in collections of more than 2,000 publications with less than 5% relevant. The paper describes automated classification, not active learning. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
- Partly supportedcites [20] Ferdinands, G. et al. Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the average time to discover relevant records. Syst. Rev. 12, 100 (2023). sourceExperimental — accuracy not yet measured
The Abstract reports substantial workload reductions, and the Discussion notes that the two worst-performing datasets had the highest proportions of relevant publications, about 5%. But the Discussion explicitly says the six-dataset sample cannot establish how performance relates to dataset characteristics. The paper does not support the claimed >2,000-publication threshold or the conclusion that substantial reductions mostly occur below 5% relevance. (LLM, conf=0.83)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
Every source the paper cites for this claim was checked and supports it.
In the paper
“Unlike existing semi-automated screening tools 9,10,14 , which primarily use active learning to prioritize records for manualscreening,AIMReviewadditionally includes a dedicated supervised prediction work fl ow.”
What the sources say
- Supportedcites [9] van de Schoot, R. et al. An open source machine learning framework for efficient and transparent systematic reviews. Nat. Mach. Intell. 3, 125–133 (2021). source
The Pipeline section describes ASReview using an active-learning cycle to present records for a user to screen and label, then ordering unlabelled records by predicted relevance. This supports the cited characterization of an existing tool; the claim about AIM Review’s additional workflow is the citing authors’ own comparison. (LLM, conf=0.92)
- Supportedcites [10] Przybyła, P. et al. Prioritising references for systematic reviews with RobotAnalyst: a user study. Res Synth. Methods 9, 470–488 (2018). source
The Introduction says multiple software solutions use machine-learning prioritisation to assist review screening. Methods > System functionality describes RobotAnalyst using an active-learning loop to recommend references for manual screening. The statement about AIM Review’s separate workflow is the citing authors’ description of their own tool, not a claim attributed to this source. (LLM, conf=0.88)
- Supportedcites [14] Ouzzani, M., Hammady, H., Fedorowicz, Z. & Elmagarmid, A. Rayyan—a web and mobile app for systematic reviews. Syst. Rev. 5, 210 (2016). source
The Methods section (‘Predicting included and excluded studies’; page unavailable) describes Rayyan learning from reviewers’ include/exclude decisions, repeatedly updating its classifier, and rating unscreened records for review. The Pilot testing section says those ratings identified potentially eligible studies for further consideration. This supports the description of Rayyan as an existing tool that prioritizes records for manual screening; the statement about AIM Review’s separate workflow is not attributed to this source. (LLM, conf=0.85)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“However, active learning has inherent limitations: it can be biased by the small number of initial screenings, leading to unrepresentative prioritizations 21 .”
What the sources say
- Partly supportedcites [21] Harmsen, W. et al. Machine learning to optimize literature screening in medical guideline development. Syst. Rev. 13, 177 (2024). sourceExperimental — accuracy not yet measured
The Methods > Simulation passage says the authors varied the initial relevant paper across runs to mitigate potential bias from the starting paper in the model’s first iteration. This supports sensitivity to initial training records, but the paper does not say that a small number of initial screenings causes unrepresentative prioritizations. (LLM, conf=0.84)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“When using single text vectorization methods alone, pre-trained, transformer models (SPECTER2, all-MiniLM-L6-v2) outperformed simpler text vectorization techniques (e.g., TF-IDF and LSA of TF-IDF), highlighting the bene fi ts of contextualized embeddings 34 -though at the cost of higher computational cost in large-scale screenings.”
What the sources say
- Partly supportedcites [34] Garrido-Merchan, E. C., Gozalo-Brizuela, R. & Gonzalez-Carvajal, S. Comparing BERT against traditional machine learning models in text classification. J. Comput. Cogn. Eng. 2, 352–356 (2023). sourceExperimental — accuracy not yet measured
The Abstract and §5 (Conclusions and Further Work; pages not provided) report that BERT outperformed TF-IDF-based traditional classifiers, supporting the general comparison. The paper does not report results for SPECTER2, all-MiniLM-L6-v2, or LSA of TF-IDF, so it cannot substantiate the claim about those specific methods. (LLM, conf=0.87)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
29 more claims · 4 with their cited sources checked
Create a free account to see all 79 claims and every source verdict.
- It may state two separate claims as one. 33
- Probably not a substantial claim. 26
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092 · cited 2×
“In recent years, several AI software tools have emerged to speed up systematic review screening 9 – 14 and automatic data extraction 15 – 17 .”
Wolpert, D. H. (1992) · Neural Networks · doi:10.1016/s0893-6080(05)80023-1 · Cited by 7,951
A STATISTICAL INTERPRETATION OF TERM SPECIFICITY AND ITS APPLICATION IN RETRIEVAL
SPARCK JONES, K. (1972) · Journal of Documentation · doi:10.1108/eb026526 · Cited by 4,505 · cited 1×
“(cid:129) Term frequency-inverse document frequency (TF-IDF; scikit-learn) 42 — default option (cid:129) Latent semantic analysis (LSA; scikit-learn) of TF-IDF using truncated singular value decomposition (cid:129) Universal sentence encoder (univBERT; tensor fl ow.js) 43 (cid:129) Document to…”
Bias in error estimation when using cross-validation for model selection
Varma, S., & Simon, R. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-91 · Cited by 2,485 · cited 1×
“Toaddressthesechallenges,nestedandcross-validated(NCV) supervisedMLoffersarobustalternative 25 .”
Universal Sentence Encoder for English
Cer, D., Yang, Y., Kong, S.-Y., Hua, N., Limtiaco, N., St. John, R., Constant, N., Guajardo-Cespedes, M., Yuan, S., Tar, C., et al. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2029 · Cited by 1,750 · cited 1×
“…analysis (LSA; scikit-learn) of TF-IDF using truncated singular value decomposition (cid:129) Universal sentence encoder (univBERT; tensor fl ow.js) 43 (cid:129) Document to vector embedding model (Doc2Vec; tensor fl ow.js native implementation) 44 (cid:129) Pretrained sentence transformers (e.g.,…”
Bornmann, L., & Mutz, R. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23329 · Cited by 1,576 · cited 1×
“The rapid expansion of scienti fi c knowledge, fueled by an ever-growing number of researchers and publications, has brought both unparalleled opportunities and challenges for academia 1,2 .”
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483
Acosta, J. N., Falcone, G. J., Rajpurkar, P., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-022-01981-2 · Cited by 1,365
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157 · cited 3×
“In recent years, several AI software tools have emerged to speed up systematic review screening 9 – 14 and automatic data extraction 15 – 17 .”
Research works printed but not linked: 2
- Martín A. et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. In Proc. 12th USENIX Conference on Operating Systems Design and Implementation 265–283 (USENIX Association, 2016).
- Le, Q. & Mikolov, T. Distributed representations of sentences and documents. In International conference on machine learning 1188–1196 (PMLR, 2014).
Metadata
- Authors
- Mena, S., Rituerto-González, E., Coutts, F., von Trott, J., Jacobs, G. R., Bryant, L., Moles, L., Sirbu, N., Promet, L., Oliver, D., Ahmed, M. S., Fusar-Poli, P., Bakermans-Kranenburg, M. J., van IJzendoorn, M. H., Koutsouleris, N., & Lalousis, P. A.
- Affiliations
- King's College London (GB)
- Max Planck Institute of Psychiatry (DE)
- Ludwig-Maximilians-Universität München (DE)
- Corresponding author
- Mena, Sergio
- ORCID
- Venue
- npj Artificial Intelligence
- Published
- 2026-02-21
- Volume
- 2
- Issue
- 1
- ISSN
- 3005-1460 (ISSN-L)
- OpenAlex
- W7130833580
- Cited by
- 0
- References
- 43
- FWCI
- 0.00
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor)
- Locations
- npj Artificial Intelligence (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- systematic review screening, active learning (machine learning), supervised learning, web application, text vectorization, nested cross-validation, semi-automated screening, evidence synthesis, privacy-preserving data analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: AIM review tool: artificial intelligence for smarter systematic review screening. https://app.cochat.ai/papers/10.1038/s44387-026-00080-8
- The paper
- Mena, S., Rituerto-González, E., Coutts, F., von Trott, J., Jacobs, G. R., Bryant, L., Moles, L., Sirbu, N., Promet, L., Oliver, D., Ahmed, M. S., Fusar-Poli, P., Bakermans-Kranenburg, M. J., van IJzendoorn, M. H., Koutsouleris, N., & Lalousis, P. A. (2026). AIM review tool: artificial intelligence for smarter systematic review screening. npj Artificial Intelligence. https://doi.org/10.1038/s44387-026-00080-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.