CoChat

The DOI matches its Crossref record; 51 of the 51 references it cites were found in a registry or an index; 18 of the 30 sources its claims cite were checked for support.

AIM review tool: artificial intelligence for smarter systematic review screening

Open access
Mena, Sergio, Rituerto-González, Esther, Coutts, Fiona, von Trott, Jana, Jacobs, Grace R., Bryant, Linda, Moles, Louise, Sirbu, Nicoleta, Promet, Liisi, Oliver, Dominic, Ahmed, Muhammad S., Fusar-Poli, Paolo, Bakermans-Kranenburg, Marian J., van IJzendoorn, Marinus H., Koutsouleris, Nikolaos, Lalousis, Paris Alexandros · 2026 · npj Artificial Intelligence

Abstract

In this study, we present the AIM Review Tool , a modern web-based application that integrates active and supervised machine learning to accelerate the screening of publications for systematic reviews. AIM Review combines advanced text vectorization methods with machine learning models executed directly in the web browser, enabling rapid and privacy-preserving analysis. Unlike existing tools, AIM Review uniquely incorporates nested cross-validation and semi-automated screening strategies, enhancing both efficiency and precision in evidence synthesis. Using six real-world case studies across various topics, we demonstrate substantial workload reductions through active learning, with the percentage of publications not requiring screening while achieving ≥95% recall (WSS 95% ) ranging from 20% to 95%. Supervised learning pipelines trained on a subset of screened records predicted the relevance of unscreened publications with balanced accuracies between 75% and 87%. AIM Review provides a flexible, scalable, and accessible solution for large-scale literature screening and can be readily integrated into existing manual workflows.

Overview

CoChat Summary

## TL;DR

AIM Review is a client-side web application that combines active learning and nested cross-validated supervised machine learning to speed up title/abstract screening in systematic reviews. Across six case studies, it reduced screening workload substantially with active learning and achieved balanced accuracies of 75% to 87% for supervised prediction.

## Key findings

  • Across six real-world case studies, active learning produced workload savings up to WSS95% of 95%.
  • For the first three case studies, active learning reduced screening burden from 16,660 to 2,448 publications in case study 1, from 5,536 to 457 in case study 2, and from 729 to 561 in case study 3.
  • The paper reports that supervised learning on a 20% training sample and 80% held-out sample achieved balanced accuracies between 75% and 87% across case studies and model configurations.
  • Reported active-learning savings varied by case study: case study 1 improved from WSS95% 73.32% to 85.35%; case study 2 from 36.67% to 91.60%; case study 3 from 13.87% to 22.94%.

## Methods

  • Built a client-side web application with three modules: labeling, inter-rater agreement, and NCV-based machine-learning prediction; all computations run in the browser.
  • Benchmarked the system on six systematic review case studies: three from psychology/psychiatry and three additional studies from computer science, environmental health, and endocrinology.
  • For supervised learning, used 20% of publications for training and 80% for OOCV, with nested 5×5 cross-validation and 5 outer permutations; evaluated classifiers including logistic regression, linear SVM, and sequential neural networks with multiple text vectorization methods and ensemble strategies.

## Limitations

  • The paper states that browser/front-end computational resources are inherently constrained, although the system was optimized for efficiency.
  • The benchmarking lacked a large systematic review (>10,000 entries) with a balanced percentage of relevant publications; the text says such studies are rarely available. This is a limitation stated in the paper.
  • The authors did not benchmark across different devices and browsers, did not test every possible classifier/vectorizer/ensemble combination, and did not include deep learning models in active-learning benchmarking because of computational/time cost; these are stated in the paper.
  • The evaluation was simulation-based and did not account for user learning time, browser memory usage, algorithmic biases, disagreement with traditional screening, or direct comparison with traditional manual screening; this is stated in the paper.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s44387-026-00080-8

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Active learning simulation results. Meanandstandard deviation of relevant publications identi fi ed during the screening simulation versus the total publications screened in A case study 1, B case study 2, and C case study 3. The top row (A1, B1, and C1) presents results using a stacked generalization model, while the bottom row (A2, B2, and C2) shows results using feature fusion. In the stacked generalization approach, the selected classi fi ers (LR or L-SVM) are trained independently using features extracted from each text vectorization method (TF-IDF, LSA of TF-IDF, SPECTER2, univBERT, Doc2Vec and all-MiniLM-L6-v2). A meta-model (LR with regularization parameter C = 1) is then trained on the predictions from the base
Fig. 1 Active learning simulation results. Meanandstandard deviation of relevant publications identi fi ed during the screening simulation versus the total publications screened in A case study 1, B case study 2, and C case study 3. The top row (A1, B1, and C1) presents results using a stacked generalization model, while the bottom row (A2, B2, and C2) shows results using feature fusion. In the stacked generalization approach, the selected classi fi ers (LR or L-SVM) are trained independently using features extracted from each text vectorization method (TF-IDF, LSA of TF-IDF, SPECTER2, univBERT, Doc2Vec and all-MiniLM-L6-v2). A meta-model (LR with regularization parameter C = 1) is then trained on the predictions from the baseFrom the PDF
Fig. 2 Supervised learning results. ROCcurves at OOT (20% of publications) and OOCVlevel (80% of publications) are shown for A case study 1, B case study 2 and C case study 3. The top row (A1, B1, and C1) presents results using a stacked generalization model, while the bottom row (A2, B2, and C2) shows results using feature fusion. In the stacked generalization approach, the chosen classi fi ers (LR, L-SVM or SeqNN) are independently trained using features derived from each text vectorization method (TF-IDF, LSA of TF-IDF, SPECTER2, univBERT, Doc2Vec and all-MiniLM-L6-v2). A meta-model (LR with regularization parameter C = 1) is then trained on the predictions of the base models to produce the fi nal predictions. In the feature fusion approach, the features extracted from all text vectorization methods are concatenated and used to train a single model (LR, L-SVM, or SeqNN) for predicting publication relevance. Models are trained and validated at the OOT level using a NCV framework of 5 outer permutations, fi ve outer folds and fi ve inner folds. The fi nal optimum model is trained with all training publications and tested in
Fig. 2 Supervised learning results. ROCcurves at OOT (20% of publications) and OOCVlevel (80% of publications) are shown for A case study 1, B case study 2 and C case study 3. The top row (A1, B1, and C1) presents results using a stacked generalization model, while the bottom row (A2, B2, and C2) shows results using feature fusion. In the stacked generalization approach, the chosen classi fi ers (LR, L-SVM or SeqNN) are independently trained using features derived from each text vectorization method (TF-IDF, LSA of TF-IDF, SPECTER2, univBERT, Doc2Vec and all-MiniLM-L6-v2). A meta-model (LR with regularization parameter C = 1) is then trained on the predictions of the base models to produce the fi nal predictions. In the feature fusion approach, the features extracted from all text vectorization methods are concatenated and used to train a single model (LR, L-SVM, or SeqNN) for predicting publication relevance. Models are trained and validated at the OOT level using a NCV framework of 5 outer permutations, fi ve outer folds and fi ve inner folds. The fi nal optimum model is trained with all training publications and tested inFrom the PDF
Fig. 3 Proposed work fl ow of AIM Review applications. This work fl ow is provided to guide users in applying the different modules of the application. Given the fl exibility of AIMReview , uncertainty about how to con fi gure the available options may arise; the work fl ow illustrates recommended practices to support consistent and effective use. Following database searches, users can import the results into the labeling application, which automatically removes duplicates, allowing them to seamlessly begin screening titles and abstracts. Once a representative number of relevant and irrelevant publications have been screened, users can activate the active learning engine to generate relevance scores, guiding their manual labeling. For large systematic reviews with few expected relevant publications, the optimal strategy is to sort entries by relevance using the active learning engine and label the most relevant publications until none remain. Conversely, if many publications are expected to be relevant, users can screen a subset of publications -either sorted by relevance or not
Fig. 3 Proposed work fl ow of AIM Review applications. This work fl ow is provided to guide users in applying the different modules of the application. Given the fl exibility of AIMReview , uncertainty about how to con fi gure the available options may arise; the work fl ow illustrates recommended practices to support consistent and effective use. Following database searches, users can import the results into the labeling application, which automatically removes duplicates, allowing them to seamlessly begin screening titles and abstracts. Once a representative number of relevant and irrelevant publications have been screened, users can activate the active learning engine to generate relevance scores, guiding their manual labeling. For large systematic reviews with few expected relevant publications, the optimal strategy is to sort entries by relevance using the active learning engine and label the most relevant publications until none remain. Conversely, if many publications are expected to be relevant, users can screen a subset of publications -either sorted by relevance or notFrom the PDF
Fig. 4 Active learning and supervised learning frameworks in AIM Review.
Fig. 4 Active learning and supervised learning frameworks in AIM Review.From the PDF
papers appear. B Supervised learning pipeline for ML prediction. A subset of documents (X%) is manually screened to train and optimize model parameters via nested cross-validation. After fi nding the optimal hyperparameters, the model is applied to all screened samples, and an ROC curve is created to assess sensitivity and speci fi city at different relevance thresholds. The remaining publications (100-X%) are labeled using a user-de fi ned threshold, with higher thresholds (e.g., 0.5 -0.75) labeling only highly probable relevant papers, and lower thresholds (e.g., 0.25-0.5) allowing more papers to be labeled as relevant. Artwork was created using Flaticon designs. Abbreviations: ROC receiver operating characteristic.
papers appear. B Supervised learning pipeline for ML prediction. A subset of documents (X%) is manually screened to train and optimize model parameters via nested cross-validation. After fi nding the optimal hyperparameters, the model is applied to all screened samples, and an ROC curve is created to assess sensitivity and speci fi city at different relevance thresholds. The remaining publications (100-X%) are labeled using a user-de fi ned threshold, with higher thresholds (e.g., 0.5 -0.75) labeling only highly probable relevant papers, and lower thresholds (e.g., 0.25-0.5) allowing more papers to be labeled as relevant. Artwork was created using Flaticon designs. Abbreviations: ROC receiver operating characteristic.From the PDF

Verification

Claims the paper makes

8 Valid · 6 Not valid · 5 Not checked · 60 Own results and interpretations

29 more claims · 4 with their cited sources checked

Create a free account to see all 79 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 33
  • Probably not a substantial claim. 26

Create a free account to see these claims.

Citations

Cites

Show all 51 references

Research works printed but not linked: 2

  1. Martín A. et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. In Proc. 12th USENIX Conference on Operating Systems Design and Implementation 265–283 (USENIX Association, 2016).
  2. Le, Q. & Mikolov, T. Distributed representations of sentences and documents. In International conference on machine learning 1188–1196 (PMLR, 2014).

Metadata

Authors
Mena, S., Rituerto-González, E., Coutts, F., von Trott, J., Jacobs, G. R., Bryant, L., Moles, L., Sirbu, N., Promet, L., Oliver, D., Ahmed, M. S., Fusar-Poli, P., Bakermans-Kranenburg, M. J., van IJzendoorn, M. H., Koutsouleris, N., & Lalousis, P. A.
Affiliations
  • King's College London (GB)
  • Max Planck Institute of Psychiatry (DE)
  • Ludwig-Maximilians-Universität München (DE)
Corresponding author
Mena, Sergio
Venue
npj Artificial Intelligence
Published
2026-02-21
Volume
2
Issue
1
ISSN
3005-1460 (ISSN-L)
OpenAlex
W7130833580
Cited by
0
References
43
FWCI
0.00
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Artificial Intelligence (publishedVersion, cc-by-nc-nd)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
systematic review screening, active learning (machine learning), supervised learning, web application, text vectorization, nested cross-validation, semi-automated screening, evidence synthesis, privacy-preserving data analysis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: AIM review tool: artificial intelligence for smarter systematic review screening. https://app.cochat.ai/papers/10.1038/s44387-026-00080-8
The paper
Mena, S., Rituerto-González, E., Coutts, F., von Trott, J., Jacobs, G. R., Bryant, L., Moles, L., Sirbu, N., Promet, L., Oliver, D., Ahmed, M. S., Fusar-Poli, P., Bakermans-Kranenburg, M. J., van IJzendoorn, M. H., Koutsouleris, N., & Lalousis, P. A. (2026). AIM review tool: artificial intelligence for smarter systematic review screening. npj Artificial Intelligence. https://doi.org/10.1038/s44387-026-00080-8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.