ASReview LAB v.2: Open-source text screening with multiple agents and a crowd of experts

TL;DR

ASReview LAB v.2 is an open-source Python platform for AI-assisted systematic review screening that adds multiagent model switching and collaborative crowd screening with a shared active-learning model. Using the SYNERGY benchmark, its optimized default model reduced mean loss by 24.1% versus v.1 and retrieved relevant PTSD records after screening far fewer papers in simulation.

Key findings

• ASReview LAB v.2 introduces a multiagent, multiexpert system: multiple AI agents can hand off sequentially within one project, and a crowd of experts can jointly label records from a shared ranked queue.
• Hyperparameter optimization on 24 SYNERGY datasets and 10 prior combinations found the v.2 default (TF-IDF + SVM) had mean validation loss 0.0623 (SD 0.0040), compared with the v.1 default (TF-IDF + naive Bayes) at 0.0821 (SD 0.0081), a 24.1% reduction. MXBAI (+SVM) had loss 0.0610 (SD 0.0046), and E5 (+SVM) had 0.0640 (SD 0.0043).
• In a PTSD systematic review example with 4,455 papers and 38 relevant records, the v.1 default would have found all relevant records after screening 542.1 papers (±189.47); the v.2 Ultra model required 270.6 papers (±11.48), ELAS-heavy required 266.8 (±25.84), and ELAS-lang required 269.9 (±2.88).
• The platform logs project setup, expert actions, and training-cycle information to balance reproducibility and storage, while supporting simulation, model switching, stopping suggestions, and extensions such as Insights, Makita, and Dory.

Methods

• Design: software and infrastructure paper describing ASReview LAB v.2’s multiagent active-learning architecture, crowd-screening workflow, simulation projects, data storage, and extension ecosystem.
• Data and analysis: hyperparameters were optimized on 24 labeled systematic reviews from the SYNERGY dataset, with 10 prior combinations per dataset sampled using a geometric distribution (p = 0.5) for test and validation sets; Optuna minimized mean loss across datasets and prior combinations.
• The PTSD review example used a fully labeled dataset of 4,455 papers with 38 relevant records to compare recall and screening effort across v.1 and v.2 model configurations.

Limitations

• Stated: automatic hand-off based on performance metrics is planned but not yet implemented; future work includes simultaneous multiagent ensembles, noise-robust learning, and data-driven stopping estimators. The paper also notes that v.2 simulations assume all labels in fully labeled datasets are accurate, and that non-text data simulation is currently supported only via CLI and Python API, not the web app.
• Stated: crowd screening assigns each record to only one expert, so no interrater reliability can be computed; the paper says this is not the goal of the crowd-screening option.
• Evident from the text: the reported performance evidence comes from simulations on already-labeled datasets, not from a prospective real-world evaluation of crowd screening or multiagent hand-off in practice.