The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Optimizing ASReview simulations: A generic multiprocessing solution for ‘light-data’and ‘heavy-data’users
Abstract
Active learning can be used for optimizing and speeding up the screening phase of systematic reviews. Running simulation studies mimicking the screening process can be used to test the performance of different machine-learning models or to study the impact of different training data. This paper presents an architecture design with a multiprocessing computational strategy for running many such simulation studies in parallel, using the ASReview Makita workflow generator and Kubernetes software for deployment with cloud technologies. We provide a technical explanation of the proposed cloud architecture and its usage. In addition to that, we conducted 1140 simulations investigating the computational time using various numbers of CPUs and RAM settings. Our analysis demonstrates the degree to which simulations can be accelerated with multiprocessing computing usage. The parallel computation strategy and the architecture design that was developed in the present paper can contribute to future research with more optimal simulation time and, at the same time, ensure the safe completion of the needed processes.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Alternatively, one could benchmark existing models but for different datasets.26,30,31,32,33,34”
From ASReview LAB v.2: Open-source text screening with multiple agents and a crowd of experts (2025) · cites this paper as [31]
In the citing paper
“Additionally, simulation tasks can be parallelized and scaled on infrastructure like Kubernetes to handle extensive workloads.31”
From ASReview LAB v.2: Open-source text screening with multiple agents and a crowd of experts (2025) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Singularity: Scientific containers for mobility of compute
Kurtzer, G. M., Sochat, V., & Bauer, M. W. (2017) · PLOS ONE · doi:10.1371/journal.pone.0177459 · Cited by 2,740
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157 · cited 1×
“The open-source research infrastructure ASReview [9] is specialized in running transparent and fully reproducible simulation studies.”
BackgroundSelf-citationReducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 1×
“…systematic review implies marking scientific papers as relevant or irrelevant “by hand”, the process of labeling can be optimized with the use of machine learning (ML) – especially pipelines based on active learning [1-3], a constant interaction between a human and a machine learning model.”
BackgroundGreen Cloud Computing and Environmental Sustainability
Kumar, S., & Buyya, R. (2012) · Harnessing Green It · doi:10.1002/9781118305393.ch16 · Cited by 211
Hamel, C., Kelly, S. E., Thavorn, K., Rice, D. B., Wells, G. A., & Hutton, B. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01129-1 · Cited by 140
Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., Teijema, J. J., & van de Schoot, R. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02257-7 · Cited by 96 · cited 1×
“…point during the screening phase [5], the work saved compared to manual screening to arrive at a given level of recall (WSS) [6], or the Average Time to Discovery (ATD) - a measure of how long it takes on average (expressed as a percentage of screened articles) to find a relevant record [7].”
BackgroundSelf-citationvan de Schoot, R., Sijbrandij, M., Depaoli, S., Winter, S. D., Olff, M., & van Loey, N. E. (2018) · Multivariate Behavioral Research · doi:10.1080/00273171.2017.1412293 · Cited by 85
Campos, D. G., Fütterer, T., Gfrörer, T., Lavelle-Hill, R., Murayama, K., König, L., Hecht, M., Zitzmann, S., & Scherer, R. (2024) · Educational Psychology Review · doi:10.1007/s10648-024-09862-5 · Cited by 75 · cited 1×
“For instance, it could imply an examination of the impact of different training datasets [13–15], tests of ML models [16,17,11], or evaluation of multiple datasets [18].”
BackgroundActive learning for screening prioritization in systematic reviews - A simulation study
Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., & van de Schoot, R. (2020) · doi:10.31219/osf.io/w6qbg · Cited by 51
Teijema, J. J., Hofstee, L., Brouwer, M., de Bruin, J., Ferdinands, G., de Boer, J., Vizan, P., van den Brand, S., Bockting, C., van de Schoot, R., et al. (2023) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2023.1178181 · Cited by 36
Self-citation
Research works printed but not linked: 32
- Settles, 2009, Active learning literature survey
- Teijema, 2023, Simulation-based active learning for systematic reviews: a systematic review of the literature
- Yerushalmy, 1947, Statistical problems in assessing methods of medical diagnosis, with special reference to X-ray techniques, Public Health Reports (1896-1970), 1432-1449
- Cohen, 2006, Reducing workload in systematic review preparation using automated citation classification, Journal of the American Medical Informatics Association, 13, 206, 10.1197/jamia.M1929 · DOI · Paper page
- Van De Schoot, 2021, An open source machine learning framework for efficient and transparent systematic reviews, Nature Machine Intelligence, 3, 125, 10.1038/s42256-020-00287-7 · DOI · Paper page
- Teijema, 2023, ASReview Makita: a workflow generator for simulation studies using the command line interface of ASReview LAB
- Ambert, 2013, Virk: an active learning-based system for bootstrapping knowledge base development in the neurosciences, Frontiers In Neuroinformatics, 7, 38, 10.3389/fninf.2013.00038 · DOI · Paper page
- Byrne, 2023, The influence of active learning model and prior knowledge choice on how long it takes to find hard-to-find relevant papers: Examining the variability of the time to discovery and the stability of its rank-orders
- Oude Wolcherink, 2023, Can artificial intelligence separate the wheat from the chaff in systematic reviews of health economic articles?, Expert Review of Pharmacoeconomics & Outcomes Research, 23, 1049, 10.1080/14737167.2023.2234639 · DOI
- Neeleman, 2022, The issue of reconstructing a database using search queries and its possible solution
- Harmsen, 2021, Artificial intelligence supports literature screening in medical guideline development: towards up-to-date medical guidelines
- Teijema, 2023, Large-Scale simulation study of active learning models for systematic reviews, 10.31234/osf.io/2w3rm · DOI
- Neudorfer, 2012, Efficient parallelization of a three-dimensional high-order particle-in-cell method for the simulation of a 170 GHz gyrotron resonator, IEEE Transactions on Plasma Science, 41, 87, 10.1109/TPS.2012.2229298 · DOI
- Shu, 2015, On speeding up stochastic simulations by parallelization of random number generation, Chemical Engineering Science, 137, 828, 10.1016/j.ces.2015.06.066 · DOI
- Gottlieb, 1989, Highly parallel computing
- Grossman, 2012, Introducing parallelism and concurrency in the data structures course, 505, 10.1145/2157136.2157285 · DOI
- Scheepers, 2014, Virtualization and containerization of application infrastructure: a comparison, 21st Twente Student Conference on IT, 21, 1
- Merkel, 2014, Docker: lightweight linux containers for consistent development and deployment, Linux Journal, 2014, 2
- Tesliuk, 2019, Kubernetes container orchestration as a framework for flexible and effective scientific data analysis, 67, 10.1109/ISPRAS47671.2019.00016 · DOI
- Erl, 2005, Service-oriented architecture: concepts, technology, and design
- Khan, 2017, Key characteristics of a container orchestration platform to enable a modern application, IEEE Cloud Computing, 4, 42, 10.1109/MCC.2017.4250933 · DOI
- Kerrisk, 2012, LCE: The failure of operating systems and how we can fix it
- ASReview LAB Developers, 2023, ASReview LAB - A tool for AI-assisted systematic reviews
- Teijema, 2023, ASReview Makita: a workflow generator for simulation studies using the command line interface of ASReview LAB
- Tange, 2011, Gnu parallel-the command-line power tool, Usenix Mag, 36, 42
- Eager, 1989, Speedup versus efficiency in parallel systems, IEEE Transactions on Computers, 38, 408, 10.1109/12.21127 · DOI
- Prasad, 2015, Topics in Parallel and Distributed Computing.
- De Bruin, 2023, SYNERGY - Open machine learning dataset on study selection in systematic reviews
- Kroft, 2022, Language morphology in active learning aided systematic reviews
- Thekkath, 1994, Impact of sharing-based thread placement on multithreaded architectures, ACM SIGARCH Computer Architecture News, 22, 176, 10.1145/192007.192027 · DOI
- Chen, 2016, A parallel random forest algorithm for big data in a spark cloud computing environment, IEEE Transactions on Parallel and Distributed Systems, 28, 919, 10.1109/TPDS.2016.2603511 · DOI
- Keckler, GPUs and the future of parallel computing, 10.1109/MM.2011.89 · DOI
Cited by
ASReview LAB v.2: Open-source text screening with multiple agents and a crowd of experts
de Bruin, J., Lombaers, P., Kaandorp, C., Teijema, J., van der Kuil, T., Yazan, B., Dong, A., & van de Schoot, R. (2025) · Patterns · doi:10.1016/j.patter.2025.101318 · Cited by 27
Self-citationMachine learning to optimize literature screening in medical guideline development
Harmsen, W., de Groot, J., Harkema, A., van Dusseldorp, I., de Bruin, J., van den Brand, S., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02590-5 · Cited by 21
Self-citationLarge-scale simulation study of active learning models for systematic reviews
Teijema, J. J., de Bruin, J., Bagheri, A., & van de Schoot, R. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00777-0 · Cited by 13
Self-citation
Metadata
- Authors
- Romanov, S., Siqueira, A. S., de Bruin, J., Teijema, J., Hofstee, L., & van de Schoot, R.
- Affiliations
- Applied Data Science, Department of Information and Computing Science, Faculty of Science, Utrecht University
- Netherlands eScience Center, Amsterdam, NL
- Department of Research and Data Management Services, Information Technology Services, Utrecht University, Utrecht, the Netherlands
- Department of Methodology and Statistics, Faculty of Social and Behavioral Sciences, Utrecht University
- Venue
- Data Intelligence
- Published
- 2024
- Volume
- 6
- Issue
- 2
- Pages
- 320-343
- ISSN
- 2641-435X
- Cited by
- 7
- References
- 10
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
