Iterative guided machine learning-assisted systematic literature reviews: A diabetes case study
Open accessPublished version of a preprint
Abstract
Background: Systematic Reviews (SR), studies of studies, use a formal process to evaluate the quality of scientific literature and determine ensuing effectiveness from qualifying articles to establish consensus findings around a hypothesis. Their value is increasing as the conduct and publication of research and evaluation has expanded and the process of identifying key insights becomes more time consuming. Text analytics and machine learning (ML) techniques may help overcome this problem of scale while still maintaining the level of rigor expected of SRs. Methods: In this article, we discuss an approach that uses existing examples of SRs to build and test a method for assisting the SR title and abstract pre-screening by reducing the initial pool of potential articles down to articles that meet inclusion criteria. Our approach differs from previous approaches to using ML as a SR tool in that it incorporates ML configurations guided by previously conducted SRs, and human confirmation on ML predictions of relevant articles during multiple iterative reviews on smaller tranches of citations. We applied the tailored method to a new SR review effort to validate performance. Results: The case study test of the approach proved a sensitivity (recall) in finding relevant articles during down selection that may rival many traditional processes and show ability to overcome most type II errors. The study achieved a sensitivity of 99.5% (213 out of 214) of total relevant articles while only conducting a human review of 31% of total articles available for review. Conclusions: We believe this iterative method can help overcome bias in initial ML model training by having humans reinforce ML models with new and relevant information, and is an applied step towards transfer learning for ML in SR.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Machine learning-assisted systematic reviews can significantly reduce workload and have proven accurate across various fields and review topics.
“In summary, ML-assisted systematic reviews can significantly reduce workload and have proven accurate across various fields and review topics.”
From Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics (2025) · cites this paper as Zimmerman et al. (2021)
The Results (SR ML down selection step 4; page not provided) report 99.5% sensitivity while humans reviewed only 31% of articles, supporting reduced screening work and high accuracy in this case. The Discussion and limitations cautions that technical work may offset time saved and that efficiency at other review volumes remains unanswered. The paper does not establish accuracy across various fields and review topics. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
The Nature of Statistical Learning Theory
Vapnik, V. N. (1995) · doi:10.1007/978-1-4757-2440-0 · Cited by 39,138 · see citing works
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865 · see citing works
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337 · see citing works
Quinlan, J. R. (1986) · Machine Learning · doi:10.1007/bf00116251 · Cited by 21,439 · see citing works
Nearest neighbor pattern classification
Cover, T., & Hart, P. (1967) · IEEE Transactions on Information Theory · doi:10.1109/tit.1967.1053964 · Cited by 16,936 · see citing works
An Introduction to Statistical Learning
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013) · Springer Texts in Statistics · doi:10.1007/978-1-4614-7138-7 · Cited by 12,106 · see citing works
Ensemble methods in machine learning
Dietterich, T. G. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45014-9_1 · Cited by 8,408 · see citing works
The use of the area under the ROC curve in the evaluation of machine learning algorithms
Bradley, A. P. (1997) · Pattern Recognition · doi:10.1016/s0031-3203(96)00142-2 · Cited by 7,217 · see citing works
Supervised Machine Learning: A Review of Classification Techniques
Kotsiantis, S. B. (2007) · Informatica · Cited by 4,192 · see citing works
An Introduction to Statistical Learning: with Applications in R
James, G. M., Witten, D., Hastie, T. J., & Tibshirani, R. J. (2018) · doi:10.25334/q4ht55 · Cited by 2,913 · see citing works
IEEE Transactions on Information Theory
(2023) · IEEE Transactions on Information Theory · doi:10.1109/tit.18 · Cited by 1,856 · see citing works
Munn, Z., Stern, C., Aromataris, E., Lockwood, C., & Jordan, Z. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0468-4 · Cited by 1,164 · see citing works
On the Early History of the Singular Value Decomposition
Stewart, G. W. (1993) · SIAM Review · doi:10.1137/1035134 · Cited by 987 · see citing works
Larsen, P. O., & von Ins, M. (2010) · Scientometrics · doi:10.1007/s11192-010-0202-z · Cited by 964 · see citing works
Optimal Thresholding of Classifiers to Maximize F1 Measure
Lipton, Z. C., Elkan, C., & Naryanaswamy, B. (2014) · Lecture notes in computer science · doi:10.1007/978-3-662-44851-9_15 · Cited by 597 · see citing works
Systematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472 · see citing works
Living systematic reviews: 2. Combining human and machine effort
Thomas, J., Noel-Storr, A., Marshall, I., Wallace, B., McDonald, S., Mavergames, C., Glasziou, P., Shemilt, I., Synnot, A., Turner, T., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.08.011 · Cited by 355 · see citing works
Semi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · see citing works
O’Keeffe, L. M., Greene, R. A., & Kearney, P. M. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-1 · Cited by 192 · see citing works
Bannach-Brown, A., Przybyła, P., Thomas, J., Rice, A. S. C., Ananiadou, S., Liao, J., & Macleod, M. R. (2019) · Systematic Reviews · doi:10.1186/s13643-019-0942-7 · Cited by 185 · see citing works
Feature Engineering and Classifier Ensemble for KDD Cup 2010
Yu, H., Lo, H.-Y., Hsieh, H.-P., Lou, J.-K., McKenzie, T. G., Chou, J.-W., Chung, P.-H., Ho, C.-H., Chang, C.-F., Weng, J.-Y., et al. (2010) · Cited by 125 · see citing works
Xiong, Z., Liu, T., Tse, G., Gong, M., Gladding, P. A., Smaill, B. H., Stiles, M. K., Gillis, A. M., & Zhao, J. (2018) · Frontiers in Physiology · doi:10.3389/fphys.2018.00835 · Cited by 118 · see citing works
Decision threshold adjustment in class prediction
Chen, J. J., Tsai, C., Moon, H., Ahn, H., Young, J., & Chen, C. (2006) · SAR and QSAR in environmental research · doi:10.1080/10659360600787700 · Cited by 88 · see citing works
Machine learning techniques for the automation of literature reviews and systematic reviews in EFSA
Jaspers, S., De Troyer, E., & Aerts, M. (2018) · EFSA Supporting Publications · doi:10.2903/sp.efsa.2018.en-1427 · Cited by 36 · see citing works
Bannach‐Brown, A., Przybyła, P., Thomas, J., Rice, A. S. C., Ananiadou, S., Liao, J., & Macleod, M. R. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/255760 · Cited by 19 · see citing works
How Preprocessing Affects Unsupervised Keyphrase Extraction
Wang, R., Liu, W., & McDonald, C. (2014) · Lecture notes in computer science · doi:10.1007/978-3-642-54906-9_14 · Cited by 18 · see citing works
Dimensionality Reduction for Text using Domain Knowledge
Mao, Y., Balasubramanian, K., & Lebanon, G. (2010) · Cited by 15 · see citing works
Moreno‐García, C. F., Aceves‐Martins, M., & Serratosa, F. (2016) · Computación y Sistemas · doi:10.13053/cys-20-1-2360 · Cited by 10 · see citing works
How Document Pre-processing affects Keyphrase Extraction PerformancePreprint
Boudin, F., Mougard, H., & Cram, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.07809 · Cited by 10 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [15]Ramos, J. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning. 2013:Vol. 242, pp. 133-142.
- [16]Zheng, Alice, and Amanda Casari. Feature engineering for machine learning: principles and techniques for data scientists. " O'Reilly Media, Inc.", 2018.
- [22]Lu S, Jin Z. Improved Stochastic gradient descent algorithm for SVM. Int J Recent Eng Science (IJRES). 2017;4(4):28-31.
Cited by
Sorted by citations
dos Santos, Á. O., da Silva, E. S., Couto, L. M., Reis, G. V. L., & Belo, V. S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104389 · Cited by 65 · see citing works
Natural Language Processing: from Bedside to Everywhere
Aramaki, E., Wakamiya, S., Yada, S., & Nakamura, Y. (2022) · Yearbook of Medical Informatics · doi:10.1055/s-0042-1742510 · Cited by 55 · see citing works
Kebede, M. M., Le Cornet, C., & Fortner, R. T. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1589 · Cited by 50 · see citing works
Sharma, R., Diksha, D., Bhute, A. R., & Bastia, B. K. (2022) · Forensic Science International · doi:10.1016/j.forsciint.2022.111473 · Cited by 45 · see citing works
Haby, M. M., Barreto, J. O. M., Kim, J. Y. H., Peiris, S., Mansilla, C., Torres, M., Guerrero‐Magaña, D. E., & Revéiz, L. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1664 · Cited by 47 · see citing works
Muthu, S. (2023) · International Orthopaedics · doi:10.1007/s00264-022-05672-y · Cited by 20 · see citing works
Cahyanti, R. D., Widyawati, W., & Hakimi, M. (2021) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-021-04212-7 · Cited by 21 · see citing works
Sliwa, S. A., Hawkins, G. T., Lee, S. M., & Hunt, H. (2023) · Journal of School Health · doi:10.1111/josh.13374 · Cited by 20 · see citing works
Automation of Legal Precedents Retrieval: Findings from a Literature Review
Mentzingen, H., António, N., & Bacao, F. (2023) · International Journal of Intelligent Systems · doi:10.1155/2023/6660983 · Cited by 16 · see citing works
A scoping review and analysis of green construction research: a machine learning aided approach
Fernando, A., Siriwardana, C., Law, D. W., Gunasekara, C., Zhang, K., & Gamage, K. G. P. N. (2024) · Smart and Sustainable Built Environment · doi:10.1108/sasbe-08-2023-0201 · Cited by 11 · see citing works
A Rapid Semi-automated Literature Review on Legal Precedents Retrieval
Mentzingen, H., António, N., & Bação, F. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16474-3_5 · Cited by 11 · see citing works
Nshakira‐Rukundo, E., Tabe‐Ojong, M. P. J., Gebrekidan, B. H., Agaba, M., Padmaja, S. S., & Dhehibi, B. (2026) · Communications Earth & Environment · doi:10.1038/s43247-025-03171-7 · Cited by 8 · see citing works
Luomala, M., Naarmala, J., & Tuomi, V. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.02.133 · Cited by 9 · see citing works
Xu, S., Zhao, Z., Liu, X., & Meng, X.-L. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03138-w · Cited by 6 · see citing works
Aguirre, A. M. G., & Figueroa-Vallejo, C. J. (2022) · Revista de Investigación Desarrollo e Innovación · doi:10.19053/20278306.v12.n2.2022.15271 · Cited by 6 · see citing works
Towards Automated Screening of Literature on Artificial Intelligence in Nursing
Moen, H., Alhuwail, D., Björne, J., Block, L., Celin, S., Jeon, E., Kreiner, K., Mitchell, J. L., Ozegovic, G., Ronquillo, C., et al. (2022) · Studies in health technology and informatics · doi:10.3233/shti220155 · Cited by 5 · see citing works
A survey of Inclusive Information Access
Zheng, Y., Liu, H., & Wald, M. (2025) · Information Retrieval Research · doi:10.54195/irrj.21092 · Cited by 4 · see citing works
Rajapaksha, D., Siriwardana, C., Ruparathna, R., Maqsood, T., Setunge, S., Rajapakse, L., & Silva, S. D. (2024) · Buildings · doi:10.3390/buildings14061864 · Cited by 4 · see citing works
Zheng, J., Pathak, J. L., Li, S., Li, A., Fang, J., Li, Z., Bi, Z., Xiao, Y., & Zhang, Q. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104911 · Cited by 2 · see citing works
Una revisión de las implementaciones de sistemas para la identificación de tendencias de la diabetes
Loja, R. I. B., & Ygnacio, M. A. C. (2022) · Interfases · doi:10.26439/interfases2022.n016.5957 · Cited by 2 · see citing works
Automation of legal precedents retrieval: findings from a rapid literature reviewPreprint
Mentzingen, H., Bação, F., & António, N. (2022) · Research Square · doi:10.21203/rs.3.rs-2292464/v1 · Cited by 2 · see citing works
Sághy, E., Elsharkawy, M., Moriarty, F., Kovács, S., Wittmann, I., & Zemplenyi, A. T. (2025) · Frontiers in Digital Health · doi:10.3389/fdgth.2025.1495879 · Cited by 1 · see citing works
Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics
Xu, Z., Zhuang, X., & Ma, S. (2025) · Education Sciences · doi:10.3390/educsci15111488
Barragán, A. M., Bonett, S. E. O., Rodríguez-Grande, E.-I., Orjuela-Cañón, Á. D., Perdómo, O., & Sánchez, G. (2026) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70098
- Authors
- Zimmerman, J., Soler, R. E., Lavinder, J., Murphy, S., Atkins, C., Hulbert, L., Lusk, R., & Ng, B. P.
- Affiliations
- Deloitte (United States) (US)
- Centers for Disease Control and Prevention (US)
- University of Central Florida (US)
- Corresponding author
- Zimmerman, John
- Venue
- Systematic Reviews
- Published
- 2021-04-02
- Volume
- 10
- Issue
- 1
- Pages
- 97-97
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC8017891
- OpenAlex
- W3113526370
- Cited by
- 24
- References
- 30
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 2 · 2023: 4 · 2022: 9 · 2021: 1
- FWCI
- 2.32
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- text analysis, prescreening, transfer learning, human-in-the-loop learning, diabetes mellitus
- MeSH terms
- Machine Learning, Diabetes Mellitus, Humans, Mass Screening, Research Design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
