Machine learning techniques for the automation of literature reviews and systematic reviews in EFSA
Open accessThe DOI matches its Crossref record; 91 of the 91 references it cites were found in a registry or an index.
Abstract
This Report presents the results from EFSA project RC/EFSA/AMU/2016/01 related to the implementation of machine learning techniques for literature reviews and systematic reviews in EFSA. An overview of the different steps of a systematic review is provided, along with possible ways for automation. Although it was found that most steps could benefit from automation, it was also observed that some steps require more sophisticated methods than those encompassed within the machine learning framework. Availability of data and methodology allowed for the development of an automatic screening tool based on several machine learning techniques. The developed shiny R application can be used for the screening of abstracts and full texts. Properties of machine learning techniques are discussed in this Report together with their most important advantages and disadvantages. The latter discussion includes both general properties, as well as context-specific properties based on their performance in three case studies. Although creating a universal automatic data extraction tool was considered to be infeasible in this stage, this step of the systematic review was addressed to allow the reviewer to scan the uploaded pdf files for certain words or string of words. Based on observations from the performed case studies, recommendations were made regarding which methods are preferred in specific situations. More explicitly, a discussion is made about the performance of the classifiers with respect to the magnitude of the pool of papers to be screened as well as to the amount of imbalance, referring to the proportion of relevant and irrelevant papers. Finally, it was concluded that the results presented in this report provide proof that the developed shiny application could be efficiently used in combination with other software such as DistillerSR.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, developing automatic screening tools based on combined ML techniques seems feasible; plenty of research and developments have been done in this field in recent years [26, 39, 41, 46, 52].”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [41]
In the citing paper
“Use of natural language processing (NLP) and machine learning (ML) algorithms to reduce time and workload in the SR process is becoming increasingly popular [29, 40, 41].”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [41]
In the citing paper
“These algorithms can be trained to develop the ability to categorise, using so called “training data sets” of records screened by human reviewers.”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [41]
In the citing paper
“their performance varies greatly depending on this previous step of training [41, 71, 72]”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [41]
In the citing paper
“the current ML algorithms require evaluation and training using a pre-selected and labelled set of records”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [41]
Citations
Cites
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421 · cited 1×
“Support Vector Machines were developed by Cortes and Vapnik (1995) with the aim of binary classification.”
Cortes, C., & Vapnik, V. (1995) · Machine-mediated learning · doi:10.1023/a:1022627411411 · Cited by 48,445
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Yet another alternative to the BOW approach is to use Word2Vec features (Mikolov et al., 2013), that represents each word as a vector in a form that reflects how close words occur.”
MethodConditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,223
Finding groups in data: an introduction to cluster analysis
(1991) · Choice Reviews Online · doi:10.5860/choice.28-4558 · Cited by 10,474
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Research works printed but not linked: 3
- Higgins, 2011, Update on the MECIR project: methodological expectations for Cochrane intervention, Cochrane Methodology, 1, 2
- Lin, 2011, Automating the process of critical appraisal and assessing the strength of evidence with information extraction technology, Journal of Evaluation in Clinical Practice, 17, 832, 10.1111/j.1365-2753.2011.01712.x · DOI
- Mitchell, 1997, Machine Learning, 400
Cited by
A Systematic Review on Supervised and Unsupervised Machine Learning Algorithms for Data Science
Alloghani, M., Al-Jumeily, D., Mustafina, J., Hussain, A., & Aljaaf, A. J. (2020) · Unsupervised and Semi-Supervised Learning · doi:10.1007/978-3-030-22475-2_1 · Cited by 702
Ofori-Boateng, R., Aceves-Martins, M., Wiratunga, N., & Moreno-Garcia, C. F. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10844-w · Cited by 86
Cierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01805-4 · Cited by 85
“Use of natural language processing (NLP) and machine learning (ML) algorithms to reduce time and workload in the SR process is becoming increasingly popular [29, 40, 41].”
BackgroundMethodInfluentialOn the use of computer‐assistance to facilitate systematic mapping
Haddaway, N., Callaghan, M., Collins, A., Lamb, W. F., Minx, J. C., Thomas, J., & John, D. (2020) · Campbell Systematic Reviews · doi:10.1002/cl2.1129 · Cited by 55
“Data extraction — Topic modelling and machine learning classifiers can be used to interrogate the body of evidence in order to gain a relatively crude understanding of the content and focus of relevant research (Jaspers et al., 2018; Stansfield et al., 2013; Westgate, 2019).”
Future perspectives for advancing regulatory science of nanotechnology-enabled health products
Halamoda‐Kenzaoui, B., Geertsma, R. E., Pouw, J., Prina‐Mello, A., Carrer, M., Roesslein, M., Sips, A. J. A. M., Weltring, K. M., Spring, K., & Bremer‐Hoffmann, S. (2022) · Drug Delivery and Translational Research · doi:10.1007/s13346-022-01165-y · Cited by 43
“Such tools are already being developed, but more effort is needed to make them more transparent, build trust and train potential users [16, 17].”
BackgroundDecision support tools for SLR search string construction
Marcos-Pablos, S., & García‐Peñalvo, F. J. (2018) · Ninth International Conference on Technological Ecosystems for Enhancing Multiculturality (TEEM'21) · doi:10.1145/3284179.3284292 · Cited by 48
Progress in data interoperability to support computational toxicology and chemical safety evaluation
Watford, S., Edwards, S. W., Angrish, M. M., Judson, R., & Friedman, K. P. (2019) · Toxicology and Applied Pharmacology · doi:10.1016/j.taap.2019.114707 · Cited by 44
Automatic classification of literature in systematic reviews on food safety using machine learning
van den Bulk, L. M., Bouzembrak, Y., Gavai, A., Liu, N., van den Heuvel, L. J., & Marvin, H. J. P. (2022) · Current Research in Food Science · doi:10.1016/j.crfs.2021.12.010 · Cited by 38
“In 2018, Jaspers et al. (2018) presented a report on the possible applications of machine learning in systematic reviews within EFSA.”
BackgroundApplying machine classifiers to update searches: Analysis from two case studies
Stansfield, C., Stokes, G., & Thomas, J. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1537 · Cited by 34
“3,8 There are a variety of different methods for machine classification 3,9 for example, support-vector machine (SVM), Naïve Bayes, neural networks and ensemble methods.”
Arjmandi, M., Fattahi, M., Motevasel, M., & Rezaveisi, H. (2023) · Scientific Reports · doi:10.1038/s41598-023-47174-w · Cited by 33
AI meets academia: transforming systematic literature reviews
Tomczyk, P., Brüggemann, P., & Vrontis, D. (2024) · EuroMed Journal of Business · doi:10.1108/emjb-03-2024-0055 · Cited by 55
Codepori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology
Rasheed, Z., Malik, S., Kemell, K.-K., Muhammad, W., Saari, M., Systä, K., & Abrahamsson, P. (2024) · doi:10.2139/ssrn.4979510 · Cited by 27
Iterative guided machine learning-assisted systematic literature reviews: A diabetes case study
Zimmerman, J., Soler, R. E., Lavinder, J., Murphy, S., Atkins, C., Hulbert, L., Lusk, R., & Ng, B. P. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01640-6 · Cited by 24
Principle-based recommendations for big data and machine learning in food safety: the P-SAFETY model
Sapienza, S., & Vedder, A. (2021) · AI & Society · doi:10.1007/s00146-021-01282-1 · Cited by 32
“Several technical reports showing the potential of MLTs in food safety risk assessment have been released by EFSA (Cavalli et al. 2019; IZSTO et al. 2017; Jaspers et al. 2018; Naydenova et al. 2019).”
BackgroundHalamoda‐Kenzaoui, B., Rolland, E., Piovesan, J., Puertas Gallardo, A., & Bremer‐Hoffmann, S. (2022) · Journal of Applied Toxicology · doi:10.1002/jat.4204 · Cited by 21
Elmousalami, H. H., Hui, F. K. P., & Aye, L. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106346 · Cited by 26
The role of mammals in Avian Influenza: a review
Occhibove, F., Knauf, S., Sauter‐Louis, C., Christoph, S., Valerie, A. S., Alina, A., Barron, S. M., Hannes, B., Bröjer, C., Elena, B., et al. (2024) · EFSA Supporting Publications · doi:10.2903/sp.efsa.2024.en-8692 · Cited by 32
Artificial intelligence and automation of systematic reviews in women's health
Amezcua‐Prieto, C., Fernández-Luna, J. M., Huete, J. F., Bueno‐Cavanillas, A., & Khan, K. S. (2020) · Current Opinion in Obstetrics & Gynecology · doi:10.1097/gco.0000000000000643 · Cited by 17
Peron, M., Arena, S., Paltrinieri, N., Sgarbossa, F., & Boustras, G. (2022) · Risk Analysis · doi:10.1111/risa.14010 · Cited by 20
“Recently, several authors presented different methodologies (Jaspers et al., 2018; Peron et al., 2020; Weißer et al., 2020), optimization techniques (Adinugroho et al., 2017; Peron et al., 2020), and frameworks (Peron et al., 2022; Simonetto et al., 2022; Tauchert et al., 2020) aiming at…”
BackgroundOsseointegration Pharmacology: A Systematic Mapping Using Artificial Intelligence
Mahri, M., Shen, N., Berrizbeitia, F., Rodan, R., Daer, A., Faigan, M., Taqi, D., Wu, K. Y., Ahmadi, M., Ducret, M., et al. (2020) · Acta Biomaterialia · doi:10.1016/j.actbio.2020.11.011 · Cited by 18
Automated Literature Review Using NLP Techniques and LLM-Based Retrieval-Augmented Generation
Ali, N. F., Mohtasim, M. M., Mosharrof, S., & Krishna, T. G. (2024) · International Conference on Innovations in Science, Engineering and Technology (ICISET) · doi:10.1109/iciset62123.2024.10939517 · Cited by 39
“Jaspers et al. [9] focused on the use of machine learning techniques for automation of literature reviews and systematic reviews.”
BERT for Complex Systematic Review Screening to Support the Future of Medical Research
Hasny, M., Vasile, A.-P., Gianni, M., Bannach‐Brown, A., Nasser, M., Mackay, M., donovan, D., Šorli, J., Domocos, I., Dulloo, M., et al. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-34344-5_21 · Cited by 12
Data collection on antibiotics for control of plant pathogenic bacteria
Verhaegen, M., Mahillon, J., Caulier, S., Mingeot‐Leclercq, M., & Bragard, C. (2024) · EFSA Supporting Publications · doi:10.2903/sp.efsa.2024.en-8522 · Cited by 11
Еl-Aal, A. K. A., GabAllah, H. M., Megahed, H. A., Selim, M. K., Hegab, M. A. E.-R., Fadl, M. E., Rebouh, N. Y., & El-Bagoury, H. (2024) · Sustainability · doi:10.3390/su162411139 · Cited by 10
Shah-Mohammadi, F., & Finkelstein, J. (2024) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc60891.2024.10427829 · Cited by 6
Towards Automated Meta-Analysis of Clinical Trials: An Overview
Christopoulou, S. C. (2023) · BioMedInformatics · doi:10.3390/biomedinformatics3010009 · Cited by 5
A Survey on Machine learning and IoT
Hardini, I. R. (2019) · ITEJ (Information Technology Engineering Journals) · doi:10.24235/itej.v4i2.51 · Cited by 5
“Karena kemampuannya untuk menyelesaikan berbagai macam masalah, maka machine learning dapat diterapkan pada berbagai bidang aplikasi kehidupan untuk membantu dalam hal prediksi dan membantu proses pengambilan keputusan [19].”
BackgroundInfluentialFood Standards Agency (FSA), A. A. P. H. A. (., Lorenzoni, A., Tsoulli, C., McCarthy, C., & Adkin, A. (2020) · EFSA Journal · doi:10.2903/j.efsa.2020.e181108 · Cited by 4
“Systematic reviews are a type of literature review used to identify, evaluate and synthesize the findings using systematic and reproducible methods (Grant and Booth, 2009, Jaspers et al., 2018).”
MethodConvolutional Neural Network for Core Sections Identification in Scientific Research Publications
Muhammad, B. A., Iqbal, R., James, A. E., & Nkantah, D. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-33607-3_29 · Cited by 5
Schaut, W., Shrivastav, A., Ramakrishnan, S., & Bowden, R. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1072767 · Cited by 2
A Systematic Survey of Business Intelligence Literature Using Machine Learning Techniques
Vavalis, M. (2022) · Journal of Economics & Management Research · doi:10.47363/jesmr/2022(3)150 · Cited by 1
Systematic reviews and evidence-based methods in toxicology
Wikoff, D. S., & Fitch, S. (2023) · Elsevier eBooks · doi:10.1016/b978-0-12-824315-2.01138-6
Sustainable Governance—Application of Artificial Intelligence
Gireesan, K. (2024) · Springer eBooks · doi:10.1007/978-981-99-7735-2_9
Aksoy, A., & Haki, Ö. (2025) · Scientific Reports · doi:10.1038/s41598-025-99488-6
“1 73 : In Eq.”
Jiménez, R. C., Lee, T., Rosillo, N., Córdova, R., Cree, I. A., González, Á., & Indave Ruiz, B. I. (2023) · Figshare · doi:10.6084/m9.figshare.22609363
Metadata
- Authors
- Jaspers, S., De Troyer, E., & Aerts, M.
- Affiliations
- Hasselt University (BE)
- UHASSELT Belgium
- Venue
- EFSA Supporting Publications
- Publisher
- Wiley
- Published
- 2018-06
- Volume
- 15
- Issue
- 6
- ISSN
- 2397-8325 (ISSN-L)
- OpenAlex
- W2808716231
- Semantic Scholar
- ececc1eb0606eb31074cadf9def86169818091e7
- Cited by
- 36
- References
- 89
- Citations by year
- 2025: 2 · 2024: 9 · 2023: 6 · 2022: 4 · 2021: 6 · 2020: 4 · 2019: 4 · 2018: 1
- FWCI
- 0.75
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- other-oa
- Locations
- EFSA Supporting Publications (publishedVersion, other-oa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Keywords
- machine learning, systematic reviews, automation, automated data extraction, classification performance, class imbalance, EFSA
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.