The DOI matches its Crossref record; 90 of the 90 references it cites were found in a registry or an index.
Machine learning techniques for the automation of literature reviews and systematic reviews in EFSA
Open accessAbstract
This Report presents the results from EFSA project RC/EFSA/AMU/2016/01 related to the implementation of machine learning techniques for literature reviews and systematic reviews in EFSA. An overview of the different steps of a systematic review is provided, along with possible ways for automation. Although it was found that most steps could benefit from automation, it was also observed that some steps require more sophisticated methods than those encompassed within the machine learning framework. Availability of data and methodology allowed for the development of an automatic screening tool based on several machine learning techniques. The developed shiny R application can be used for the screening of abstracts and full texts. Properties of machine learning techniques are discussed in this Report together with their most important advantages and disadvantages. The latter discussion includes both general properties, as well as context-specific properties based on their performance in three case studies. Although creating a universal automatic data extraction tool was considered to be infeasible in this stage, this step of the systematic review was addressed to allow the reviewer to scan the uploaded pdf files for certain words or string of words. Based on observations from the performed case studies, recommendations were made regarding which methods are preferred in specific situations. More explicitly, a discussion is made about the performance of the classifiers with respect to the magnitude of the pool of papers to be screened as well as to the amount of imbalance, referring to the proportion of relevant and irrelevant papers. Finally, it was concluded that the results presented in this report provide proof that the developed shiny application could be efficiently used in combination with other software such as DistillerSR.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, developing automatic screening tools based on combined ML techniques seems feasible; plenty of research and developments have been done in this field in recent years [26, 39, 41, 46, 52].”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [41]
In the citing paper
“Use of natural language processing (NLP) and machine learning (ML) algorithms to reduce time and workload in the SR process is becoming increasingly popular [29, 40, 41].”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [41]
In the citing paper
“These algorithms can be trained to develop the ability to categorise, using so called “training data sets” of records screened by human reviewers.”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [41]
In the citing paper
“their performance varies greatly depending on this previous step of training [41, 71, 72]”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [41]
In the citing paper
“the current ML algorithms require evaluation and training using a pre-selected and labelled set of records”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [41]
Citations
Cites
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421 · cited 1×
“Support Vector Machines were developed by Cortes and Vapnik (1995) with the aim of binary classification.”
Cortes, C., & Vapnik, V. (1995) · Machine-mediated learning · doi:10.1023/a:1022627411411 · Cited by 48,445
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Yet another alternative to the BOW approach is to use Word2Vec features (Mikolov et al., 2013), that represents each word as a vector in a form that reflects how close words occur.”
MethodConditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,223
Finding groups in data: an introduction to cluster analysis
(1991) · Choice Reviews Online · doi:10.5860/choice.28-4558 · Cited by 10,474
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Research works printed but not linked: 3
- Higgins, 2011, Update on the MECIR project: methodological expectations for Cochrane intervention, Cochrane Methodology, 1, 2
- Lin, 2011, Automating the process of critical appraisal and assessing the strength of evidence with information extraction technology, Journal of Evaluation in Clinical Practice, 17, 832, 10.1111/j.1365-2753.2011.01712.x · DOI
- Mitchell, 1997, Machine Learning, 400
Cited by
A Systematic Review on Supervised and Unsupervised Machine Learning Algorithms for Data Science
Alloghani, M., Al-Jumeily, D., Mustafina, J., Hussain, A., & Aljaaf, A. J. (2020) · Unsupervised and Semi-Supervised Learning · doi:10.1007/978-3-030-22475-2_1 · Cited by 702
Ofori-Boateng, R., Aceves‐Martins, M., Wiratunga, N., & Moreno‐García, C. F. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10844-w · Cited by 146
Cierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01805-4 · Cited by 85
“Use of natural language processing (NLP) and machine learning (ML) algorithms to reduce time and workload in the SR process is becoming increasingly popular [29, 40, 41].”
BackgroundMethodInfluentialOn the use of computer‐assistance to facilitate systematic mapping
Haddaway, N., Callaghan, M., Collins, A., Lamb, W. F., Minx, J. C., Thomas, J., & John, D. (2020) · Campbell Systematic Reviews · doi:10.1002/cl2.1129 · Cited by 55
“Data extraction — Topic modelling and machine learning classifiers can be used to interrogate the body of evidence in order to gain a relatively crude understanding of the content and focus of relevant research (Jaspers et al., 2018; Stansfield et al., 2013; Westgate, 2019).”
Future perspectives for advancing regulatory science of nanotechnology-enabled health products
Halamoda‐Kenzaoui, B., Geertsma, R. E., Pouw, J., Prina‐Mello, A., Carrer, M., Roesslein, M., Sips, A. J. A. M., Weltring, K. M., Spring, K., & Bremer‐Hoffmann, S. (2022) · Drug Delivery and Translational Research · doi:10.1007/s13346-022-01165-y · Cited by 43
“Such tools are already being developed, but more effort is needed to make them more transparent, build trust and train potential users [16, 17].”
Background
Metadata
- Authors
- Jaspers, S., De Troyer, E., & Aerts, M.
- Affiliations
- Hasselt University (BE)
- UHASSELT Belgium
- Venue
- EFSA Supporting Publications
- Publisher
- Wiley
- Published
- 2018-06
- Volume
- 15
- Issue
- 6
- ISSN
- 2397-8325 (ISSN-L)
- OpenAlex
- W2808716231
- Semantic Scholar
- ececc1eb0606eb31074cadf9def86169818091e7
- Cited by
- 36
- References
- 89
- Citations by year
- 2025: 2 · 2024: 9 · 2023: 6 · 2022: 4 · 2021: 6 · 2020: 4 · 2019: 4 · 2018: 1
- FWCI
- 0.75
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- other-oa
- Locations
- EFSA Supporting Publications (publishedVersion, other-oa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Keywords
- machine learning, systematic reviews, automation, automated data extraction, classification performance, class imbalance, EFSA
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Machine learning techniques for the automation of literature reviews and systematic reviews in EFSA. https://app.cochat.ai/papers/10.2903/sp.efsa.2018.en-1427
- The paper
- Jaspers, S., De Troyer, E., & Aerts, M. (2018). Machine learning techniques for the automation of literature reviews and systematic reviews in EFSA. EFSA Supporting Publications. https://doi.org/10.2903/sp.efsa.2018.en-1427
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.