Machine learning techniques for the automation of literature reviews and systematic reviews in EFSA
Open accessThe DOI matches its Crossref record; 90 of the 90 references it cites were found in a registry or an index.
Abstract
This Report presents the results from EFSA project RC/EFSA/AMU/2016/01 related to the implementation of machine learning techniques for literature reviews and systematic reviews in EFSA. An overview of the different steps of a systematic review is provided, along with possible ways for automation. Although it was found that most steps could benefit from automation, it was also observed that some steps require more sophisticated methods than those encompassed within the machine learning framework. Availability of data and methodology allowed for the development of an automatic screening tool based on several machine learning techniques. The developed shiny R application can be used for the screening of abstracts and full texts. Properties of machine learning techniques are discussed in this Report together with their most important advantages and disadvantages. The latter discussion includes both general properties, as well as context-specific properties based on their performance in three case studies. Although creating a universal automatic data extraction tool was considered to be infeasible in this stage, this step of the systematic review was addressed to allow the reviewer to scan the uploaded pdf files for certain words or string of words. Based on observations from the performed case studies, recommendations were made regarding which methods are preferred in specific situations. More explicitly, a discussion is made about the performance of the classifiers with respect to the magnitude of the pool of papers to be screened as well as to the amount of imbalance, referring to the proportion of relevant and irrelevant papers. Finally, it was concluded that the results presented in this report provide proof that the developed shiny application could be efficiently used in combination with other software such as DistillerSR.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, developing automatic screening tools based on combined ML techniques seems feasible; plenty of research and developments have been done in this field in recent years [26, 39, 41, 46, 52].”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [41]
In the citing paper
“Use of natural language processing (NLP) and machine learning (ML) algorithms to reduce time and workload in the SR process is becoming increasingly popular [29, 40, 41].”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [41]
In the citing paper
“These algorithms can be trained to develop the ability to categorise, using so called “training data sets” of records screened by human reviewers.”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [41]
In the citing paper
“their performance varies greatly depending on this previous step of training [41, 71, 72]”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [41]
In the citing paper
“the current ML algorithms require evaluation and training using a pre-selected and labelled set of records”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [41]
Citations
Cites
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421 · cited 1×
“Support Vector Machines were developed by Cortes and Vapnik (1995) with the aim of binary classification.”
Cortes, C., & Vapnik, V. (1995) · Machine-mediated learning · doi:10.1023/a:1022627411411 · Cited by 48,445
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 1×
“Yet another alternative to the BOW approach is to use Word2Vec features (Mikolov et al., 2013), that represents each word as a vector in a form that reflects how close words occur.”
MethodConditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,223
Finding groups in data: an introduction to cluster analysis
(1991) · Choice Reviews Online · doi:10.5860/choice.28-4558 · Cited by 10,474
Domain-Adversarial Training of Neural Networks
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., & Lempitsky, V. (2017) · Advances in Computer Vision and Pattern Recognition · doi:10.1007/978-3-319-58347-1_10 · Cited by 8,318
Machine learning in automated text categorization
Sebastiani, F. (2002) · ACM Computing Surveys · doi:10.1145/505282.505283 · Cited by 7,926 · cited 1×
“Once the features have been identified, it can be advisable to further reduce the dimension of the input space by performing term space reduction (Sebastiani, 2002).”
BackgroundFinding Groups in Data: An Introduction to Cluster Analysis
Kaufman, L., & Rousseeuw, P. J. (1990) · Wiley Series in Probability and Statistics · doi:10.1002/9780470316801 · Cited by 7,344
Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.1. The Cochrane Collaboration
Higgins, J. P. T., Green, S. R., & Higgins, J. P. T. (2008) · Cited by 7,171
Primary, Secondary, and Meta-Analysis of Research
GLASS, G. V. (1976) · Educational Researcher · doi:10.3102/0013189x005010003 · Cited by 4,890 · cited 1×
“As such, the term meta-analysis is used to refer to an analysis of analyses (Glass, 1976).”
MethodA Comparative Study on Feature Selection in Text Categorization
Yang, Y., & Pedersen, J. (1997) · Cited by 4,790
De-duplication of database search results for systematic reviews in EndNote
Bramer, W. M., Giustini, D., de Jonge, G. B., Holland, L., & Bekhuis, T. (2016) · Journal of the Medical Library Association : JMLA · doi:10.3163/1536-5050.104.3.014 · Cited by 2,103
(2000) · Queen Mary Research Online (Queen Mary University of London) · Cited by 1,795
Physical activity and exercise for chronic pain in adults: an overview of Cochrane Reviews
Geneen, L. J., Moore, R. A., Clarke, C., Martin, D., Colvin, L. A., & Smith, B. H. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011279.pub3 · Cited by 1,222
Tackling the poor assumptions of naive bayes text classifiers
Rennie, J. D. M., Shih, L., Teevan, J. B., & Karger, D. R. (2003) · Cited by 977
Not So Naive Bayes: Aggregating One-Dependence Estimators
Webb, G. I., Boughton, J. R., & Wang, Z. (2005) · Machine Learning · doi:10.1007/s10994-005-4258-6 · Cited by 708 · cited 1×
“Another way to improve the NB classifier is the Averaged One-Dependence Estimators (AODE) approach, developed by Webb et al. (2005). In AODE, an aggregate of one-dependence classifiers is learned and the final prediction is made by taking the average of the predictions of all these qualified one-dependence classifiers.”
MethodAuthority, E. G. F. T. C. O. S. R. E. F. S. (2010) · EFSA Journal · doi:10.2903/j.efsa.2010.1637 · Cited by 693
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 3×
“Of course, there was a large amount of overlap between the new articles and those retrieved in O’Mara-Eves et al. (2015) and the formerly mentioned google scholar search.”
MethodResultInfluentialNordhausen, K. (2009) · International Statistical Review · doi:10.1111/j.1751-5823.2009.00095_18.x · Cited by 637
ROSE: a Package for Binary Imbalanced Learning
Lunardon, N., Menardi, G., & Torelli, N. (2014) · The R Journal · doi:10.32614/rj-2014-008 · Cited by 530 · cited 1×
“More specifically, SMOTE and ROSE sampling (Lunardon et al., 2014) can be selected to create a more balanced training dataset.”
MethodBANNER: AN EXECUTABLE SURVEY OF ADVANCES IN BIOMEDICAL NAMED ENTITY RECOGNITION
LEAMAN, R., & GONZALEZ, G. (2007) · Biocomputing 2008 · doi:10.1142/9789812776136_0062 · Cited by 479
Systematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472 · cited 8×
“At the same time, it poses challenges due to the exponential growth in evidence production and the consequent high demand in terms of time and resources (Tsafnat et al., 2014).”
BackgroundMethodInfluentialFeature selection, perception learning, and a usability case study for text categorization
Ng, H. T., Goh, W. B., & Low, K. L. (1997) · Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '97 · doi:10.1145/258525.258537 · Cited by 450 · cited 1×
“, 2001), the NGL coefficient (Ng et al., 1997) and the GSS coefficient (Galavotti et al.”
MethodReducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 1×
“Another example is the voting perceptron-based automated citation classification system discussed in Cohen et al. (2006). They employed varying learning weights to penalize for misclassification of the relevant class; A drawback is that there is no universal way for choosing the most optimal weights across all reviews…”
MethodO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Semi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351
ParsCit: an Open-source CRF Reference String Parsing Package
Councill, I. G., Giles, C. L., & Kan, M. (2008) · Language Resources and Evaluation · Cited by 326
Automating data extraction in systematic reviews: a systematic review
Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0066-7 · Cited by 287 · cited 1×
“The fourth article, being Jonnalagadda et al. (2015) specifically focused on the data extraction step of the review.”
BackgroundApplications of text mining within systematic reviews
Thomas, J., McNaught, J., & Ananiadou, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.27 · Cited by 212
Experiments on the Use of Feature Selection and Negative Evidence in Automated Text Categorization
Galavotti, L., Sebastiani, F., & Simi, M. (2000) · Lecture Notes in Computer Science · doi:10.1007/3-540-45268-0_6 · Cited by 210
Rathbone, J., Carter, M., Hoffmann, T., & Glasziou, P. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-6 · Cited by 206
Text summarization in the biomedical domain: A systematic review of recent research
Mishra, R., Bian, J., Fiszman, M., Weir, C. R., Jonnalagadda, S., Mostafa, J., & Del Fiol, G. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.06.009 · Cited by 200 · cited 1×
“, part of speech tagging, word sense disambiguation), grammatical inference, and robust parsing (Mishra et al., 2014).”
BackgroundSWIFT-Review: a text-mining workbench for systematic review
Howard, B. E., Phillips, J., Miller, K., Tandon, A., Mav, D., Shah, M. R., Holmgren, S., Pelch, K. E., Walker, V., Rooney, A. A., et al. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0263-z · Cited by 196 · cited 1×
“For example, SWIFT-review provides a workbench with tools to assist the reviewer in literature prioritisation (Howard et al., 2016).”
MethodExaCT: automatic extraction of clinical trial characteristics from journal publications
Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-56 · Cited by 188 · cited 3×
“(2011) present a conditional random field classifier for automatically calculating summary statistics like the absolute risk reduction and number needed to treat in clinical trials (see also Lafferty et al., 2001). Other examples include Rosario and Hearst (2005), who focused on a probabilistic graphical model for ide…”
BackgroundInfluentialCusack, L., Del Mar, C. B., Chalmers, I., & Hoffmann, T. C. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0213-9 · Cited by 188
Caropreso, M. F., Matwin, S., & Sebastiani, F. (2001) · Cited by 179
Identifying and removing duplicate records from systematic review searches
Kwon, Y., Lemieux, M., McTavish, J., & Wathen, N. (2015) · Journal of the Medical Library Association : JMLA · doi:10.3163/1536-5050.103.4.004 · Cited by 178
Shemilt, I., Simon, A., Hollands, G. J., Marteau, T. M., Ogilvie, D., O'Mara‐Eves, A., Kelly, M. P., & Thomas, J. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1093 · Cited by 170
(ANS), E. P. O. F. A. A. N. S. A. T. F. (2015) · EFSA Journal · doi:10.2903/j.efsa.2015.4246 · Cited by 168
Feature selection, perceptron learning, and a usability case study for text categorization
Ng, H. T., Goh, W. B., & Low, K. L. (1997) · ACM SIGIR Forum · doi:10.1145/278459.258537 · Cited by 155
(2016) · PubMed · Cited by 134
Combining classifiers for robust PICO element detection.
Boudin, F., Nie, J.-Y., Bartlett, J. C., Grad, R., Pluye, P., & Dawes, M. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-29 · Cited by 131
Chanlekha, H., Kawazoe, A., & Collier, N. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-1 · Cited by 109
How to conduct systematic reviews more expeditiously?
Tsertsvadze, A., Chen, Y.-F., Moher, D., Sutcliffe, P., & McCarthy, N. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0147-7 · Cited by 104 · cited 1×
“This approach was followed in Wallace et al. (2010), who developed an active learning algorithm for SVMs, applied to four different feature spaces.”
MethodTopic detection using paragraph vectors to support active learning in systematic reviews
Hashimoto, K., Kontonatsios, G., Miwa, M., & Ananiadou, S. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.06.001 · Cited by 100 · cited 2×
“Hashimoto et al. (2016) introduces a new topic detection method that induces an informative representation of studies with the aim of improving the performance of the underlying MLT. It uses a neural network-based vector space model to capture semantic similarities between documents. First, documents are represented w…”
MethodKhabsa, M., Elmagarmid, A., Ilyas, I., Hammady, H., & Ouzzani, M. (2016) · Machine Learning · doi:10.1007/s10994-015-5535-7 · Cited by 98
Find Duplicates among the PubMed, EMBASE, and Cochrane Library Databases in Systematic Review
Qi, X., Yang, M., Ren, W., Jia, J., Wang, J., Han, G., & Fan, D. (2013) · PLoS ONE · doi:10.1371/journal.pone.0071838 · Cited by 98
Document clustering of scientific texts using citation contexts
Aljaber, B., Stokes, N., Bailey, J., & Pei, J. (2009) · Information Retrieval · doi:10.1007/s10791-009-9108-x · Cited by 96
Weightily Averaged One-Dependence Estimators
Jiang, L., & Zhang, H. (2006) · Lecture notes in computer science · doi:10.1007/978-3-540-36668-3_116 · Cited by 84
Automatic Evidence Retrieval for Systematic Reviews
Choong, M. K., Galgani, F., Dunn, A. G., & Tsafnat, G. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3369 · Cited by 84 · cited 1×
“The only reference to automating this step was found to be Choong et al. (2014), who evaluate an automatic evidence retrieval system based on a modified version of ParsCit (Councill et al.”
MethodOn transductive support vector machines
Wang, J., Shen, X., & Pan, W. (2007) · Contemporary mathematics - American Mathematical Society · doi:10.1090/conm/443/08551 · Cited by 84
Supporting systematic reviews using LDA-based document representations
Mo, Y., Kontonatsios, G., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0117-0 · Cited by 83 · cited 3×
“The authors noted that these results were in line with the findings of Qi et al. (2013) and Rathbone et al. (2015). More specifically, these authors argued that the automatic deduplication option in EndNote is not fully adequate and should be supplemented by a manual search.”
ResultInfluentialMulti-way relation classification
Rosario, B., & Hearst, M. A. (2005) · doi:10.3115/1220575.1220667 · Cited by 81
Hartling, L., Bond, K., Vandermeer, B., Seida, J., Dryden, D. M., & Rowe, B. H. (2011) · PLoS ONE · doi:10.1371/journal.pone.0017242 · Cited by 78 · cited 3×
“It has been shown that different reviewers often identify different levels of risk of bias for the same studies, most often explained by the fact that reviewers might miss key sentences (Hartling et al., 2011; Lensen et al., 2014). Automating certain aspects of assessing the risk of bias can lead to reduce the time re…”
BackgroundBui, D. D. A., Del Fiol, G., Hurdle, J. F., & Jonnalagadda, S. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.10.014 · Cited by 70
Machine learning to assist risk-of-bias assessments in systematic reviews
Millard, L. A., Flach, P. A., & Higgins, J. P. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyv306 · Cited by 65
A new iterative method to reduce workload in systematic review process
Jonnalagadda, S., & Petitti, D. (2013) · International Journal of Computational Biology and Drug Design · doi:10.1504/ijcbdd.2013.052198 · Cited by 61 · cited 2×
“Jonnalagadda et al. (2015) created a systematic review related to automating data extraction in SRs.”
BackgroundTowards Automating the Initial Screening Phase of a Systematic Review
Tanja, B., & Dina, D.-F. (2010) · Studies in Health Technology and Informatics · doi:10.3233/978-1-60750-588-4-146 · Cited by 60 · cited 1×
“Bekhuis and Demner-Fushman (2010) compared the evolutionary support vector machine with the Weightily Averaged One-Dependence Estimators and Naïve Bayes classifier.”
MethodAutomating Risk of Bias Assessment for Clinical Trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2015) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2015.2431314 · Cited by 59
Automatic Summarization of Results from Clinical Trials
Summerscales, R. L., Argamon, S., Bai, S., Hupert, J., & Schwartz, A. (2011) · 2011 IEEE International Conference on Bioinformatics and Biomedicine · doi:10.1109/bibm.2011.72 · Cited by 49 · cited 1×
“In addition to the paper by Tsafnat et al. (2014), four SRs addressing the concept of automation were found.”
BackgroundClinical Information Retrieval using Document and PICO Structure
Boudin, F., Nie, J., & Dawes, M. G. (2010) · North American Chapter of the Association for Computational Linguistics · Cited by 48
Olorisade, B. K., de Quincey, E., Brereton, P., & Andras, P. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2915982 · Cited by 42 · cited 7×
“Following the results from O’Mara-Eves et al. (2015), the three performance measures that are used throughout the most studies are recall, precision and the F measure.”
BackgroundMethodInfluentialLiu, J., Timsina, P., & El-Gayar, O. (2016) · Information Systems Frontiers · doi:10.1007/s10796-016-9724-0 · Cited by 41
Machine Learning Techniques applied in risk assessment related to food safety
IZSTO, UBESP, DSCB, ZETA, Ru, G., Crescio, M. I., Ingravalle, F., Maurella, C., Gregori, D., Lanera, C., et al. (2017) · EFSA Supporting Publications · doi:10.2903/sp.efsa.2017.en-1254 · Cited by 38 · cited 1×
“Especially the work done by Ru et al. (2016) proved to be very useful.”
BackgroundKuper, H., Nicholson, A., & Hemingway, H. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-4 · Cited by 36
Advanced analytics for the automation of medical systematic reviews
Timsina, P., Liu, J., & El-Gayar, O. (2016) · Information Systems Frontiers · doi:10.1007/s10796-015-9589-7 · Cited by 33 · cited 1×
“Following both the EFSA guidance and Tsafnat et al. (2014), all the steps of literature searches and/or systematic reviews were first identified and described.”
MethodInfluential(2010) · Cited by 32
Dalal, S. R., Shekelle, P. G., Hempel, S., Newberry, S. J., Motala, A., & Shetty, K. D. (2013) · Medical Decision Making · doi:10.1177/0272989x12457243 · Cited by 32 · cited 2×
“The Gradient Boosting Machine (GBM) applied by Dalal et al. (2012) is a specific implementation of boosting and consists of a general, automated, data-adaptive modelling algorithm that is able to estimate the nonlinear relationship between a variable of interest (in this case whether an article is relevant or not) and…”
BackgroundMethodFigure mining for biomedical research
Rodriguez-Esteban, R., & Iossifov, I. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp318 · Cited by 32 · cited 1×
“Summerscales et al. (2011) present a conditional random field classifier for automatically calculating summary statistics like the absolute risk reduction and number needed to treat in clinical trials (see also Lafferty et al.”
Challenges and opportunities in applied machine learning
Brodley, C. E., Rebbapragada, U., Small, K., & Wallace, B. C. (2012) · AI Magazine · doi:10.1609/aimag.v33i1.2367 · Cited by 32
Improving the utility of MeSH® terms using the TopicalMeSH representation
Yu, Z., Bernstam, E. V., Cohen, T., Wallace, B., & Johnson, T. R. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.03.013 · Cited by 25
‘Clustering’ documents automatically to support scoping reviews of research: a case study
Stansfield, C., Thomas, J., & Kavanagh, J. (2013) · Research Synthesis Methods · doi:10.1002/jrsm.1082 · Cited by 20 · cited 2×
“Some more advanced methods, which incorporate also tables and figures, are presented by Thomas et al. (2011) and by Rodriguez- Esteban and Iossifov (2009). In step 10 of the SR process, the researcher should follow the citations of the selected, relevant trials to find some additional sources of information that can b…”
MethodUsing MEDLINE Elemental Similarity to Assist in the Article Screening Process for Systematic Reviews
Ji, X., & Yen, P. (2015) · JMIR Medical Informatics · doi:10.2196/medinform.3982 · Cited by 20 · cited 2×
“In addition to the conclusions in Jonnalagadda et al. (2015), an additional search was performed using the query “automatic data extraction in systematic reviews” in the Google search engine.”
MethodMaximum margin semi-supervised learning with irrelevant data
Yang, H., Huang, K., King, I., & Lyu, M. R.-T. (2015) · Neural Networks · doi:10.1016/j.neunet.2015.06.004 · Cited by 16
Shallow semantic parsing of randomized controlled trial reports.
Paek, H., Kogan, Y., Thomas, P. G., Codish, S., & Krauthammer, M. O. (2006) · Europe PMC (PubMed Central) · Cited by 15
Kate, K., Chaudhari, S., Prapanca, A., & Kalagnanam, J. (2014) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2623330.2623369 · Cited by 15 · cited 10×
“Semi-supervised learning is also used in the text mining system FoodSIS, which was developed to improve the state of food safety in Singapore (Kate et al., 2014).”
BackgroundMethodInfluentialConsoli, S., & Stilianakis, N. I. (2015) · International Transactions in Operational Research · doi:10.1111/itor.12240 · Cited by 12 · cited 1×
“For example, Dalal et al. (2012) combined the output from their gradient boosting machine and GLMnet approaches into a single classifier, indicating that an article was relevant in case either one of the two classifiers showed a high probability of being relevant.”
Systematic literature review (SLR) automation: a systematic literature review
Hamad, Z., & Salim, N. B. (2014) · Journal of Theoretical and Applied Information Technology · Cited by 9
Lin, J., Chang, C., Lin, M., Ebell, M. H., & Chiang, J.-H. (2011) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2011.01712.x · Cited by 8
Using rule-based classifiers in systematic reviews
Sellak, H., Ouhbi, B., & Frikh, B. (2015) · International Conference on Information Integration and Web-based Applications & Services (iiWAS) · doi:10.1145/2837185.2837279 · Cited by 8
Basu, T., Kumar, S., Kalyan, A., Jayaswal, P., Goyal, P., Pettifer, S., & Jonnalagadda, S. R. (2016) · arXiv · doi:10.48550/arxiv.1606.06424 · Cited by 7 · cited 7×
“On the other hand, the study by Basu et al. (2016), mentioned above, provides a more general approach. Their system is developed in two stages. In the start, it uses information that is contained in existing SRs to identify sentences in the included references that contain specific data elements using a modified Jacca…”
BackgroundMethodInfluentialKouznetsov, A., & Japkowicz, N. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13059-5_33 · Cited by 6
Finding patient-oriented evidence in PubMed abstracts
Robinson, D. A. (2012) · Cited by 5
A large-scale study of SVM-based methods for abstract screening in systematic reviewsPreprint
Saha, T. K., Ouzzani, M., Hammady, H. M., et al. (2017) · arXiv (Cornell University) · doi:10.6084/m9.figshare.4522916.v1 · Cited by 3
Risk of bias:are judgements consistent between reviews?
Lensen, S., Farquhar, C., & Jordan-Cole, V. (2014) · ResearchSpace (University of Auckland) · Cited by 3
A large-scale study of SVM-based methods for abstract screening in systematic reviewsPreprint
Saha, T. K., Ouzzani, M., Hammady, H. M., & Elmagarmid, A. K. (2017) · arXiv (Cornell University) · doi:10.6084/m9.figshare.4522916 · Cited by 3
Research works printed but not linked: 3
- Higgins, 2011, Update on the MECIR project: methodological expectations for Cochrane intervention, Cochrane Methodology, 1, 2
- Lin, 2011, Automating the process of critical appraisal and assessing the strength of evidence with information extraction technology, Journal of Evaluation in Clinical Practice, 17, 832, 10.1111/j.1365-2753.2011.01712.x · DOI
- Mitchell, 1997, Machine Learning, 400
Cited by
A Systematic Review on Supervised and Unsupervised Machine Learning Algorithms for Data Science
Alloghani, M., Al-Jumeily, D., Mustafina, J., Hussain, A., & Aljaaf, A. J. (2020) · Unsupervised and Semi-Supervised Learning · doi:10.1007/978-3-030-22475-2_1 · Cited by 702
Ofori-Boateng, R., Aceves-Martins, M., Wiratunga, N., & Moreno-Garcia, C. F. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10844-w · Cited by 86
Cierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01805-4 · Cited by 85
“Use of natural language processing (NLP) and machine learning (ML) algorithms to reduce time and workload in the SR process is becoming increasingly popular [29, 40, 41].”
BackgroundMethodInfluentialOn the use of computer‐assistance to facilitate systematic mapping
Haddaway, N., Callaghan, M., Collins, A., Lamb, W. F., Minx, J. C., Thomas, J., & John, D. (2020) · Campbell Systematic Reviews · doi:10.1002/cl2.1129 · Cited by 55
“Data extraction — Topic modelling and machine learning classifiers can be used to interrogate the body of evidence in order to gain a relatively crude understanding of the content and focus of relevant research (Jaspers et al., 2018; Stansfield et al., 2013; Westgate, 2019).”
Future perspectives for advancing regulatory science of nanotechnology-enabled health products
Halamoda‐Kenzaoui, B., Geertsma, R. E., Pouw, J., Prina‐Mello, A., Carrer, M., Roesslein, M., Sips, A. J. A. M., Weltring, K. M., Spring, K., & Bremer‐Hoffmann, S. (2022) · Drug Delivery and Translational Research · doi:10.1007/s13346-022-01165-y · Cited by 43
“Such tools are already being developed, but more effort is needed to make them more transparent, build trust and train potential users [16, 17].”
Background
Metadata
- Authors
- Jaspers, S., De Troyer, E., & Aerts, M.
- Affiliations
- Hasselt University (BE)
- UHASSELT Belgium
- Venue
- EFSA Supporting Publications
- Publisher
- Wiley
- Published
- 2018-06
- Volume
- 15
- Issue
- 6
- ISSN
- 2397-8325 (ISSN-L)
- OpenAlex
- W2808716231
- Semantic Scholar
- ececc1eb0606eb31074cadf9def86169818091e7
- Cited by
- 36
- References
- 89
- Citations by year
- 2025: 2 · 2024: 9 · 2023: 6 · 2022: 4 · 2021: 6 · 2020: 4 · 2019: 4 · 2018: 1
- FWCI
- 0.75
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- other-oa
- Locations
- EFSA Supporting Publications (publishedVersion, other-oa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Keywords
- machine learning, systematic reviews, automation, automated data extraction, classification performance, class imbalance, EFSA
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.