Towards the automation of systematic reviews using natural language processing, machine learning, and deep learning: a comprehensive review
Open accessThe DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.
Abstract
Systematic reviews (SRs) constitute a critical foundation for evidence-based decision-making and policy formulation across various disciplines, particularly in healthcare and beyond. However, the inherently rigorous and structured nature of the SR process renders it laborious for human reviewers. Moreover, the exponential growth in daily published literature exacerbates the challenge, as SRs risk missing out on incorporating recent studies that could potentially influence research outcomes. This pressing need to streamline and enhance the efficiency of SRs has prompted significant interest in leveraging Artificial Intelligence (AI) techniques to automate various stages of the SR process. This review paper provides a comprehensive overview of the current AI methods employed for SR automation, a subject area that has not been exhaustively covered in previous literature. Through an extensive analysis of 52 related works and an original online survey, the primary AI techniques and their applications in automating key SR stages, such as search, screening, data extraction, and risk of bias assessment, are identified. The survey results offer practical insights into the current practices, experiences, opinions, and expectations of SR practitioners and researchers regarding future SR automation. Synthesis of the literature review and survey findings highlights gaps and challenges in the current landscape of SR automation using AI techniques. Based on these insights, potential future directions are discussed. This review aims to equip researchers and practitioners with a foundational understanding of the basic concepts, primary methodologies, and recent advancements in AI-driven SR automation while guiding computer scientists in exploring novel techniques to invigorate further and advance this field.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Literature review as a research methodology: An overview and guidelines
Snyder, H. (2019) · Journal of Business Research · doi:10.1016/j.jbusres.2019.07.039 · Cited by 11,524
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Machine learning: Algorithms, real-world applications and research directions
Sarker, I. H. (2021) · SN Computer Science · doi:10.1007/s42979-021-00592-x · Cited by 5,583
Systematic Literature Reviews in Software Engineering—A Systematic Literature Review
Kitchenham, B., Pearl Brereton, O., Budgen, D., Turner, M., Bailey, J., & Linkman, S. (2009) · Information and Software Technology · doi:10.1016/j.infsof.2008.09.009 · Cited by 4,537
Moher, D. (2001) · JAMA · doi:10.1001/jama.285.15.1987 · Cited by 2,545
The k-means Algorithm: A Comprehensive Survey and Performance Evaluation
Ahmed, M., Seraj, R., & Islam, S. M. S. (2020) · Electronics · doi:10.3390/electronics9081295 · Cited by 1,717
Systematic Reviews in Health Care: Meta‐Analysis in Context
(2001) · doi:10.1002/9780470693926 · Cited by 1,693
Bornmann, L., & Mutz, R. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23329 · Cited by 1,576
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638
The Systematic Review: An Overview
Aromataris, E., & Pearson, A. (2014) · AJN, American Journal of Nursing · doi:10.1097/01.naj.0000444496.24228.2c · Cited by 598
Marshall, I. J., Noel‐Storr, A., Kuiper, J., Thomas, J., & Wallace, B. C. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1287 · Cited by 435
Automation of systematic literature reviews: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106589 · Cited by 432
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411
Viewing systematic reviews and meta-analysis in social research through different lenses
Davis, J., Mengersen, K., Bennett, S., & Mazerolle, L. (2014) · SpringerPlus · doi:10.1186/2193-1801-3-511 · Cited by 364
Semi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351
RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv044 · Cited by 317
LitCovid: an open database of COVID-19 literature
Chen, Q., Allot, A., & Lu, Z. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkaa952 · Cited by 253
Nye, B., Li, J. J., Patel, R., Yang, Y., Marshall, I., Nenkova, A., & Wallace, B. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1019 · Cited by 224
SWIFT-Review: a text-mining workbench for systematic review
Howard, B. E., Phillips, J., Miller, K., Tandon, A., Mav, D., Shah, M. R., Holmgren, S., Pelch, K. E., Walker, V., Rooney, A. A., et al. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0263-z · Cited by 196
Tools to support the automation of systematic reviews: a scoping review.
Khalil, H., Ameen, D., & Zarnegar, A. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.12.005 · Cited by 190
Chai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01635-3 · Cited by 190
Reducing systematic review workload through certainty-based screening
Miwa, M., Thomas, J., O’Mara-Eves, A., & Ananiadou, S. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.06.005 · Cited by 190
ExaCT: automatic extraction of clinical trial characteristics from journal publications
Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-56 · Cited by 188
Bannach-Brown, A., Przybyła, P., Thomas, J., Rice, A. S. C., Ananiadou, S., Liao, J., & Macleod, M. R. (2019) · Systematic Reviews · doi:10.1186/s13643-019-0942-7 · Cited by 185
Howard, B. E., Phillips, J., Tandon, A., Maharana, A., Elmore, R., Mav, D., Sedykh, A., Thayer, K., Merrick, B. A., Walker, V., et al. (2020) · Environment International · doi:10.1016/j.envint.2020.105623 · Cited by 157
Using machine learning to advance synthesis and use of conservation and environmental evidence
Cheng, S. H., Augustin, C., Bethel, A., Gill, D., Anzaroot, S., Brun, J., DeWilde, B., Minnich, R. C., Garside, R., Masuda, Y. J., et al. (2018) · Conservation Biology · doi:10.1111/cobi.13117 · Cited by 155
Evaluation of machine-learning protocols for technology-assisted review in electronic discovery
Cormack, G. V., & Grossman, M. R. (2014) · Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval · doi:10.1145/2600428.2609601 · Cited by 149
Prioritising references for systematic reviews with RobotAnalyst: A user study
Przybyła, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M., McNaught, J., von Elm, E., Nolan, K., & Ananiadou, S. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1311 · Cited by 148
Gates, A., Johnson, C., & Hartling, L. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0707-8 · Cited by 148
Automating Biomedical Evidence Synthesis: RobotReviewer
Marshall, I., Kuiper, J., Banner, E., & Wallace, B. C. (2017) · Proceedings of ACL 2017, System Demonstrations · doi:10.18653/v1/p17-4002 · Cited by 96
Cross-Topic Learning for Work Prioritization in Systematic Review Creation and Update
Cohen, A. M., Ambert, K., & McDonagh, M. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3162 · Cited by 88
Trialstreamer: A living, automatically updated database of clinical trial reports
Marshall, I. J., Nye, B., Kuiper, J., Noel-Storr, A., Marshall, R., Maclean, R., Soboczenski, F., Nenkova, A., Thomas, J., & Wallace, B. C. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa163 · Cited by 83
Finding better active learners for faster literature reviews
Yu, Z., Kraft, N. A., & Menzies, T. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9587-0 · Cited by 83
Scott, A. M., Forbes, C., Clark, J., Carter, M., Glasziou, P., & Munn, Z. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.06.030 · Cited by 81
A Machine Learning Approach for Semi-Automated Search and Selection in Literature Studies
Ros, R., Bjarnason, E., & Runeson, P. (2017) · Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/3084226.3084243 · Cited by 80
Cohen, A. M., Smalheiser, N. R., McDonagh, M. S., Yu, C., Adams, C. E., Davis, J. M., & Yu, P. S. (2015) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocu025 · Cited by 78
BioReader: a text mining tool for performing classification of biomedical literature
Simon, C., Davidsen, K., Hansen, C., Seymour, E., Barnkob, M. B., & Olsen, L. R. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-2607-x · Cited by 74
Bui, D. D. A., Del Fiol, G., Hurdle, J. F., & Jonnalagadda, S. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.10.014 · Cited by 70
Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers
Bekhuis, T., & Demner-Fushman, D. (2012) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2012.05.002 · Cited by 70
A clustering approach for topic filtering within systematic literature reviews
Weißer, T., Saßmannshausen, T., Ohrndorf, D., Burggräf, P., & Wagner, J. (2020) · MethodsX · doi:10.1016/j.mex.2020.100831 · Cited by 64
LitSuggest: a web-based system for literature recommendation and curation using machine learning
Allot, A., Lee, K., Chen, Q., Luo, L., & Lu, Z. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab326 · Cited by 61
A visual analysis approach to validate the selection review of primary studies in systematic reviews
Felizardo, K. R., Andery, G. F., Paulovich, F. V., Minghim, R., & Maldonado, J. C. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.04.003 · Cited by 60
Thalia: semantic search engine for biomedical abstracts
Soto, A. J., Przybyła, P., & Ananiadou, S. (2019) · Bioinformatics · doi:10.1093/bioinformatics/bty871 · Cited by 60
Machine learning techniques for the automation of literature reviews and systematic reviews in EFSA
Jaspers, S., De Troyer, E., & Aerts, M. (2018) · EFSA Supporting Publications · doi:10.2903/sp.efsa.2018.en-1427 · Cited by 36
Advanced analytics for the automation of medical systematic reviews
Timsina, P., Liu, J., & El-Gayar, O. (2016) · Information Systems Frontiers · doi:10.1007/s10796-015-9589-7 · Cited by 33
The use of bibliography enriched features for automatic citation screening
Olorisade, B. K., Brereton, P., & Andras, P. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103202 · Cited by 21
A distantly supervised dataset for automated data extraction from diagnostic studies.
Norman, C., Leeflang, M., Spijker, R., Kanoulas, E., & Névéol, A. (2019) · Proceedings of the 18th BioNLP Workshop and Shared Task · doi:10.18653/v1/w19-5012 · Cited by 9
Research works printed but not linked: 65
- Abramovich F, Grinshtein V, Levy T (2021) Multiclass classification by sparse multinomial logistic regression. IEEE Trans Inf Theory 67(7):4637–4646. https://doi.org/10.1109/tit.2021.3075137 · DOI
- Aceves-Martins M, López-Cruz L, García-Botello M et al (2021) Interventions to prevent obesity in Mexican children and adolescents: systematic review. Prev Sci 23(4):563–586. https://doi.org/10.1007/s11121-021-01316-6 · DOI
- AHO AV (1990) Algorithms for finding patterns in strings. Elsevier, Amsterdam, pp 255–300. https://doi.org/10.1016/b978-0-444-88071-0.50010-2 · DOI · Paper page
- Aklouche B, Bounhas I, Slimani Y (2018) Query expansion based on NLP and word embeddings. In: Text retrieval conference. https://api.semanticscholar.org/CorpusID:155085448 · doi:10.6028/nist.sp.500-331.core-jarir
- Aklouche B, Bounhas I, Slimani Y (2019) Automatic query reweighting using co-occurrence graphs. In: Proceedings of the 16th international conference on applied computing 2019. IADIS Press, AC 2019. https://doi.org/10.33965/ac2019_201912l005 · doi:10.33965/ac2019_201912l005
- Alaofi M, Gallagher L, Sanderson M et al (2023) Can generative LLMS create query variants for test collections? An exploratory study. In: Proceedings of the 46th international ACM SIGIR conference on research and development in information retrieval. ACM, SIGIR ’23. https://doi.org/10.1145/3539618.3591960 · DOI
- Albawi S, Mohammed TA, Al-Zawi S (2017) Understanding of a convolutional neural network. In: 2017 international conference on engineering and technology (ICET). pp 1–6. https://doi.org/10.1109/ICEngTechnol.2017.8308186 · DOI · Paper page
- Almeida H, Meurs MJ, Kosseim L et al (2016) Data sampling and supervised learning for HIV literature screening. IEEE Trans Nanobiosci 15(4):354–361. https://doi.org/10.1109/bibm.2015.7359733 · DOI
- Angluin D (1988) Queries and concept learning. Mach Learn 2:319–342 (https://api.semanticscholar.org/CorpusID:11357867) · DOI · Paper page
- August ST (2001) Active learning: theory and applications. Stanford University 13(4):182
- Baranwal A, Bagwe BR, Vanitha M (2022) Machine learning in Python: diabetes prediction using machine learning. IGI Global, pp 882–908. https://doi.org/10.4018/978-1-6684-6291-1.ch046 · DOI
- Blaizot A, Veettil SK, Saidoung P et al (2022) Using artificial intelligence methods for systematic review in health sciences: a systematic review. Res Synth Methods 13(3):353–362. https://doi.org/10.1002/jrsm.1553 · DOI · Paper page
- Blei DM, Ng AY, Jordan MI (2003) Latent Dirichlet allocation. J Mach Learn Res 3(null):993–1022
- Booth A, Sutton A, Papaioannou D (2016) Systematic approaches to a successful literature review, 2nd edn. Sage, Thousand Oaks
- Bui DDA, Jonnalagadda S, Del Fiol G (2015) Automatically finding relevant citations for clinical guideline development. J Biomed Inform 57:436–445. https://doi.org/10.1016/j.jbi.2015.09.003 · DOI · Paper page
- Cawley M, Beardslee R, Beverly B et al (2020) Novel text analytics approach to identify relevant literature for human health risk assessments: a pilot study with health effects of in utero exposures. Environ Int 134:105228. https://doi.org/10.1016/j.envint.2019.105228 · DOI
- Cessie SL, Houwelingen JCV (1992) Ridge estimators in logistic regression. Appl Stat 41(1):191. https://doi.org/10.2307/2347628 · DOI · Paper page
- Chiu B, Crichton G, Korhonen A et al (2016) How to train good word embeddings for biomedical NLP. In: Proceedings of the 15th workshop on biomedical natural language processing. Association for Computational Linguistics. https://doi.org/10.18653/v1/w16-2922 · DOI · Paper page
- Cho K, van Merrienboer B, Gulcehre C et al (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). Association for Computational Linguistics. https://doi.org/10.3115/v1/d14-1179 · DOI · Paper page
- Cohn D, Atlas L, Ladner R (1994) Improving generalization with active learning. Mach Learn 15(2):201–221. https://doi.org/10.1007/bf00993277 · DOI · Paper page
- Devlin J, Chang MW, Lee K et al (2019) BERT: pre-training of deep bidirectional transformers for language understanding. http://arxiv.org/abs/1810.04805 · arXiv
- Frunza O, Inkpen D, Matwin S et al (2011) Exploiting the systematic review protocol for classification of medical abstracts. Artif Intell Med 51(1):17–25. https://doi.org/10.1016/j.artmed.2010.10.005 · DOI
- Gonzalez-Toral S, Freire R, Gualan R et al (2019) A ranking-based approach for supporting the initial selection of primary studies in a systematic literature review. In: 2019 XLV Latin American computing conference (CLEI). IEEE. https://doi.org/10.1109/clei47609.2019.235079 · DOI
- Gosavi A (2009) Reinforcement learning: a tutorial survey and recent advances. INFORMS J Comput 21(2):178–192. https://doi.org/10.1287/ijoc.1080.0305 · DOI · Paper page
- Gulo CA, Rúbio TR, Tabassum S et al (2015) Mining scientific articles powered by machine learning techniques. In: 2015 Imperial College computing student workshop (ICCSW 2015). Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik. https://doi.org/10.4230/OASIcs.ICCSW.2015.21 · DOI
- Guo G, Wang H, Bell D et al (2003) KNN model-based approach in classification. Springer, Berlin/Heidelberg, pp 986–996. https://doi.org/10.1007/978-3-540-39964-3_62 · DOI · Paper page
- Hans C (2011) Elastic net regression modeling with the orthant normal prior. J Am Stat Assoc 106(496):1383–1393. https://doi.org/10.1198/jasa.2011.tm09241 · DOI · Paper page
- Hashimoto K, Kontonatsios G, Miwa M et al (2016) Topic detection using paragraph vectors to support active learning in systematic reviews. J Biomed Inform 62:59–65. https://doi.org/10.1016/j.jbi.2016.06.001 · DOI · Paper page
- Hasny M, Vasile AP, Gianni M et al (2023) BERT for complex systematic review screening to support the future of medical research. Springer Nature Switzerland, Cham, pp 173–182. https://doi.org/10.1007/978-3-031-34344-5_21 · DOI
- Higgins JPT, Altman DG, Gotzsche PC et al (2011) The Cochrane collaboration’s tool for assessing risk of bias in randomised trials. BMJ 343(oct18 2):d5928–d5928. https://doi.org/10.1136/bmj.d5928 · DOI · Paper page
- Hoi SCH, Jin R, Lyu MR (2006) Large-scale text categorization by batch mode active learning. In: Proceedings of the 15th international conference on World Wide Web. ACM, WWW06. https://doi.org/10.1145/1135777.1135870 · DOI · Paper page
- Iparragirre A, Barrio I, Aramendi J et al (2023) Estimation of logistic regression parameters for complex survey data: a real data based simulation study. http://arxiv.org/abs/2303.01754 · arXiv
- Jelodar H, Wang Y, Yuan C et al (2018) Latent Dirichlet allocation (LDA) and topic modeling: models, applications, a survey. http://arxiv.org/abs/1711.04305 · arXiv
- Jha KK, Jha R, Jha AK et al (2021) A brief comparison on machine learning algorithms based on various applications: a comprehensive survey. In: 2021 IEEE international conference on computation system and information technology for sustainable solutions (CSITSS). IEEE. https://doi.org/10.1109/csitss54238.2021.9683524 · DOI
- Joachims T (2006) Training linear SVMs in linear time. In: Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, KDD06. https://doi.org/10.1145/1150402.1150429 · DOI · Paper page
- Jolliffe I (2014) Principal component analysis. https://doi.org/10.1002/9781118445112.stat06472 · DOI · Paper page
- Klein D, Manning CD (2003) Accurate unlexicalized parsing. In: Proceedings of the 41st annual meeting on association for computational linguistics—ACL ’03. Association for Computational Linguistics, ACL ’03. https://doi.org/10.3115/1075096.1075150 · DOI · Paper page
- Kontonatsios G, Spencer S, Matthew P et al (2020) Using a neural network-based feature extraction method to facilitate citation screening for systematic reviews. Expert Syst Appl X 6:100030. https://doi.org/10.1016/j.eswax.2020.100030 · DOI
- Kotsiantis SB (2011) Decision trees: a recent overview. Artif Intell Rev 39(4):261–283. https://doi.org/10.1007/s10462-011-9272-4 · DOI · Paper page
- Lewis DD (1998) Naive (Bayes) at forty: the independence assumption in information retrieval. Springer, Berlin/Heidelberg, pp 4–15. https://doi.org/10.1007/bfb0026666 · DOI · Paper page
- Mahendra MFR, Azizah NL (2023) Implementation of machine learning to predict the weather using a support vector machine: Implementasi machine learning untuk memprediksi cuaca menggunakan support vector machine. Preprint. https://doi.org/10.21070/ups.2889 · doi:10.21070/ups.2889
- McGreevy KM, Church FC (2020). Active learning survey. https://doi.org/10.1037/t81767-000 · doi:10.1037/t81767-000
- Mergel GD, Silveira MS, da Silva TS (2015) A method to support search string building in systematic literature reviews through visual text mining. In: Proceedings of the 30th annual ACM symposium on applied computing. ACM, SAC 2015. https://doi.org/10.1145/2695664.2695902 · DOI · Paper page
- Mitchell TM (1997) Machine learning. McGraw-Hill, New York
- Moreno-Garcia CF, Jayne C, Elyan E et al (2023) A novel application of machine learning and zero-shot classification methods for automated abstract screening in systematic reviews. Decis Anal J 6:100162. https://doi.org/10.1016/j.dajour.2023.100162 · DOI · Paper page
- Nadkarni PM (2002) An introduction to information retrieval: applications in genomics. Pharmacogenomics J 2(2):96–102. https://doi.org/10.1038/sj.tpj.6500084 · doi:10.1038/sj.tpj.6500084
- Natukunda A, Muchene LK (2023) Unsupervised title and abstract screening for systematic review: a retrospective case-study using topic modelling methodology. Syst Rev. https://doi.org/10.1186/s13643-022-02163-4 · DOI
- Ofori-Boateng R, Aceves-Martins M, Jayne C et al (2023) Evaluation of attention-based LSTM and Bi-LSTM networks for abstract text classification in systematic literature review automation. Procedia Comput Sci 222:114–126. https://doi.org/10.1016/j.procs.2023.08.149 · DOI
- Orel E, Ciglenecki I, Thiabaud A et al (2023) An automated literature review tool (literev) for streamlining and accelerating research using natural language processing and machine learning: descriptive performance evaluation study. J Med Internet Res 25:e39736. https://doi.org/10.2196/39736 · DOI
- Paul L, Suman A, Sultan N (2013) Methodological analysis of principal component analysis (PCA) method. Int J Comput Eng Manag 16:32–38
- Popuri SK (2022) An approximation method for fitted random forests. http://arxiv.org/2207.02184. https://api.semanticscholar.org/CorpusID:250279991
- Radford A, Wu J, Child R et al (2019) Language models are unsupervised multitask learners. OpenAI. https://api.semanticscholar.org/CorpusID:160025533
- Rogers A, Gardner M, Augenstein I (2023) QA dataset explosion: a taxonomy of NLP resources for question answering and reading comprehension. ACM Comput Surv 55(10):1–45. https://doi.org/10.1145/3560260 · DOI · Paper page
- Rúbio TR, Gulo CA (2016) Enhancing academic literature review through relevance recommendation: using bibliometric and text-based features for classification. In: 2016 11th Iberian conference on information systems and technologies (CISTI). IEEE, pp 1–6. https://doi.org/10.1109/cisti.2016.7521620 · DOI
- Russell-Rose T, Chamberlain J, Shokraneh F (2019) A visual approach to query formulation for systematic search. In: Proceedings of the 2019 conference on human information interaction and retrieval. ACM, CHIIR ’19. https://doi.org/10.1145/3295750.3298919 · DOI
- Scells H, Zuccon G, Koopman B et al (2020) Automatic Boolean query formulation for systematic review literature search. In: Proceedings of the web conference 2020. ACM, WWW ’20. https://doi.org/10.1145/3366423.3380185 · DOI · Paper page
- Scheffer T, Decomain C, Wrobel S (2001) Active hidden Markov models for information extraction. In: International symposium on intelligent data analysis. Springer, pp 309–318 · DOI · Paper page
- Schmidt L, Weeds J, Higgins J (2020) Data mining in clinical trial text: transformers for classification and question answering tasks. In: Proceedings of the 13th international joint conference on biomedical engineering systems and technologies. SCITEPRESS—Science and Technology Publications. https://doi.org/10.5220/0008945700830094 · DOI
- Shannon CE (1948) A mathematical theory of communication. Bell Syst Tech J 27(3):379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x · DOI · Paper page
- Song J, Lee JK, Choi J et al (2020) Deep learning-based extraction of predicate-argument structure (PAS) in building design rule sentences★. J Comput Des Eng 7(5):563–576. https://doi.org/10.1093/jcde/qwaa046 · DOI
- Tomassetti F, Rizzo G, Vetro A et al (2011) Linked data approach for selection process automation in systematic reviews. In: 15th annual conference on evaluation and assessment in software engineering (EASE 2011). IET. https://doi.org/10.1049/ic.2011.0004 · DOI · Paper page
- Vaswani A, Shazeer N, Parmar N et al (2023) Attention is all you need. http://arxiv.org/abs/1706.03762 · arXiv
- Walkowiak T, Datko S, Maciejewski H (2018) Bag-of-Words, Bag-of-Topics and Word-to-Vec based subject classification of text documents in polish–a comparative study. Springer International Publishing, Cham, pp 526–535. https://doi.org/10.1007/978-3-319-91446-6_49 · doi:10.1007/978-3-319-91446-6_49
- Xie Q, Bishop JA, Tiwari P et al (2022) Pre-trained language models with domain knowledge for biomedical extractive summarization. Knowl-Based Syst 252:109460. https://doi.org/10.1016/j.knosys.2022.109460 · DOI · Paper page
- Zhang D, Baclawski KP, Tsotras VJ (2009) B+-Tree. Springer US, pp 197–200. https://doi.org/10.1007/978-0-387-39940-9_739 · doi:10.1007/978-0-387-39940-9_739
Cited by
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2025) · F1000Research · doi:10.12688/f1000research.51117.3 · Cited by 27
ChatGPT as a Tool for Biostatisticians: A Tutorial on Applications, Opportunities, and Limitations
Dobler, D., Binder, H., Boulesteix, A., Igelmann, J., Köhler, D., Mansmann, U., Pauly, M., Scherag, A., Schmid, M., Al Tawil, A., et al. (2025) · Statistics in Medicine · doi:10.1002/sim.70263 · Cited by 20
Nykvist, B., Macura, B., Xylia, M., & Olsson, E. (2025) · Environmental Evidence · doi:10.1186/s13750-025-00360-x · Cited by 17
Robleto, E., & Shehadeh, L. A. (2025) · American Journal of Physiology. Heart and Circulatory Physiology · doi:10.1152/ajpheart.00374.2025 · Cited by 2
Metadata
- Authors
- Ofori-Boateng, R., Aceves-Martins, M., Wiratunga, N., & Moreno-Garcia, C. F.
- Venue
- Artificial Intelligence Review
- Published
- 2024-07-09
- Volume
- 57
- Issue
- 8
- ISSN
- 0269-2821 (ISSN-L)
- 1573-7462
- OpenAlex
- W4400454376
- Cited by
- 86
- References
- 54
- FWCI
- 11.80
- Type
- journal article
- Language
- en
- Open access
- hybrid
- Licence
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.