A critical analysis of studies that address the use of text mining for citation screening in systematic reviews
Open accessThe DOI matches its Crossref record; 78 of the 78 references it cites were found in a registry or an index.
Abstract
Background: Since the introduction of the systematic review process to Software Engineering in 2004, researchers have investigated a number of ways to mitigate the amount of effort and time taken to filter through large volumes of literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These algorithms can be trained to develop the ability to categorise, using so called “training data sets” of records screened by human reviewers.”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [55]
In the citing paper
“As mentioned above, these algorithms/software are more and more used for article screening in systematic reviews [53, 55, 56]”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [55]
In the citing paper
“This gap manifests as policyrelevant information being lost amongst a sea of websites, reports and peer-reviewed articles 9,10 .”
From Software support for environmental evidence synthesis (2018) · cites this paper as [9]
Citations
Cites
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750
Machine learning in automated text categorization
Sebastiani, F. (2002) · ACM Computing Surveys · doi:10.1145/505282.505283 · Cited by 7,926
Supervised Machine Learning: A Review of Classification Techniques
Kotsiantis, S. B. (2007) · Informatica · Cited by 6,169
A systematic review of systematic review process research in software engineering
Kitchenham, B., & Brereton, P. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.07.010 · Cited by 1,094
Self-citationBuilding knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 8×
“We only retrieved and worked with the papers O’Mara-Eves et al. included in [62].”
BackgroundMethodInfluentialA Review of Machine Learning Algorithms for Text-Documents Classification
Baharudin, B., Lee, L. H., & Khan, K. (2010) · Journal of Advances in Information Technology · doi:10.4304/jait.1.1.4-20 · Cited by 653
Deploying an interactive machine learning system in an evidence-based practice center: abstrackr
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012) · Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium · doi:10.1145/2110363.2110464 · Cited by 532
Text Classification Using Machine Learning Techniques
Ikonomakis, M., Kotsiantis, S. B., & Tampakas, V. T. (2005) · Cited by 490
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 6×
“Mean recall was 95% and above in [4, 5, 17, 30, 48, 56, 74, 77, 82] while it was below 95% in [9, 49].”
BackgroundMethodInfluential
Research works printed but not linked: 5
- Cohen , A. M. , Ambert , K. and McDonagh , M. 2010 . A Prospective Evaluation of an Automated Classification System to Support Evidence-based Medicine and Systematic Review . AMIA Annual Symposium Proceedings (2010), 121--125. Cohen, A. M., Ambert, K. and McDonagh, M. 2010. A Prospective Evaluation of an Automated Classification System to Support Evidence-based Medicine and Systematic Review. AMIA Annual Symposium Proceedings (2010), 121--125.
- 10.1109/ESEM.2011.16 · DOI · Paper page
- Handbook for Systematic Reviews of Interventions : 2015 . http://handbook.cochrane.org/. Accessed: 2015-09-10. Handbook for Systematic Reviews of Interventions: 2015. http://handbook.cochrane.org/. Accessed: 2015-09-10.
- Jonnalagadda S. Petitti D. and Manuscript A. 2013. A New Iterative Method to Reduce Workload in the Systematic Review Process. International journal of computational biology and drug design. 6 0 (Feb. 2013) 5--17. Jonnalagadda S. Petitti D. and Manuscript A. 2013. A New Iterative Method to Reduce Workload in the Systematic Review Process. International journal of computational biology and drug design . 6 0 (Feb. 2013) 5--17. · DOI · Paper page
- Olorisade B. K. 2012. Informal Aggregation Technique for Software Engineering Experiments. IJCSI International Journal of Computer Science Issues. 9 (2012) 199--204. Olorisade B. K. 2012. Informal Aggregation Technique for Software Engineering Experiments. IJCSI International Journal of Computer Science Issues . 9 (2012) 199--204.
Cited by
Automation of systematic literature reviews: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106589 · Cited by 432
“Besides, [12,17 – 20] studied tools to automate a – part of – the systematic review process, not always the full process.”
BackgroundArtificial intelligence to automate the systematic review of scientific literature
de la Torre-López, J., Ramírez, A., & Romero, J. R. (2023) · Computing · doi:10.1007/s00607-023-01181-x · Cited by 170
“Two other secondary studies are focused on the analysis of ML techniques for the paper selection task [9, 12].”
BackgroundGates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1222-2 · Cited by 133
Cierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01805-4 · Cited by 85
“These algorithms are trained according to the computer–human interaction where the availability of a training data set and the purpose of the classification strategy are the key points [41, 49, 51, 55].”
BackgroundMethodFinding better active learners for faster literature reviews
Yu, Z., Kraft, N. A., & Menzies, T. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9587-0 · Cited by 83
“pe that other researchers will explore similar work on different datasets). Existing active learning techniques in citation screening have been criticized by Olorisade et al. for being not replicable [43,44]. To this end, we have published all our code at https://github.com/fastread/src and all our data at https://doi…”
BackgroundInfluentialA Machine Learning Approach for Semi-Automated Search and Selection in Literature Studies
Ros, R., Bjarnason, E., & Runeson, P. (2017) · Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/3084226.3084243 · Cited by 80
“…performed a re-review of O’Mara-Eves et al.’s SLR to further investigate the text mining techniques used. claim that there is insufficient evidence of automatic primary study selection working in practice, and that researchers have insufficient insight into how the tools they are using works [24].”
Information retrieval methodology for aiding scientific database search
Marcos-Pablos, S., & García‐Peñalvo, F. J. (2018) · Soft Computing · doi:10.1007/s00500-018-3568-0 · Cited by 93
Olorisade, B. K., Brereton, P., & Andras, P. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.07.010 · Cited by 54
Self-citationSoftware support for environmental evidence synthesis
Westgate, M. J., Haddaway, N. R., Cheng, S. H., McIntosh, E. J., Marshall, C., & Lindenmayer, D. B. (2018) · Nature Ecology & Evolution · doi:10.1038/s41559-018-0502-x · Cited by 51
“For example, text-mining approaches are associated with a risk of missing up to 5% of relevant studies (only 95% recall) when compared with manual screening processes 9 .”
MethodAutomated screening of research studies for systematic reviews using study characteristics
Tsafnat, G., Glasziou, P., Karystianis, G., & Coiera, E. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0724-7 · Cited by 44
“A more recent review of these methods questions the confidence in these results due to the similarity of the methods and the wide range of results [12].”
BackgroundA comparison of automatic Boolean query formulation for systematic reviews
Scells, H., Zuccon, G., & Koopman, B. (2020) · Information Retrieval · doi:10.1007/s10791-020-09381-1 · Cited by 57
Past and future uses of text mining in ecology and evolution
Farrell, M. J., Brierley, L., Willoughby, A. R., Yates, A., & Mideo, N. (2022) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2021.2721 · Cited by 50
Machine learning techniques for the automation of literature reviews and systematic reviews in EFSA
Jaspers, S., De Troyer, E., & Aerts, M. (2018) · EFSA Supporting Publications · doi:10.2903/sp.efsa.2018.en-1427 · Cited by 36
“Following the results from O’Mara-Eves et al. (2015), the three performance measures that are used throughout the most studies are recall, precision and the F measure.”
BackgroundMethodInfluentialMelo, I. C., Alves, P. N., Queiroz, G. A., Yushimito, W. F., & Pereira, J. (2023) · Sustainability · doi:10.3390/su15064917 · Cited by 84
Sundaram, G., & Berleant, D. (2023) · Lecture notes in networks and systems · doi:10.1007/978-981-99-3243-6_7 · Cited by 47
Measuring the impact of screening automation on meta-analyses of diagnostic test accuracy
Norman, C. R., Leeflang, M. M. G., Porcher, R., & Névéol, A. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1162-x · Cited by 33
Popoff, E., Besada, M., Jansen, J. P., Cope, S. F., & Kanters, S. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01520-5 · Cited by 31
Codepori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology
Rasheed, Z., Malik, S., Kemell, K.-K., Muhammad, W., Saari, M., Systä, K., & Abrahamsson, P. (2024) · doi:10.2139/ssrn.4979510 · Cited by 27
The use of bibliography enriched features for automatic citation screening
Olorisade, B. K., Brereton, P., & Andras, P. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103202 · Cited by 21
“The SVM was chosen for this study because a previous study has found it to be the most used in the context of automatic CS [14] and it has also been shown that it outperforms other classical ML algorithms good at text classification like Na¨ıve Bayes [15].”
MethodSelf-citationReproducibility and Data Storage for Active Learning-Aided Systematic Reviews
Lombaers, P., de Bruin, J., & van de Schoot, R. (2024) · Applied Sciences · doi:10.3390/app14093842 · Cited by 20
Artificial intelligence and automation of systematic reviews in women's health
Amezcua‐Prieto, C., Fernández-Luna, J. M., Huete, J. F., Bueno‐Cavanillas, A., & Khan, K. S. (2020) · Current Opinion in Obstetrics & Gynecology · doi:10.1097/gco.0000000000000643 · Cited by 17
An Interactive Relevance Feedback Interface for Evidence-Based Health Care
Donoso-Guzmán, I., & Parra, D. (2018) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3172944.3172953 · Cited by 17
Large-scale simulation study of active learning models for systematic reviews
Teijema, J. J., de Bruin, J., Bagheri, A., & van de Schoot, R. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00777-0 · Cited by 13
Transformative Automation: AI in Scientific Literature Reviews
Zala, K., Acharya, B., Mashru, M., Palaniappan, D., Gerogiannis, V. C., Kanavos, A., & Karamitsos, I. (2024) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2024.01501122 · Cited by 21
slr-kit: A semi-supervised machine learning framework for systematic literature reviews
Facchinetti, T., Benetti, G., Giuffrida, D., & Nocera, A. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109266 · Cited by 12
Norman, C. R., Gargon, E., Leeflang, M., Névéol, A., & Williamson, P. R. (2019) · Database · doi:10.1093/database/baz109 · Cited by 11
Synthesizing Knowledge through A Data Analytics-Based Systematic Literature Review Protocol
Xiong, R. R., Liu, C. Z., & Choo, K. R. (2023) · Information Systems Frontiers · doi:10.1007/s10796-023-10432-3 · Cited by 16
Gates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L. (2019) · doi:10.23970/ahrqepcmethmachineperformance · Cited by 8
Large-Scale Simulation Study of Active Learning Models for Systematic Reviews
Teijema, J. J., de Bruin, J., Bagheri, A., & van de Schoot, R. (2023) · doi:10.31234/osf.io/2w3rm · Cited by 8
Felizardo, K. R., Ramos, A. M., Melo, C. D. O., de Souza, É. F., Vijaykumar, N. L., & Nakagawa, E. Y. (2021) · Journal of the Brazilian Computer Society · doi:10.1186/s13173-021-00109-7 · Cited by 11
Machine learning e revisão sistemática de literatura automatizada: uma revisão sistemática
Tsunoda, D. F., Moreira, P. S. D. C., & Guimarães, A. J. R. (2020) · Revista Tecnologia e Sociedade · doi:10.3895/rts.v16n45.12119 · Cited by 12
Reproducibility and Data storage Checklist for Active Learning-Aided Systematic ReviewsPreprint
Lombaers, P., Bruin, J. D., & van de Schoot, R. (2023) · doi:10.31234/osf.io/g93zf · Cited by 6
Systematic review automation methods
Norman, C. R. (2020) · theses.fr (ABES) · doi:10.70675/044ffa34z5c9bz433az99c8z64697dcd28a2 · Cited by 8
Napoleão, B. M., Petrillo, F. D. S., & Hallé, S. (2021) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa53835.2021.00015 · Cited by 7
Silva, G., Neto, P. D. A. D. S., Moura, R. S., Araujo, A. C., Castro, O. C. C., & Ibiapina, I. M. D. S. (2019) · Brazilian Conference on Intelligent Systems · doi:10.1109/bracis.2019.00147 · Cited by 5
Reporting Statistical Validity and Model Complexity in Machine Learning based Computational Studies
Olorisade, B. K., Brereton, P., & András, P. (2017) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3084226.3084283 · Cited by 3
Self-citationAlvarez, M. C. Z., Barreneche, J. G., & Aristizabal, J. K. N. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-86702-7_17
Seed-driven document ranking for systematic reviews in evidence-based medicine
Lee, E. (2020) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/146709
Pauzi, Z., & Capiluppi, A. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-64182-4_6
Rule-Based Classification and Clustering for Thematic Analysis of Research Papers
Abid, M. H., & Yao, J. (2026) · IEEE Annual Computing and Communication Workshop and Conference (CCWC) · doi:10.1109/ccwc67433.2026.11393808
Shankar, R., Goh, Z., & Xu, Q. (2026) · Campbell Systematic Reviews · doi:10.1177/18911803261469813
Metadata
- Authors
- Olorisade, B. K., de Quincey, E., Brereton, P., & Andras, P.
- Affiliations
- Keele University (GB)
- Keele University
- Venue
- Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2016-06
- Pages
- 1-11
- OpenAlex
- W2406978180
- Cited by
- 42
- References
- 78
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 4 · 2023: 6 · 2022: 3 · 2021: 4 · 2020: 6 · 2019: 6 · 2018: 6 · 2017: 4
- FWCI
- 1.92
- Type
- journal article
- Language
- en
- Open access
- green
- eprints.keele.ac.uk
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- Research Output (Edinburgh Napier University) (submittedVersion)
- Edinburgh Napier Research Repository (Edinburgh Napier University) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- text mining, citation screening, systematic reviews, software engineering literature
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.