A critical analysis of studies that address the use of text mining for citation screening in systematic reviews
Open accessThe DOI matches its Crossref record; 78 of the 78 references it cites were found in a registry or an index.
Abstract
Background: Since the introduction of the systematic review process to Software Engineering in 2004, researchers have investigated a number of ways to mitigate the amount of effort and time taken to filter through large volumes of literature.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These algorithms can be trained to develop the ability to categorise, using so called “training data sets” of records screened by human reviewers.”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [55]
In the citing paper
“As mentioned above, these algorithms/software are more and more used for article screening in systematic reviews [53, 55, 56]”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [55]
In the citing paper
“This gap manifests as policyrelevant information being lost amongst a sea of websites, reports and peer-reviewed articles 9,10 .”
From Software support for environmental evidence synthesis (2018) · cites this paper as [9]
Citations
Cites
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750
Machine learning in automated text categorization
Sebastiani, F. (2002) · ACM Computing Surveys · doi:10.1145/505282.505283 · Cited by 7,926
Supervised Machine Learning: A Review of Classification Techniques
Kotsiantis, S. B. (2007) · Informatica · Cited by 6,007
A systematic review of systematic review process research in software engineering
Kitchenham, B., & Brereton, P. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.07.010 · Cited by 1,094
Self-citationBuilding knowledge through families of experiments
Basili, V. R., Shull, F., & Lanubile, F. (1999) · IEEE Transactions on Software Engineering · doi:10.1109/32.799939 · Cited by 852
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 8×
“We only retrieved and worked with the papers O’Mara-Eves et al. included in [62].”
BackgroundMethodInfluentialA Review of Machine Learning Algorithms for Text-Documents Classification
Baharudin, B., Lee, L. H., & Khan, K. (2010) · Journal of Advances in Information Technology · doi:10.4304/jait.1.1.4-20 · Cited by 653
Deploying an interactive machine learning system in an evidence-based practice center: abstrackr
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012) · Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium · doi:10.1145/2110363.2110464 · Cited by 532
Text Classification Using Machine Learning Techniques
Ikonomakis, M., Kotsiantis, S. B., & Tampakas, V. T. (2005) · Cited by 490
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 6×
“Mean recall was 95% and above in [4, 5, 17, 30, 48, 56, 74, 77, 82] while it was below 95% in [9, 49].”
BackgroundMethodInfluentialO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Semi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · cited 4×
“(Features Simple) [81] Yield, burden SVM Ensembles”
InfluentialStatistical Analyses and Reproducible Research
Gentleman, R., & Temple Lang, D. (2007) · Journal of Computational and Graphical Statistics · doi:10.1198/106186007x178663 · Cited by 348
A Visual Text Mining approach for Systematic Reviews
Malheiros, V., Hohn, E., Pinho, R., Mendonca, M., & Maldonado, J. C. (2007) · First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) · doi:10.1109/esem.2007.21 · Cited by 329
The role of replications in Empirical Software Engineering
Shull, F. J., Carver, J. C., Vegas, S., & Juristo, N. (2008) · Empirical Software Engineering · doi:10.1007/s10664-008-9060-1 · Cited by 315
Automating data extraction in systematic reviews: a systematic review
Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0066-7 · Cited by 287
Text Classification and Classifiers:A Survey
Korde, V. (2012) · International Journal of Artificial Intelligence & Applications · doi:10.5121/ijaia.2012.3208 · Cited by 249
Proceedings of the 23rd International Conference on Computational Linguistics: Posters
Joshi, A. K., Huang, C., & Jurafsky, D. (2010) · Cited by 215
Reducing systematic review workload through certainty-based screening
Miwa, M., Thomas, J., O’Mara-Eves, A., & Ananiadou, S. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.06.005 · Cited by 190
Rathbone, J., Hoffmann, T., & Glasziou, P. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0067-6 · Cited by 184 · cited 1×
“, that has been evaluated by another group in [66].”
BackgroundShemilt, I., Simon, A., Hollands, G. J., Marteau, T. M., Ogilvie, D., O'Mara‐Eves, A., Kelly, M. P., & Thomas, J. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1093 · Cited by 170
Design of High Performance MIPS Cryptography Processor Based on T-DES Algorithm
Singh, K. P., & Parmar, S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1503.03166 · Cited by 161
Using GQM and TAM to evaluate StArt – a tool that supports Systematic Review
Hernandes, E., Zamboni, A., Fabbri, S., & Di Thommazo, A. (2012) · CLEI Electronic Journal · doi:10.19153/cleiej.15.1.2 · Cited by 146
Knowledge-Sharing Issues in Experimental Software Engineering
Shull, F. J., Mendoncça, M. G., Basili, V. R., Carver, J. C., Maldonado, J. C., Fabbri, S., Travassos, G. H., & de Oliveira, M. C. F. (2004) · Empirical Software Engineering · doi:10.1023/b:emse.0000013516.80487.33 · Cited by 137
Automatic text classification to support systematic reviews in medicine
García Adeva, J. J., Pikatza Atxa, J. M., Ubeda Carrillo, M., & Ansuategi Zengotitabengoa, E. (2014) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.08.047 · Cited by 125 · cited 1×
“The majority of the studies used cross validation (CV) – 5x2 cross validation was used in [6, 10, 12, 16, 17, 49, 56] and 10-fold cross validation in [4, 5, 9, 31, 48, 74, 76], 5-fold cross validation was used in [18]; [48, 49] used both 5x2 and 10-fold CV with stratified random sampling, multiple n-way cross validati…”
MethodActive learning for biomedical citation screening
Wallace, B. C., Small, K., Brodley, C. E., & Trikalinos, T. A. (2010) · Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1835804.1835829 · Cited by 123
Identifying Barriers to the Systematic Literature Review Process
Carver, J. C., Hassler, E. E., Hernandes, E. C. M., & Kraft, N. A. (2013) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2013.28 · Cited by 104
Gonzalez-Barahona, J. M., & Robles, G. (2011) · Empirical Software Engineering · doi:10.1007/s10664-011-9181-9 · Cited by 103
A new algorithm for reducing the workload of experts in performing systematic reviews
Matwin, S., Kouznetsov, A., Inkpen, D., Frunza, O., & O'Blenis, P. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.004325 · Cited by 100 · cited 5×
“Mean recall was 95% and above in [4, 5, 17, 30, 48, 56, 74, 77, 82] while it was below 95% in [9, 49].”
BackgroundMethodInfluentialCross-Topic Learning for Work Prioritization in Systematic Review Creation and Update
Cohen, A. M., Ambert, K., & McDonagh, M. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3162 · Cited by 88 · cited 1×
“…in [4, 5, 9, 31, 48, 74, 76], 5-fold cross validation was used in [18]; [48, 49] used both 5x2 and 10-fold CV with stratified random sampling, multiple n-way cross validation with n ranging between 2 to 256 increasing by power of 2 was used in [15] while cost rejection sampling was used in [11].”
MethodTools to Support Systematic Literature Reviews in Software Engineering: A Mapping Study
Marshall, C., & Brereton, P. (2013) · 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2013.32 · Cited by 87 · cited 1×
“Citation screening: This stage has attracted the most attention in terms of an individual SR stage automation [52].”
BackgroundReplicating software engineering experiments: a poisoned chalice or the Holy Grail
Miller, J. (2005) · Information and Software Technology · doi:10.1016/j.infsof.2004.08.005 · Cited by 84
Who Should Label What? Instance Allocation in Multiple Expert Active Learning
Wallace, B., Small, K., Brodley, C. E., & Trikalinos, T. A. (2011) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972818.16 · Cited by 75
Using Visual Text Mining to Support the Study Selection Activity in Systematic Literature Reviews
Felizardo, K. R., Salleh, N., Martins, R. M., Mendes, E., MacDonell, S. G., & Maldonado, J. C. (2011) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2011.16 · Cited by 71
Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers
Bekhuis, T., & Demner-Fushman, D. (2012) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2012.05.002 · Cited by 70 · cited 4×
“…majority of the studies used cross validation (CV) – 5x2 cross validation was used in [6, 10, 12, 16, 17, 49, 56] and 10-fold cross validation in [4, 5, 9, 31, 48, 74, 76], 5-fold cross validation was used in [18]; [48, 49] used both 5x2 and 10-fold CV with stratified random sampling, multiple…”
BackgroundMethodInfluentialCohen, A. M., Ambert, K. H., & McDonagh, M. S. (2012) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-12-33 · Cited by 70
Text Map Explorer: a Tool to Create and Explore Document Maps
Paulovich, F. V., & Minghim, R. (2006) · International Conference on Information Visualisation (IV) · doi:10.1109/iv.2006.104 · Cited by 66
Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., Schmid, C. H., Bertram, L., Lill, C. M., Cohen, J. T., & Trikalinos, T. A. (2012) · Genetics in Medicine · doi:10.1038/gim.2012.7 · Cited by 62
Optimizing feature representation for automated systematic review work prioritization.
Cohen, A. M. (2008) · PubMed Central · Cited by 62
A new iterative method to reduce workload in systematic review process
Jonnalagadda, S., & Petitti, D. (2013) · International Journal of Computational Biology and Drug Design · doi:10.1504/ijcbdd.2013.052198 · Cited by 61
Towards Automating the Initial Screening Phase of a Systematic Review
Bekhuis, T., & Demner‐Fushman, D. (2010) · Studies in health technology and informatics · doi:10.3233/978-1-60750-588-4-146 · Cited by 61
A visual analysis approach to validate the selection review of primary studies in systematic reviews
Felizardo, K. R., Andery, G. F., Paulovich, F. V., Minghim, R., & Maldonado, J. C. (2012) · Information and Software Technology · doi:10.1016/j.infsof.2012.04.003 · Cited by 60
Hassler, E. E., Carver, J. C., Kraft, N. A., & Hale, D. P. (2014) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/2601248.2601274 · Cited by 60
Analysis of the influence of communication between researchers on experiment replication
Vegas, S., Juristo, N., Moreno, A. M., Solari, M., & Letelier, P. (2006) · International Symposium on Empirical Software Engineering · doi:10.1145/1159733.1159741 · Cited by 57
Tools to support systematic reviews in software engineering: a feature analysis
Marshall, C., Brereton, P., & Kitchenham, B. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601270 · Cited by 56
Yu, W., Clyne, M., Dolan, S. M., Yesupriya, A., Wulf, A., Liu, T., Khoury, M. J., & GWINN, M. L. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-205 · Cited by 52
Exploiting the systematic review protocol for classification of medical abstracts
Frunza, O., Inkpen, D. Z., Matwin, S., Klement, W., & O’Blenis, P. (2010) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2010.10.005 · Cited by 45
Bekhuis, T., Tseytlin, E., Mitchell, K. J., & Demner-Fushman, D. (2014) · PLoS ONE · doi:10.1371/journal.pone.0086277 · Cited by 45 · cited 3×
“Feature extraction approach was rarely used, LDA was used in [60] and topic modelling in [6].”
BackgroundMethodAn effective general purpose approach for automated biomedical document classification.
(2006) · Europe PMC (PubMed Central) · Cited by 44
SLR-TOOL - A Tool for Performing Systematic Literature Reviews
(2010) · Proceedings of the 5th International Conference on Software and Data Technologies · doi:10.5220/0003003601570166 · Cited by 42
Linked data approach for selection process automation in systematic reviews
Tomassetti, F., Rizzo, G. F., Vetrò, A., Ardito, L., Torchiano, M., & Morisio, M. (2011) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1049/ic.2011.0004 · Cited by 41
SLuRp: a tool to help large complex systematic literature reviews deliver valid and rigorous results
Bowes, D., Hall, T., & Beecham, S. (2012) · International Workshop on Evidential Assessment of Software Technologies (EAST) · doi:10.1145/2372233.2372243 · Cited by 39
Felizardo, K. R., Nakagawa, E. Y., Feitosa, D., Minghim, R., & Maldonado, J. C. (2010) · Electronic Workshops in Computing · doi:10.14236/ewic/ease2010.5 · Cited by 35
(2010) · Cited by 32
Dalal, S. R., Shekelle, P. G., Hempel, S., Newberry, S. J., Motala, A., & Shetty, K. D. (2013) · Medical Decision Making · doi:10.1177/0272989x12457243 · Cited by 32
Cohen, A. M. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.008177 · Cited by 32
Towards evidence-based ontology for supporting systematic literature review
Sun, Y., Yang, Y., Zhang, H., Zhang, W., & Wang, Q. (2012) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1049/ic.2012.0022 · Cited by 28
SESRA: a web-based automated tool to support the systematic literature review process
Molléri, J. S., & Benitti, F. B. V. (2015) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/2745802.2745825 · Cited by 28
A Federated Search Approach to Facilitate Systematic Literature Review in Software Engineering
Ghafari, M. (2012) · International Journal of Software Engineering & Applications · doi:10.5121/ijsea.2012.3202 · Cited by 27
Cohen, A. M., Ambert, K. H., & McDonagh, M. S. (2010) · PubMed · Cited by 26
Modeling annotation time to reduce workload in comparative effectiveness reviews
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2010) · Proceedings of the 1st ACM International Health Informatics Symposium · doi:10.1145/1882992.1882999 · Cited by 25
Matwin, S., & Sazonova, V. (2012) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001072 · Cited by 24
Combining relevancy and methodological quality into a single ranking for evidence-based medicine
Choi, S., Ryu, B., Yoo, S., & Choi, J. (2012) · Information Sciences · doi:10.1016/j.ins.2012.05.027 · Cited by 23
Felizardo, K. R., Souza, S. R. S., & Maldonado, J. C. (2013) · International Workshop on Replication in Empirical Software Engineering Research (RESER) · doi:10.1109/reser.2013.9 · Cited by 22
Adeva, J. J. G., & Calvo, R. A. (2006) · IEEE Internet Computing · doi:10.1109/mic.2006.85 · Cited by 21
Facilitating biomedical systematic reviews using ranked text retrieval and classification
Martínez, D., Karimi, S., Cavedon, L., & Baldwin, T. (2008) · Cited by 20
Kim, S., & Choi, J. (2012) · Healthcare Informatics Research · doi:10.4258/hir.2012.18.1.18 · Cited by 19
Text Data Pre-processing and Dimensionality Reduction Techniques for Document Clustering
Kumar, S. (2012) · Cited by 19
Classifying Biomedical Abstracts Using Committees of Classifiers and Collective Ranking Techniques
Kouznetsov, A., Matwin, S., Inkpen, D. Z., Razavi, A. H., Frunza, O., Sehatkar, M., Seaward, L., & O’Blenis, P. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-01818-3_29 · Cited by 17
Analysing the Use of Graphs to Represent the Results of Systematic Reviews in Software Engineering
Felizardo, K. R., Riaz, M., Sulayman, M., Mendes, E., MacDonell, S. G., & Maldonado, J. C. (2011) · Brazilian Symposium on Software Engineering · doi:10.1109/sbes.2011.9 · Cited by 17
Razavi, A. H., Matwin, S., Inkpen, D. Z., & Kouznetsov, A. (2009) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw.2009.49 · Cited by 14
Combining relevance assignment with quality of the evidence to support guideline development.
Marcelo Fiszman, B., Bray, B. E., Shin, D., Halil Kilicoglu, B., Bennett, G. C., Bodenreider, O., & Rindflesch, T. C. (2010) · PubMed · Cited by 14
Semantic processing to support clinical guideline development.
Marcelo Fiszman, B., Ortíz, E., Bray, B. E., & Rindflesch, T. C. (2008) · PubMed Central · Cited by 13
Performance of SVM and Bayesian classifiers on the systematic review classification task
Matwin, S., Kouznetsov, A., Inkpen, D. Z., Frunza, O., & O’Blenis, P. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.009555 · Cited by 13
Text classification on imbalanced data: Application to systematic reviews automation
Ma, Y. (2007) · uO Research (University of Ottawa) · doi:10.20381/ruor-12126 · Cited by 10
Facilitating biomedical systematic reviews using text classification and ranked retrieval
Martínez, D., Karimi, S., Cavedon, L., & Baldwin, T. (2008) · RMIT Research Repository (RMIT University Library) · Cited by 7
Kouznetsov, A., & Japkowicz, N. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13059-5_33 · Cited by 6
Determining the effectiveness of three software evaluation techniques through informal aggregation
Olorisade, B. K., Vegas, S., & Juristo, N. (2013) · Information and Software Technology · doi:10.1016/j.infsof.2013.02.015 · Cited by 6
Self-citation
Research works printed but not linked: 5
- Cohen , A. M. , Ambert , K. and McDonagh , M. 2010 . A Prospective Evaluation of an Automated Classification System to Support Evidence-based Medicine and Systematic Review . AMIA Annual Symposium Proceedings (2010), 121--125. Cohen, A. M., Ambert, K. and McDonagh, M. 2010. A Prospective Evaluation of an Automated Classification System to Support Evidence-based Medicine and Systematic Review. AMIA Annual Symposium Proceedings (2010), 121--125.
- 10.1109/ESEM.2011.16 · DOI
- Handbook for Systematic Reviews of Interventions : 2015 . http://handbook.cochrane.org/. Accessed: 2015-09-10. Handbook for Systematic Reviews of Interventions: 2015. http://handbook.cochrane.org/. Accessed: 2015-09-10.
- Jonnalagadda S. Petitti D. and Manuscript A. 2013. A New Iterative Method to Reduce Workload in the Systematic Review Process. International journal of computational biology and drug design. 6 0 (Feb. 2013) 5--17. Jonnalagadda S. Petitti D. and Manuscript A. 2013. A New Iterative Method to Reduce Workload in the Systematic Review Process. International journal of computational biology and drug design . 6 0 (Feb. 2013) 5--17. · DOI · Paper page
- Olorisade B. K. 2012. Informal Aggregation Technique for Software Engineering Experiments. IJCSI International Journal of Computer Science Issues. 9 (2012) 199--204. Olorisade B. K. 2012. Informal Aggregation Technique for Software Engineering Experiments. IJCSI International Journal of Computer Science Issues . 9 (2012) 199--204.
Cited by
Automation of systematic literature reviews: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106589 · Cited by 432
Artificial intelligence to automate the systematic review of scientific literature
de la Torre-López, J., Ramírez, A., & Romero, J. R. (2023) · Computing · doi:10.1007/s00607-023-01181-x · Cited by 170
Gates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1222-2 · Cited by 133
Cierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01805-4 · Cited by 85
“These algorithms are trained according to the computer–human interaction where the availability of a training data set and the purpose of the classification strategy are the key points [41, 49, 51, 55].”
BackgroundMethodFinding better active learners for faster literature reviews
Yu, Z., Kraft, N. A., & Menzies, T. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9587-0 · Cited by 83
“pe that other researchers will explore similar work on different datasets). Existing active learning techniques in citation screening have been criticized by Olorisade et al. for being not replicable [43,44]. To this end, we have published all our code at https://github.com/fastread/src and all our data at https://doi…”
BackgroundInfluential
Metadata
- Authors
- Olorisade, B. K., de Quincey, E., Brereton, P., & Andras, P.
- Affiliations
- Keele University (GB)
- Keele University
- Venue
- Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering
- Publisher
- ACM
- Published
- 2016-06
- Pages
- 1-11
- OpenAlex
- W2406978180
- Cited by
- 42
- References
- 78
- Citations by year
- 2026: 2 · 2025: 1 · 2024: 4 · 2023: 6 · 2022: 3 · 2021: 4 · 2020: 6 · 2019: 6 · 2018: 6 · 2017: 4
- FWCI
- 1.92
- Type
- journal article
- Language
- en
- Open access
- green
- eprints.keele.ac.uk
- Locations
- International Conference on Evaluation and Assessment in Software Engineering (EASE) (publishedVersion)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- Research Output (Edinburgh Napier University) (submittedVersion)
- Edinburgh Napier Research Repository (Edinburgh Napier University) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- text mining, citation screening, systematic reviews, software engineering literature
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.