CoChat
Sign Up
PreprintCited by 27Meta-analysis and systematic reviewsSoftware Engineering ResearchInformation Retrieval and Search Behavior

Assessing the Ability of ChatGPT to Screen Articles for Systematic Reviews

Open access
Syriani, Eugene, David, Istvan, Kumar, Gauransh·2023·arXiv
DOI 10.48550/arxiv.2307.06464arXiv 2307.06464

Preprint — published as 10.1016/j.cola.2024.101287 · Paper page

Open this paper in CoChat — free accountOpen PDF

Abstract

By organizing knowledge within a research field, Systematic Reviews (SR) provide valuable leads to steer research. Evidence suggests that SRs have become first-class artifacts in software engineering. However, the tedious manual effort associated with the screening phase of SRs renders these studies a costly and error-prone endeavor. While screening has traditionally been considered not amenable to automation, the advent of generative AI-driven chatbots, backed with large language models is set to disrupt the field. In this report, we propose an approach to leverage these novel technological developments for automating the screening of SRs. We assess the consistency, classification performance, and generalizability of ChatGPT in screening articles for SRs and compare these figures with those of traditional classifiers used in SR automation. Our results indicate that ChatGPT is a viable option to automate the SR processes, but requires careful considerations from developers when integrating ChatGPT into their SR tools.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.06464

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 59

  1. Ahmed Al-Zubidy, Jeffrey C Carver, David P Hale, and Edgar E Hassler. 2017. Vision for SLR tooling infrastructure: Prioritizing value-added requirements. Information and Software Technology 91 (2017), 72-81. https://doi.org/10.1016/j.infsof.2017.06.007 · DOI · Paper page
  2. Amal Alharbi, William Briggs, and Mark Stevenson. 2018. Retrieving and Ranking Studies for Systematic Reviews: University of Sheffield's Approach to CLEF eHealth 2018 Task 2. In Working Notes of CLEF 2018 -Conference and Labs of the Evaluation Forum, Vol. 2125. CEUR-WS.org.
  3. Amarjeet and Jitender Kumar Chhabra. 2018. FP-ABC: Fuzzy-Pareto dominance driven artificial bee colony algorithm for many-objective software module clustering. Computer Languages, Systems & Structures 51 (2018), 1-21. https://doi.org/10.1016/j.cl.2017.08.001 8 https://crfm.stanford.edu/2023/03/13/alpaca.html 9 https://github.com/tloen/alpaca-lora 10 https://www.databricks.com/blog/2023/03/24/hello-dolly-democratizing-magic-chatgpt-open-models.html · DOI
  4. Ankica Barišić, Ivan Ruchkin, Dušan Savić, Mustafa Abshir Mohamed, Rima Al-Ali, Letitia W. Li, Hana Mkaouar, Raheleh Eslampanah, Moharram Challenger, Dominique Blouin, Oksana Nikiforova, and Antonio Cicchetti. 2022. Multi-paradigm modeling for cyber-physical systems: A systematic mapping review. J Syst Softw 183 (2022), 111081. https://doi.org/10.1016/j.jss.2021.111081 · DOI
  5. Angela Barriga, Rogardt Heldal, Adrian Rutle, and Ludovico Iovino. 2022. PARMOREL: a framework for customizable model repair. Software and Systems Modeling 21, 5 (Oct 2022), 1739-1762. https://doi.org/10.1007/s10270-022-01005-0 · DOI
  6. Mohamed Bekkar, Hassiba Kheliouane Djemaa, and Taklit Akrouf Alitouche. 2013. Evaluation Measures for Models Assessment over Imbalanced Data Sets. Journal of Information Engineering and Applications 13, 10 (2013), 27-38.
  7. James Bergstra and Yoshua Bengio. 2012. Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research 13, 10 (2012), 281-305.
  8. Brice Bigendako and Eugene Syriani. 2018. Modeling a Tool for Conducting Systematic Reviews Iteratively. In Proceedings of the 6th International Conference on Model-Driven Engineering and Software Development -MODELSWARD,. INSTICC, SciTePress, 552-559. https://doi.org/10.5220/ 0006664405520559 · DOI
  9. Som S. Biswas. 2023. Potential Use of Chat GPT in Global Warming. Annals of Biomedical Engineering (Mar 2023). https://doi.org/10.1007/s10439- 023-03171-8 · DOI
  10. Som S. Biswas. 2023. Role of Chat GPT in Public Health. Annals of Biomedical Engineering 51, 5 (May 2023), 868-869. https://doi.org/10.1007/s10439- 023-03172-7 · DOI · Paper page
  11. Rohit Borah, Andrew W Brown, Patrice L Capers, and Kathryn A Kaiser. 2017. Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry. BMJ Open 7, 2 (2017). https://doi.org/10.1136/bmjopen-2016-012545 · DOI · Paper page
  12. Léa Brunschwig, Esther Guerra, and Juan de Lara. 2022. Modelling on mobile devices. Software and Systems Modeling 21, 1 (Feb 2022), 179-205. https://doi.org/10.1007/s10270-021-00897-8 · DOI
  13. Nitesh V Chawla, David A Cieslak, Lawrence O Hall, and Ajay Joshi. 2008. Automatically countering imbalance and its empirical relationship to cost. Data Mining and Knowledge Discovery 17 (2008), 225-252. https://doi.org/10.1007/s10618-008-0087-0 · DOI
  14. Davide Chicco and Guiseppe Jurman. 2023. The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification. BioData Mining 16, 4 (2023). https://doi.org/10.1186/s13040-023-00322-4 · DOI
  15. A. M. Cohen, W. R. Hersh, K. Peterson, and Po-Yin Yen. 2006. Reducing Workload in Systematic Review Preparation Using Automated Citation Classification. Journal of the American Medical Informatics Association 13, 2 (2006), 206-219. https://doi.org/doi.org/10.1197/jamia.M1929 · DOI · Paper page
  16. Istvan David, Kousar Aslam, Sogol Faridmoayer, Ivano Malavolta, Eugene Syriani, and Patricia Lago. 2021. Collaborative Model-Driven Software Engineering: A Systematic Update. In 2021 ACM/IEEE 24th International Conference on Model Driven Engineering Languages and Systems (MODELS). 273-284. https://doi.org/10.1109/MODELS50736.2021.00035 · DOI
  17. Katia R. Felizardo and Jeffrey C. Carver. 2020. Automating Systematic Literature Review. Springer International Publishing, Cham, 327-355. https://doi.org/10.1007/978-3-030-32489-6_12 · DOI
  18. Gerbrich Ferdinands, Raoul Schram, Jonathan de Bruin, Ayoub Bagheri, Daniel L Oberski, Lars Tummers, and Rens van de Schoot. 2020. Active learning for screening prioritization in systematic reviews -A simulation study. (Sep 2020). https://doi.org/10.31219/osf.io/w6qbg OSF Preprints. · DOI · Paper page
  19. Dalibor Fiala and Gabriel Tutoky. 2017. Computer Science Papers in Web of Science: A Bibliometric Analysis. Publications 5, 4 (2017). https: //doi.org/10.3390/publications5040023 · DOI
  20. Joseph L Fleiss, Bruce Levin, and Myunghee Cho Paik. 2003. Statistical methods for rates and proportions (3 ed.). Wiley-Interscience. https: //doi.org/10.1002/0471445428 · DOI · Paper page
  21. Luciano Floridi and Massimo Chiriatti. 2020. GPT-3: Its Nature, Scope, Limits, and Consequences. Minds and Machines 30, 4 (2020), 681-694.
  22. Rebecca Ganann, Donna Ciliska, and Helen Thomas. 2010. Expediting systematic reviews: methods and implications of rapid reviews. Implementation Science 5, 1 (Jul 2010), 56. https://doi.org/10.1186/1748-5908-5-56 · DOI · Paper page
  23. Md Zahidul Islam, Jixue Liu, Jiuyong Li, Lin Liu, and Wei Kang. 2019. A Semantics Aware Random Forest for Text Classification. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM '19). Association for Computing Machinery, New York, NY, USA, 1061-1070. https://doi.org/10.1145/3357384.3357891 · DOI
  24. Xiaonan Ji, Alan Ritter, and Po-Yin Yen. 2017. Using ontology-based semantic similarity to facilitate the article screening process for systematic reviews. J of Biomedical Informatics 69 (2017), 33-42. https://doi.org/10.1016/j.jbi.2017.03.007 · DOI
  25. Thorsten Joachims. 1998. Text categorization with Support Vector Machines: Learning with many relevant features. In Machine Learning: ECML-98 (LNAI, Vol. 1398). Springer, 137-142. https://doi.org/10.1007/BFb0026683 · DOI
  26. Siddhartha R Jonnalagadda, Pawan Goyal, and Mark D Huffman. 2015. Automating data extraction in systematic reviews: a systematic review. Systematic Reviews 4, 1 (2015), 78.
  27. Madian Khabsa, Ahmed Elmagarmid, Ihab Ilyas, Hossam Hammady, and Mourad Ouzzani. 2016. Learning to identify relevant studies for systematic reviews using random forest and external information. Machine Learning 102 (2016), 465-482. https://doi.org/10.1007/s10994-015-5535-7 · DOI · Paper page
  28. B.A. Kitchenham, T. Dyba, and M. Jorgensen. 2004. Evidence-based software engineering. In Proceedings. 26th International Conference on Software Engineering. 273-281. https://doi.org/10.1109/ICSE.2004.1317449 · DOI
  29. Tomaz Kosar, Sudev Bohra, and Marjan Mernik. 2018. A Systematic Mapping Study driven by the margin of error. Journal of Systems and Software 144 (2018), 439-449. https://doi.org/10.1016/j.jss.2018.06.078 · DOI
  30. Wojciech Kusa, Aldo Lipani, Petr Knoth, and Allan Hanbury. 2023. An analysis of work saved over sampling in the evaluation of automated citation screening in systematic literature reviews. Intelligent Systems with Applications 18, 200193 (2023). https://doi.org/10.1016/j.iswa.2023.200193 · DOI · Paper page
  31. Michael P. LaValley. 2008. Logistic Regression. Circulation 117, 18 (2008), 2395-2399. https://doi.org/10.1161/CIRCULATIONAHA.106.682658 · DOI
  32. Luigi Lavazza, Sandro Morasca, and Gabriele Rotoloni. 2023. On the Reliability of the Area Under the ROC Curve in Empirical Software Engineering. In International Conference on Evaluation and Assessment in Software Engineering. ACM, 93-100. https://doi.org/10.1145/3593434.3593456 · DOI
  33. Quoc Le and Tomas Mikolov. 2014. Distributed Representations of Sentences and Documents. In International Conference on Machine Learning, Vol. 32. PMLR, 1188-1196.
  34. Percy Liang et al. 2022. Holistic Evaluation of Language Models. arXiv:2211.09110 [cs.CL] · arXiv
  35. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-Train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. Comput. Surveys 55, 9 (2023), 1-35. https://doi.org/10.1145/3560815 · DOI · Paper page
  36. Richard Mallett, Jessica Hagen-Zanker, Rachel Slater, and Maren Duvendack. 2012. The benefits and challenges of using systematic reviews in international development research. Journal of Development Effectiveness 4, 3 (2012), 445-455. https://doi.org/10.1080/19439342.2012.711342 · DOI · Paper page
  37. Iain Marshall and Byron Wallace. 2019. Toward systematic review automation: a practical guide to using machine learning tools in research synthesis. Systematic reviews 8, 163 (2019), 1-10. https://doi.org/10.1186/s13643-019-1074-9 · DOI · Paper page
  38. David Martinez, Sarvnaz Karimi, Lawrence Cavedon, and Timothy Baldwin. 2008. Facilitating biomedical systematic reviews using ranked text retrieval and classification. In Australasian Document Computing Symposium. ADCS, 53-60. https://doi.org/2008/proceedings/p09-martinez.pdf
  39. Stan Matwin, Alexandre Kouznetsov, Diana Inkpen, Oana Frunza, and Peter O'Blenis. 2010. A new algorithm for reducing the workload of experts in performing systematic reviews. Journal of the American Medical Informatics Association 17, 4 (2010), 446-453. https://doi.org/10.1136/jamia.2010. 004325 · DOI · Paper page
  40. Robert W. McGee. 2023. What Will the United States Look Like in 2050? A Chatgpt Short Story. SSRN Electronic Journal (2023). https: //doi.org/10.2139/ssrn.4413442 · DOI
  41. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems, Vol. 26. Curran Associates, Inc.
  42. Makoto Miwa, James Thomas, Alison O'Mara-Eves, and Sophia Ananiadou. 2014. Reducing systematic review workload through certainty-based screening. Journal of Biomedical Informatics 51 (2014), 242-253. https://doi.org/10.1016/j.jbi.2014.06.005 · DOI · Paper page
  43. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 85 (2011), 2825-2830.
  44. Paul Ralph and Sebastian Baltes. 2022. Paving the Way for Mature Secondary Research: The Seven Types of Literature Review. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2022). Association for Computing Machinery, New York, NY, USA, 1632-1636. https://doi.org/10.1145/3540250.3560877 · DOI
  45. Paul et al. Ralph. 2021. Empirical Standards for Software Engineering Research. Technical Report 2010.03525. arXiv. https://doi.org/EmpiricalStandards/ docs/?standard=DataScience
  46. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. CoRR abs/1908.10084 (2019). · arXiv
  47. Jason D. M. Rennie, Lawrence Shih, Jaime Teevan, and David R. Karger. 2003. Tackling the Poor Assumptions of Naive Bayes Text Classifiers. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning (ICML'03). AAAI Press, 616-623.
  48. Igor Rozanc and Marjan Mernik. 2021. Chapter Three -The screening phase in systematic reviews: Can we speed up the process? In Advances in Computers. Vol. 123. Elsevier, 115-191. https://doi.org/10.1016/bs.adcom.2021.01.006 · DOI
  49. Marina Sokolova, Nathalie Japkowicz, and Stan Szpakowicz. 2006. Beyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation. In AI 2006: Advances in Artificial Intelligence. Springer Berlin Heidelberg, Berlin, Heidelberg, 1015-1021.
  50. Guy Tsafnat, Paul Glasziou, Miew Keen Choong, Adam Dunn, Filippo Galgani, and Enrico Coiera. 2014. Systematic review automation technologies. Systematic Reviews 3, 1 (2014), 74.
  51. Rens van de Schoot, Jonathan De Bruin, Raoul Schram, Parisa Zahedi, Jan De Boer, Felix Weijdema, Bianca Kramer, Martijn Huijts, Maarten Hoogerwerf, Gerbrich Ferdinands, Albert Harkema, Joukje Willemsen, Yongchao Ma, Qixiang Fang, Sybren Hindriks, Lars Tummers, and Daniel L Oberski. 2021. An open source machine learning framework for efficient and transparent systematic reviews. Nature machine intelligence 3, 2 (2021), 125-133. https://doi.org/10.1038/s42256-020-00287-7 · DOI · Paper page
  52. Raymon van Dinter, Bedir Tekinerdogan, and Cagatay Catal. 2021. Automation of systematic literature reviews: A systematic literature review. Information and Software Technology 136 (2021). https://doi.org/10.1016/j.infsof.2021.106589 · DOI · Paper page
  53. Byron C. Wallace, Kevin Small, Carla E. Brodley, Joseph Lau, and Thomas A. Trikalinos. 2012. Deploying an Interactive Machine Learning System in an Evidence-Based Practice Center: Abstrackr. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium (IHI'12). ACM, 819-824. https://doi.org/10.1145/2110363.2110464 · DOI · Paper page
  54. Byron C Wallace, Thomas A Trikalinos, Joseph Lau, Carla Brodley, and Christopher H Schmid. 2010. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinformatics 11, 55 (2010). https://doi.org/10.1186/1471-2105-11-55 · DOI · Paper page
  55. Shuai Wang, Harrisen Scells, Bevan Koopman, and Guido Zuccon. 2023. Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search? Technical Report 2302.03495. arXiv. https://doi.org/abs/2302.03495
  56. Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. 2020. Generalizing from a Few Examples: A Survey on Few-Shot Learning. ACM Comput. Surv. 53, 3, Article 63 (jun 2020), 34 pages. https://doi.org/10.1145/3386252 · DOI · Paper page
  57. Muhammad Waseem, Aakash Ahmad, Peng Liang, Mahdi Fehmideh, Pekka Abrahamsson, and Tommi Mikkonen. 2023. Conducting System- atic Literature Reviews with ChatGPT. (mar 2023). https://doi.org/publication/369062219_Conducting_Systematic_Literature_Reviews_with_ ChatGPT_ChatGPT_for_SLRs_A_Proposal https://www.researchgate.net/publication/369062219_Conducting_Systematic_Literature_Reviews_ with_ChatGPT_ChatGPT_for_SLRs_A_Proposal.
  58. Shuo Xu, Yan Li, and Zheng Wang. 2017. Bayesian Multinomial Naïve Bayes Classifier to Text Classification. In Advanced Multimedia and Ubiquitous Engineering. Springer Singapore, Singapore, 347-352.
  59. Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, Hao Peng, Jianxin Li, Jia Wu, Ziwei Liu, Pengtao Xie, Caiming Xiong, Jian Pei, Philip S. Yu, and Lichao Sun. 2023. A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT. Technical Report 2302.09419. arXiv. https://doi.org/abs/2302.09419

Cited by

Sorted by citations

Authors
Syriani, E., David, I., & Kumar, G.
ORCID
Syriani, Eugene
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2307.06464
arXiv
2307.06464
OpenAlex
W4384389814
Cited by
27
References
0
Citations by year
2026: 1 · 2025: 10 · 2024: 16
FWCI
1.68
Type
journal article
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
ChatGPT, large language models, systematic reviews, article screening, generative AI, software engineering literature, classifier comparison, generalizability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review