The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
ChatGPT and Software Testing Education: Promises & Perils
Open accessAbstract
Over the past decade, predictive language modeling for code has proven to be a valuable tool for enabling new forms of automation for developers. More recently, we have seen the ad-vent of general purpose "large language models", based on neural transformer architectures, that have been trained on massive datasets of human written text, which includes code and natural language. However, despite the demonstrated representational power of such models, interacting with them has historically been constrained to specific task settings, limiting their general applicability. Many of these limitations were recently overcome with the introduction of ChatGPT, a language model created by OpenAI and trained to operate as a conversational agent, enabling it to answer questions and respond to a wide variety of commands from end users.The introduction of models, such as ChatGPT, has already spurred fervent discussion from educators, ranging from fear that students could use these AI tools to circumvent learning, to excitement about the new types of learning opportunities that they might unlock. However, given the nascent nature of these tools, we currently lack fundamental knowledge related to how well they perform in different educational settings, and the potential promise (or danger) that they might pose to traditional forms of instruction. As such, in this paper, we examine how well ChatGPT performs when tasked with answering common questions in a popular software testing curriculum. We found that given its current capabilities, ChatGPT is able to respond to 77.5% of the questions we examined and that, of these questions, it is able to provide correct or partially correct answers in 55.6% of cases, provide correct or partially correct explanations of answers in 53.0% of cases, and that prompting the tool in a shared question context leads to a marginally higher rate of correct answers and explanations. Based on these findings, we discuss the potential promises and perils related to the use of ChatGPT by students and instructors.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In this method, researchers evaluate the agents by letting them conduct tasks such as software testing tasks, such as generating test cases, reproducing bugs, debugging code, and interacting with developers and external tools [166,168,169,170].Then, one can use metrics like test coverage and bug detection rate to measure the effectiveness of LLM-based agents.”
From A survey on large language model based autonomous agents (2024) · cites this paper as [169]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“In parallel, the machine learning and natural language processing communities began building large-scale models centered around a specific type of neural architecture, the transformer [3]–[5], trained on massive datasets of text.”
BackgroundJakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 2×
“To enable this type of interaction, OpenAI made use of reinforcement learning from human feedback, refining methods from past work on InstructGPT [11], which trained LLMs with both unsupervised data and with supervision in the form of task instruction.”
MethodKung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 2×
“ChatGPT produces responses that are at or near the passing threshold for all three parts of the US Medical Licensing Exam without any additional information or prompt engineering [26].”
BackgroundResultTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2×
“To enable this type of interaction, OpenAI made use of reinforcement learning from human feedback, refining methods from past work on InstructGPT [11], which trained LLMs with both unsupervised data and with supervision in the form of task instruction.”
MethodCodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
Research works printed but not linked: 1
- Puryear, 2022, Github CoPilot in the Classroom: Learning to Code with AI Assistance, CCSC
Cited by
Sorted by citations
Items 101–200 of 275
Tanay, B. A., Arinze, L., Joshi, S. S., Davis, K., & Davis, J. C. (2024) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--46557 · Cited by 12
Guidelines for Effective Use of ChatGPT in Introductory Programming Education
Ramabu, T. J., & Malebane, T. (2024) · IST-Africa Conference (IST-Africa) · doi:10.23919/ist-africa63983.2024.10569684 · Cited by 11
“The work in Jalil et al. [13] has tested ChatGPT’s abilities in aiding software testing.”
BackgroundTransformative landscape of ChatGPT in higher education: unveiling the academic discourse
Shrivastava, A. (2024) · Global Knowledge Memory and Communication · doi:10.1108/gkmc-02-2024-0068 · Cited by 9
Hussain, F., & Anwar, M. A. (2024) · Journal of Asian Public Policy · doi:10.1080/17516234.2024.2422917 · Cited by 12
Tang, L., Bao, L., Xia, X., & Huang, Z. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00037 · Cited by 11
“Therefore, we believe that N EURAL SZZ is a more effective method for identifying the root cause in the bug-fixing commits than the baselines.”
MethodUsing ChatGPT-4 to Grade Open Question Exams
Alers, H., Malinowska, A., Meghoe, G., & Apfel, E. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-53960-2_1 · Cited by 10
Navigating (in)Security of AI-Generated Code
Ambati, S. H., Ridley, N., Branca, E., & Stakhanova, N. (2024) · IEEE International Conference on Cyber Security and Resilience (CSR) · doi:10.1109/csr61664.2024.10679468 · Cited by 12
Wang, Q., & Li, H. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9030050 · Cited by 13
“In addition, LLMs have been widely applied in other subject domains, e.g., software engineering [43], law [44], and finance [45].”
Zhu, M. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70013 · Cited by 12
Kohn, T. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70043 · Cited by 14
Chatgpt With Risks And Opportunities
Kabak, T., & Kırbaş, İ. (2023) · Mehmet Akif Ersoy Üniversitesi Fen Bilimleri Enstitüsü Dergisi · doi:10.29048/makufebed.1271477 · Cited by 10
“When the characteristics of the wrong answers are analyzed; • Lack of information • Faulty assumption • Lack of information and faulty assumption together three main reasons stand out (Jalil et al., 2023).”
Lukens, S., & Ali, A. (2023) · Annual Conference of the PHM Society · doi:10.36001/phmconf.2023.v15i1.3487 · Cited by 9
“In (Jalil, Rafi, LaToza, Moran, & Lam, 2023), Chat-GPT’s ability to answer questions related to software testing was evaluated using a popular textbook.”
MethodNeural Network-Based Test Case Prioritization in Continuous Integration
Veșcan, A., Găceanu, R., & Szederjesi-Dragomir, A. (2023) · IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew60602.2023.00014 · Cited by 9
“…is presented in Table I. Additionally, in order to structure the information around a test case and not around cycle or execution , we extracted and reorganized the information provided in the three datasets, thus characterizing each test case with execution cycles, fails versus passes, etc.”
MethodRequirements Verification Through the Analysis of Source Code by Large Language Models
Couder, J. O., Gomez, D., & Ochoa, O. (2024) · IEEE SoutheastCon · doi:10.1109/southeastcon52093.2024.10500073 · Cited by 11
“For example, researchers are developing new LLM training methods that improve the accuracy of LLMs on code verification tasks [17].”
MethodTeaching IT Software Fundamentals: Strategies and Techniques for Inclusion of Large Language Models
Gumina, S., Dalton, T., & Gerdes, J. H. (2023) · Conference on Information Technology Education · doi:10.1145/3585059.3611409 · Cited by 9
“Lack of contextual understanding within AI models may lead to generated responses that do not meet the assignment requirements [6, 17].”
BackgroundMethodInfluentialA review study of ChatGPT applications in education
Zaabi, M., Hariri, W., & Smaoui, N. (2023) · International Conference on Innovations in Intelligent Systems and Applications (INISTA) · doi:10.1109/inista59065.2023.10310439 · Cited by 10
“The authors in [14] looked at how well ChatGPT does when asked to respond to sample questions from a software testing course.”
BackgroundA Language Model of Java Methods with Train/Test Deduplication
Su, C., Bansal, A., Jain, V., Ghanavati, S., & McMillan, C. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3613090 · Cited by 15
“Today, language models consisting of billions of parameters and trillions of tokens of training data are the heart of several high-profile products such as ChatGPT and Copilot [19].”
BackgroundLarge Language Model vs. Stack Overflow in Addressing Android Permission Related Challenges
Oishwee, S. J., Stakhanova, N., & Codabux, Z. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644933 · Cited by 9
“Potentially, ChatGPT performance can be improved by additional context, as was shown by Jalil et al. [25].”
BackgroundMethodInfluentialThe ChatGPT revolution: Reshaping education through artificial intelligence
Kaur, K., Shilpa, & Jindal, R. (2024) · AIP Conference Proceedings · doi:10.1063/5.0221430 · Cited by 7
Khan, S. A., Oshin, N. T., Nizam, M., Ahmed, I., Musfique, M. M., & Hasan, M. (2024) · Lecture notes in networks and systems · doi:10.1007/978-981-99-8346-9_28 · Cited by 8
Generative AI for Software Test Modelling with a focus on ERP Software
Garg, A., & Sharma, D. (2023) · International Conference on Advances in Computation, Communication and Information Technology (ICAICCIT) · doi:10.1109/icaiccit60255.2023.10466102 · Cited by 9
“By responding to inquiries, offering direction, and presenting relevant details during the testing procedure, they may also give real-time assistance and support to testers[11].”
BackgroundInfluentialEvaluation of ChatGPT Usage in Preschool Education: Teacher Perspectives
Uğraş, M. (2024) · Eğitim Ve İnsani Bilimler Dergisi Teori Ve Uygulama · doi:10.58689/eibd.1537337 · Cited by 17
Li, Z., Wu, J., Ling, X., Luo, T., Rui, Z., & Wu, Y. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00132 · Cited by 14
Uğraş, M., Çakır, Z., Zacharis, G., & Kalogiannakis, M. (2025) · Computers · doi:10.3390/computers14090368 · Cited by 12
“In line with these concerns, other studies have indicated that ChatGPT can generate false or misleading content, raising ongoing doubts about its accuracy and reliability [103,104].”
Cautious Optimism: The Influence of Generative AI Tools in Software Development Projects
Mbizo, T., Oosterwyk, G., Tsibolane, P., & Kautondokwa, P. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-64881-6_21 · Cited by 15
Almeida, D., & Barr, N. (2025) · IGI Global eBooks · doi:10.4018/979-8-3693-7630-0.ch007 · Cited by 8
Perspective Chapter: Recent Trends in Deep Learning for Conversational AI
Wassan, J. T., & Ghuriani, V. (2023) · Artificial intelligence · doi:10.5772/intechopen.113250 · Cited by 11
ChatGPT as a New Frontier in Arabic Education Technology
Sulaeman, I., Syuhadak, S., & Sulaeman, I. (2023) · Al-Arabi Jurnal Bahasa Arab dan Pengajarannya = Al-Arabi Journal of Teaching Arabic as a Foreign Language · doi:10.17977/um056v7i1p83-105 · Cited by 12
Cao, S., & Zhou, T. (2025) · SAGE Open · doi:10.1177/21582440251369204 · Cited by 80
“Jalil et al. (2023).”
What Is the Impact of ChatGPT on Education?
NAGAR, L. (2024) · INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT · doi:10.55041/ijsrem32175 · Cited by 7
Liu, X., Vatn, J., Yin, S., & Maithani, V. (2023) · Annual Conference of the IEEE Industrial Electronics Society (IECON) · doi:10.1109/iecon51785.2023.10311736 · Cited by 8
“…diagnosis and treatment recommendations [9], medical education [10]), Business and Finance (market analysis and forecasting [11], fraud detection [12]), Education and Training (test preparation [13], online tutoring [14]) Programming and Code Debugging (code generation [15], code review [16])...”
BackgroundEarly ChatGPT User Portrait through the Lens of Data
Deng, Y., Zhao, N., & Huang, X. S. (2023) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata59044.2023.10386415 · Cited by 9
“…theme, researchers have assessed the use of ChatGPT in various fields, including medical care( [3], [4], [5]), education( [6], [7] [8]), reasoning [9], software development [10], translation [11], and scientific research( [12], [13]), and highlighted the feasibility of using ChatGPT in these areas.”
BackgroundUtilization of Artificial Intelligence in Education: A Perspective on Learning Strategies
Yalçınalp, S., Türkoğlu, H., Koç, S. E., & Ersoy, H. (2024) · Artificial intelligence · doi:10.5772/intechopen.1005086 · Cited by 12
Šavelka, J., Agarwal, A., Bogart, C., Song, Y., & Sakr, M. F. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.09325 · Cited by 7
Nzobonimpa, S., Savard, J., Caron, I., & Lawarée, J. (2024) · AI & Society · doi:10.1007/s00146-024-02103-x · Cited by 6
“Using software testing methodology on Chat-GPT, Jalil et al. (2023) found that while the model had been praised in the medical and legal fields, its performance in other domains, such as education, had not been as impressive and promising: We found that given its current capabilities, ChatGPT is able to respond to 77.…”
InfluentialInvestigating Developers' Preferences for Learning and Issue Resolution Resources in the ChatGPT Era
Tayeb, A. J., Alahmadi, M. D., Tajik, E., & Haiduc, S. (2024) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme58944.2024.00045 · Cited by 15
“Furthermore, studies in the field of computer science education have shown the effectiveness and implications of using ChatGPT in education, from generating code solutions to participating in assessments and personalized learning support [9, 13, 18, 19, 36, 38, 39, 46].”
The Transformative Influence of LLMs on Software Development & Developer Productivity
Jalil, S. (2025) · International Conference on Artificial Intelligence, Computer, Data Sciences and Applications (ACDSA) · doi:10.1109/acdsa65407.2025.11166110 · Cited by 15
“Furthermore, the unpredictable nature of LLMs indicates that they can produce varying outputs for identical prompts [47].”
Self-citationIndustry perceptions of the competencies needed by novice software tester
Hamid, B., & Ikram, N. (2023) · Education and Information Technologies : Official Journal of the IFIP technical committee on Education · doi:10.1007/s10639-023-12055-2 · Cited by 5
Preserving Academic Integrity in Teaching with ChatGPT: Practical Strategies
Raza, M. R., & Hussain, W. (2023) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat59888.2023.00027 · Cited by 7
Exploring the Competency of ChatGPT in Solving Competitive Programming Challenges
(2023) · International Journal of Advanced Trends in Computer Science and Engineering · doi:10.30534/ijatcse/2024/031312024 · Cited by 14
A Review ChatGPT and the Future of Education
Machado, A. D. B., Pesqueira, A. M., & Sousa, M. J. (2024) · EAI/Springer Innovations in Communication and Computing · doi:10.1007/978-3-031-52296-3_9 · Cited by 6
Zhang, H., & Li, M. (2024) · 2024 4th International Conference on Educational Technology (ICET) · doi:10.1109/icet62460.2024.10869172 · Cited by 5
ChatGPT and Other AI Tools for Academic Research and Education
Shtykalo, O., & Yamnenko, I. (2024) · Lecture notes in electrical engineering · doi:10.1007/978-3-031-61221-3_29 · Cited by 7
Quality Assurance for LLM-Generated Test Cases: A Systematic Literature Review
Edirisinghe, H., & Wickramaarachchi, D. (2024) · SLAAI International Conference on Artificial Intelligence (SLAAI-ICAI) · doi:10.1109/slaai-icai63667.2024.10844968 · Cited by 7
“However, while AI-enhanced frameworks offer benefits, heir implementation poses challenges [64].”
ChatGPT in Research and Education: A SWOT Analysis of Its Academic Impact
Miah, A. S. M., Tusher, M. M. R., Hossain, M. M., Hossain, M. M., Rahim, M. A., Hamid, M. E., Islam, M. S., & Shin, J. (2025) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2025.064168 · Cited by 11
“Research has also examined ChatGPT’s use in specific subject areas, such as Economics [51], English Language [52], Law [39], Sports Science [49], Medical Education [53–55], Higher-Order Thinking [47], Mathematics [56,57], Programming [34,58], and Software Testing [59].”
The Impact of ChatGPT on Software Engineering Education: A Quick Peek
Li, Y., Xu, J., Zhu, Y., Liu, H., & Liu, P. (2023) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa59317.2023.00087 · Cited by 5
“A N A CTUAL C ASE S TUDY Jalil et al. [9] explore the practicality of using ChatGPT to answer questions commonly encountered in a software testing curriculum.”
BackgroundAI at Doorstep: ChatGPT and Academia
Khan, S. H. (2023) · Journal of College of Physicians And Surgeons Pakistan · doi:10.29271/jcpsp.2023.10.1085 · Cited by 6
Large Language Models:Expectations for Semantics-Driven Systems EngineeringPreprint
Buchmann, R. A., Eder, J., Fill, H.-G., Frank, U., Karagiannis, D., Laurenzi, E., Mylopoulis, J., Plexousakis, D., & Santos, M. Y. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4633719 · Cited by 5
Evidências sobre o uso do ChatGPT no ensino de modelagem de software: um experimento controlado
Murad, S. B., Lemos, F. F. S., Melo, S. M., Paschoal, L. N., & Prates, J. M. (2025) · Anais do Workshop sobre Educação em Computação (WEI) · doi:10.5753/wei.2025.9261 · Cited by 4
Katz, K., Moshtari, S., Mujhid, I., Mirakhorli, M. T., & Garcia, D. (2025) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem64174.2025.00012 · Cited by 4
“Jalil et al. [15] investigated the effectiveness of ChatGPT in answering test questions, while Ahmad et al. [2] explored its application in system architecture.”
Gambo, O. O., Onifade, O. J., Olanrewaju-Smart, W., Abuya, O. J., Salawu, R. E., Massenon, R., & Gambo, I. (2026) · Discover Education · doi:10.1007/s44217-026-01142-4 · Cited by 10
“Jalil et al. [37] found more modest results in software testing education, with ChatGPT achieving only 37.5% accuracy in answering technical questions, suggesting limitations in specialized programming domains.”
InfluentialThe Game-Changing Impact of AI Chatbots on Education ChatGPT and Beyond
Vasudevan, A., Lama, A. V., & Sain, Z. H. (2024) · Journal of Information Systems and Technology Research · doi:10.55537/jistr.v3i1.770 · Cited by 4
Automation of Test Skeletons Within Test-Driven Development Projects
Ölmez, M. M., & Gehringer, E. F. (2024) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet62301.2024.10663016 · Cited by 5
“[27] is asking multiple questions in one chat until the context is destroyed so the last answer is the most consistent one.”
BackgroundInfluentialContext Matters: Investigating Its Impact on ChatGPT's Bug Fixing Performance
Qu, X., Zuo, F., Li, X., & Rhee, J. (2024) · IEEE/ACIS International Conference on Software Engineering Research, Management and Applications (SERA) · doi:10.1109/sera61261.2024.10685568 · Cited by 4
“In addition, Jalil et al. [28] investigate the effectiveness of ChatGPT in answering typical questions within a popular software testing textbook.”
Khan, M., Akbar, M. A., & Kasurinen, J. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3727967.3756831 · Cited by 12
“This fosters an environment where teamwork and knowledge sharing become more effective, as AI can facilitate discussions, provide relevant insights, and help students understand complex topics collaboratively.”
Kimya Sorularının Cevaplanmasında Yapay Zekâ Tabanlı Sohbet Robotlarının Performansının İncelenmesi
ÇELİK, A. Y., & K.ÇOBAN, Ö. (2023) · Türk Eğitim Bilimleri Dergisi · doi:10.37217/tebd.1361401 · Cited by 8
University Students’ Perspective on ChatGPT and Technology Literacies
Dawa, T., Dhendup, S., Tashi, S., & Rosso, M. A. (2024) · Educational innovation and practice · doi:10.17102/eip.7.2024.10 · Cited by 4
Tao, Y., Chen, W., Ye, Q., & Zhao, Y. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640065 · Cited by 4
“Jalil et al. reported that ChatGPT was able to answer over 55% of the questions from a widely used software testing textbook [17].”
MethodEducating Augmented Programmers
Sánchez‐Gordón, M., Tovar, E., Colomo‐Palacios, R., Piedra, N., & Castro, M. A. (2023) · Computer · doi:10.1109/mc.2023.3313325 · Cited by 4
Üniversite Öğrencilerinin Yazılı Üretimlerinin OpenAI GPT ile Değerlendirilmesi
SAYIN, A., & Melanlıoğlu, D. (2024) · Yuksekogretim Dergisi · doi:10.53478/yuksekogretim.1418870 · Cited by 3
ChatGPT na Universidade de Perugia: Potencializando a Interação entre Docentes e Discentes
Brognoli, P. C., & Rodrigues, M. B. (2025) · Encontros Bibli Revista Eletrônica de Biblioteconomia e Ciência da Informação · doi:10.5007/1518-2924.2025.e102135 · Cited by 3
Intention to use ChatGPT among law educators in Saudi Arabia
Sarabdeen, J. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1631413 · Cited by 4
Bassner, P., Lottner, A., & Krusche, S. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770025 · Cited by 8
Oran, A. C., Montenegro, L. B., Schuster, H. A., Duarte, J. C., Silva, W., & Lima, R. R. (2024) · Brazilian Symposium on Software Quality (SBQS) · doi:10.1145/3701625.3701684 · Cited by 8
“We also found studies that investigated the impacts of using ChatGPT in other disciplines within the software engineering course, such as Testing [21, 53], Data Structures or Coding [26, 28], and Systems Analysis [6].”
Can ChatGPT Solve Undergraduate Exams from Warehousing Studies? An Investigation
Franke, S., Pott, C., Rutinowski, J., Pauly, M., Reining, C., & Kirchheim, A. (2025) · Computers · doi:10.3390/computers14020052 · Cited by 2
Vito, G. D., D’Amici, D., Izzo, F., Ferrucci, F., & Nucci, D. D. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00024 · Cited by 5
“5 [27] and GPT-4 [28], which are decoder-only models, that have shown the most promise in this field [5], [6].”
A Systematic Review of Research on ChatGPT
Guan, C., Ding, D., Gupta, P., Hung, Y.-C., & Jiang, Z. (2023) · Advances in digital crime, forensics, and cyber terrorism book series · doi:10.4018/978-1-6684-8422-7.ch007 · Cited by 3
Wicaksono, A. W., & Sembiring, D. (2023) · SKYHAWK Jurnal Aviasi Indonesia · doi:10.52074/skyhawk.v3i2.135 · Cited by 3
Analysis and Improvement of Engineering Exams Toward Competence Orientation by Using an AI Chatbot
Fuhrmann, T., & Niemetz, M. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-51979-6_42 · Cited by 3
Pesqueira, A. M., Sousa, M. J., Machado, A. D. B., Bolog, S., Vieira, L., & Bolog, I. (2024) · Advances in business information systems and analytics book series · doi:10.4018/979-8-3693-1210-0.ch008 · Cited by 3
Exploring the Profile of University Assessments Flagged as Containing AI-Generated Material
Gooch, D., Waugh, K., Richards, M., Slaymaker, M., & Woodthorpe, J. (2024) · ACM Inroads · doi:10.1145/3656478 · Cited by 3
“Jalil et al. [11] took 31 questions from five chapters of a standard software testing textbook and generated three answers to each question using ChatGPT, also examining whether the answers were better if they were produced in a single thread, or shared context.”
MethodHuman Evaluation of GPT for Scalable Python Programming Exercise Generation
Khan, M. F. A., Ramsdell, M., Nguyen, H., & Karimi, H. R. (2024) · IEEE International Conference on Data Science and Advanced Analytics (DSAA) · doi:10.1109/dsaa61799.2024.10722841 · Cited by 3
“Its widespread application promises significant enhancements in teaching methodologies and learning experiences [11], [27].”
BackgroundRethinking the Future of Mental Health
Pesqueira, A. M., & Almeida, D. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-0659-9.ch012 · Cited by 3
AI as a Co-Engineer: A Case Study of ChatGPT in Software LifecyclePreprint
Djaber, R., & Ismail, H. (2023) · Research Square · doi:10.21203/rs.3.rs-3809973/v1 · Cited by 3
Incorporating ChatGPT Into Online Discussions in a Literacy Course
Hu, R., & Lin, X. Y. (2025) · International Journal of Artificial Intelligence · doi:10.4018/ijaitl.366589 · Cited by 2
Extracting Norms from Contracts Via ChatGPT
Haque, A., & Singh, M. P. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-82039-7_8 · Cited by 6
Leveraging AI and Blockchain for Privacy and Security in Smart Medical Systems
Pesqueira, A. M., Barr, N., & Almeida, D. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-0593-6.ch011 · Cited by 2
Automatic Unit Test Generation for Programming Assignments Using Large Language Models
Zheng, K., Shen, Y., & Tao, Y. (2025) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet66350.2025.00031 · Cited by 2
“In particular, researchers have proposed LLM-based AI tutoring for learning activities such as program comprehension [21], software testing [22], writing UML diagrams [8], and coding or debugging [6] [7] [9] [23].”
The Influence of ChatGPT’s Questions and Answers on Student Learning in Software Design Patterns
Gerede, Ç. E. (2025) · International Conference on Computer Science and Engineering (UBMK) · doi:10.1109/ubmk67458.2025.11206917 · Cited by 2
Extending Test-driven development to Softwarized Networks and Intent Based Networking
Berardi, D., Fontana, M., & Martini, B. (2025) · International Conference on Network and Service Management (CNSM) · doi:10.23919/cnsm67658.2025.11297527 · Cited by 2
“The topic of Test-Driven-Development is one of the most important in modern Software Engineering [1].”
Evaluating the Accuracy and Stability of Frontier LLMs on ENADE Computer Science Questions
Carvalho, L. D. M., Carvalho, C. M., & Mendonça, N. C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-15993-9_18 · Cited by 2
Pesqueira, A. M., Cucul, C., Egelhof, T., Fuchs, S., Tang, L., Sofía, N., & Machado, A. A. (2026) · Systems · doi:10.3390/systems14040414 · Cited by 2
Deep research and analysis of ChatGPT based on multiple testing experiments
Wei, S., Luo, Y., Chen, S., Huang, T., & Xiang, Y. (2023) · International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC) · doi:10.1109/cyberc58899.2023.00030 · Cited by 3
“In addition, in the context of shared questions, ChatGPT provides slightly higher answer and explanation rates [7] .”
BackgroundDigital Rubber Duck: Leveraging Large Language Models for Extreme Programming
Elvira, T., Procko, T., Couder, J. O., & Ochoa, O. (2023) · Congress in Computer Science, Computer Engineering, and Applied Computing (CSCE) · doi:10.1109/csce60160.2023.00051 · Cited by 5
“Moreover, the authors being up the topic of ChatGPT in other domains where it has been proven to be successful, passing medicine and law exams yet, it struggles to answer more elaborate questions which don’t have a clear answer [46].”
BackgroundMethodLast Tendencies in Acquiring Text Competence in the Field of Tourism. The Case of Chatbots and AI
Madueño, M. D. F. D. L. T. (2024) · Springer proceedings in business and economics · doi:10.1007/978-3-031-52607-7_17 · Cited by 2
Code Comprehension Problems in Introductory Programming to Overcome ChatGPT
Rebollido, F. Y., & Caro, J. D. L. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-73344-4_50 · Cited by 4
The Emergency of Chatbots and Student Learning Dilemma: A Comprehensive Review
Mudhombo, I. (2024) · Springer eBooks · doi:10.1007/978-3-031-69954-2_10 · Cited by 2
GPT AI in Computer Science Education: A Systematic Mapping Study
Strik, B., Menolli, A., & Brancher, J. D. (2024) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2024.242103 · Cited by 2
Are We Asking the Right Questions to ChatGPT for Learning Software Design Patterns?
Gerede, Ç. E. (2024) · International Conference on Computer Science and Engineering (UBMK) · doi:10.1109/ubmk63289.2024.10773596 · Cited by 4
“Jalil et al. [12] studied ChatGPT's performance in answering common questions from a software testing curriculum and found that it could respond to most questions correctly.”
The Impact of AIGC on Modern Education: Transforming Learning and Teaching Methods
Cheng, Y., & Wang, T. (2024) · International Conference on Intelligent Education and Intelligent Research (IEIR) · doi:10.1109/ieir62538.2024.10959761 · Cited by 2
“On the other hand, concerns have been raised about its potential for misuse [8], the spread of misinfor-mation [9], academic misconduct [10] [11] , and ethical issues [12] [13], given its ability to effortlessly generate seemingly authentic information.”
Ewais, A. (2025) · International Journal of Artificial Intelligence Tools · doi:10.1142/s0218213025500150 · Cited by 3
ChatGPT: Vriend en vyand in die Geografie-vakdidaktiek-klaskamer? Implikasies vir die praktyk
Staden, C. V. (2023) · Litnet Akademies : 'n Joernaal vir die Geesteswetenskappe, Natuurwetenskappe, Regte en Godsdienswetenskappe · doi:10.56273/1995-5928/2023/j20n2d5 · Cited by 4
Learning AI Coding Style for Software Plagiarism Detection
Ambati, S. H., Stakhanova, N., & Branca, E. (2024) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-64954-7_24 · Cited by 3
Zhao, H., & Silverajan, B. (2024) · International Conference on Availability, Reliability and Security (ARES) · doi:10.1145/3664476.3670943 · Cited by 1
“[2] have explored the integration of ChatGPT in software testing education, assessing its effectiveness in a software testing curriculum.”
The Benefits and Challenges of Using ChatGPT to Develop Programming Codes
Al-Shaar, W. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-2284-0.ch009 · Cited by 1
Artificial Intelligence Internet of Medical Things for Personalized Care
Pesqueira, A. M., & Almeida, D. (2025) · Advances in medical technologies and clinical practice book series · doi:10.4018/979-8-3373-0690-2.ch011 · Cited by 1
Venturing ChatGPT's lens to explore human values in software artifacts: a case study of mobile APIs
Mougouei, D., Rafi, S., Fahmideh, M., Mougouei, E., Khan, J. A., Dam, H. K., Nurwidyantoro, A., & Chaudron, M. R. V. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2025.2478278 · Cited by 1
Self-citationPesqueira, A. M., & Almeida, D. (2025) · IGI Global eBooks · doi:10.4018/979-8-3373-0350-5.ch004 · Cited by 2
Improving Requirement Comprehension in GPT Models for Enhanced Consistency in Test Case Generation
Ratmayanti, P. V. A., Kusumo, D. S., & Riskiana, R. R. (2025) · International Conference on Software Engineering and Information Technology (ICoSEIT) · doi:10.1109/icoseit67010.2025.11290782 · Cited by 1
“In late 2022 OpenAI introduced ChatGPT as an AI that builds on the existing LLM [7].”
Zuo, F., Qian, G., Qu, X., Rhee, J., & Fu, J. (2025) · Conference on Information Technology Education · doi:10.1145/3769694.3771150 · Cited by 1
“Jalil et al. [11] studied the effectiveness of ChatGPT in answering typical software testing questions.”
Metadata
- Authors
- Jalil, S., Rafi, S., LaToza, T. D., Moran, K., & Lam, W.
- Affiliations
- George Mason University (US)
- Venue
- 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)
- Publisher
- IEEE
- Published
- 2023-04
- Pages
- 4130-4137
- OpenAlex
- W4378676756
- Semantic Scholar
- c1014f86c8d801c93b37eed58311da5f9c9da2e4
- Cited by
- 270
- References
- 23
- Citations by year
- 2026: 34 · 2025: 62 · 2024: 104 · 2023: 69
- FWCI
- 69.13
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- ChatGPT, large language models, software testing education, artificial intelligence in education, chatbots, automatic question answering, academic dishonesty, prompting strategies, explanation quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
