ChatGPT and Software Testing Education: Promises & Perils

Jalil, Sajed; Rafi, Suzzana; LaToza, Thomas D.; Moran, Kevin; Lam, Wing

doi:10.1109/icstw58534.2023.00078

Cited by 274Artificial Intelligence in EducationSoftware Engineering ResearchTeaching and Learning Programming

ChatGPT and Software Testing Education: Promises & Perils

Open access

Jalil, Sajed, Rafi, Suzzana, LaToza, Thomas D., Moran, Kevin, Lam, Wing·2023·2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)

DOI 10.1109/icstw58534.2023.00078

Open this paper in CoChat — free account Open PDF

Abstract

Over the past decade, predictive language modeling for code has proven to be a valuable tool for enabling new forms of automation for developers. More recently, we have seen the ad-vent of general purpose "large language models", based on neural transformer architectures, that have been trained on massive datasets of human written text, which includes code and natural language. However, despite the demonstrated representational power of such models, interacting with them has historically been constrained to specific task settings, limiting their general applicability. Many of these limitations were recently overcome with the introduction of ChatGPT, a language model created by OpenAI and trained to operate as a conversational agent, enabling it to answer questions and respond to a wide variety of commands from end users.The introduction of models, such as ChatGPT, has already spurred fervent discussion from educators, ranging from fear that students could use these AI tools to circumvent learning, to excitement about the new types of learning opportunities that they might unlock. However, given the nascent nature of these tools, we currently lack fundamental knowledge related to how well they perform in different educational settings, and the potential promise (or danger) that they might pose to traditional forms of instruction. As such, in this paper, we examine how well ChatGPT performs when tasked with answering common questions in a popular software testing curriculum. We found that given its current capabilities, ChatGPT is able to respond to 77.5% of the questions we examined and that, of these questions, it is able to provide correct or partially correct answers in 55.6% of cases, provide correct or partially correct explanations of answers in 53.0% of cases, and that prompting the tool in a shared question context leads to a marginally higher rate of correct answers and explanations. Based on these findings, we discuss the potential promises and perils related to the use of ChatGPT by students and instructors.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Integrity signals

Registry

Crossref · 10.1109/icstw58534.2023.00078

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Software testing evaluates agents by having them conduct software testing tasks such as generating test cases, reproducing bugs, debugging code, and interacting with developers and external tools, using metrics like test coverage and bug detection rate.
“In this method, researchers evaluate the agents by letting them conduct tasks such as software testing tasks, such as generating test cases, reproducing bugs, debugging code, and interacting with developers and external tools [166,168,169,170].Then, one can use metrics like test coverage and bug detection rate to measure the effectiveness of LLM-based agents.”
From A survey on large language model based autonomous agents (2024) · cites this paper as [169]
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Attention Is All You Need
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1× · see citing works
“In parallel, the machine learning and natural language processing communities began building large-scale models centered around a specific type of neural architecture, the transformer [3]–[5], trained on massive datasets of text.”
Background
MizAR 60 for Mizar 50
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Aion Framework: Dimensional Emergence of AI Consciousness, Observer-Induced Collapse, and Cosmological Portal Dynamics
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 2× · see citing works
“To enable this type of interaction, OpenAI made use of reinforcement learning from human feedback, refining methods from past work on InstructGPT [11], which trained LLMs with both unsupervised data and with supervision in the form of task instruction.”
Method
Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models
Kung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115 · see citing works
Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 2× · see citing works
“ChatGPT produces responses that are at or near the passing threshold for all three parts of the US Medical Licensing Exam without any additional information or prompt engineering [26].”
BackgroundResult
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2× · see citing works
“To enable this type of interaction, OpenAI made use of reinforcement learning from human feedback, refining methods from past work on InstructGPT [11], which trained LLMs with both unsupervised data and with supervision in the form of task instruction.”
Method
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 3× · see citing works
“AlphaCode was found to generate code that was often similar to human-generated code [6] and achieved a simulated average ranking in the top 54% on Codeforces [17], a programming competition platform.”
Background
Learning to Represent Programs with GraphsPreprint
Allamanis, M., Brockschmidt, M., & Khademi, M. (2017) · International Conference on Learning Representations · doi:10.48550/arxiv.1711.00740 · Cited by 947 · see citing works
Evaluating large language models trained on code
Chen, M. (2024) · TIB Data Manager · doi:10.57702/wbv3e61b · Cited by 737 · see citing works
Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660 · cited 1× · see citing works
“Similarly, a second user study found that the productivity benefits of CoPilot were mixed: while it sometimes could make developers faster, it could also lead developers down timeconsuming rabbit holes debugging incorrect code [24].”
Background
Introduction to Software Testing
Ammann, P., & Offutt, J. (2016) · Cambridge University Press eBooks · doi:10.1017/9781316771273 · Cited by 609 · see citing works
ChatGPT Goes to Law SchoolPreprint
Choi, J. H., Hickman, K. E., Monahan, A., & Schwarcz, D. B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4335905 · Cited by 522 · cited 2× · see citing works
“Our results are in contrast to the results of applying ChatGPT in other domains, such as in medicine [26] or law [27] where ChatGPT is shown to pass certain parts of their exams.”
BackgroundResult
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463 · cited 2× · see citing works
“One study found that Codex already performs better than most students on the code writing questions found in typical introductory programming exams [20] as well as more advanced exams on data structures and algorithms [21].”
Background
Explanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402 · cited 1× · see citing works
“Including explanations may help reduce overreliance on potentially incorrect answers, but only in situations when the benefits of engaging with explanations outweigh the costs [25].”
Background
On the naturalness of software
Hindle, A., Barr, E. T., Gabel, M., Su, Z., & Devanbu, P. (2016) · Communications of the ACM · doi:10.1145/2902362 · Cited by 307 · see citing works
Toward Deep Learning Software Repositories
White, M., Vendome, C., Linares‐Vásquez, M., & Poshyvanyk, D. (2015) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2015.38 · Cited by 222 · cited 1× · see citing works
“As techniques for language modeling improved, researchers began to utilize Deep Learning (DL) architectures to learn rich, hierarchical representations of code that could then be used for various downstream tasks [2].”
Method
Examining Zero-Shot Vulnerability Repair with Large Language Models
Pearce, H. A., Tan, B., Ahmad, B., Karri, R., & Dolan-Gavitt, B. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179420 · Cited by 206 · see citing works
My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises
Finnie-Ansley, J., Denny, P., Luxton-Reilly, A., Santos, E. A., Prather, J., & Becker, B. A. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576134 · Cited by 186 · cited 1× · see citing works
“One study found that Codex already performs better than most students on the code writing questions found in typical introductory programming exams [20] as well as more advanced exams on data structures and algorithms [21].”
Background
In-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487569 · Cited by 154 · cited 1× · see citing works
“One study found that, despite participants themselves enjoying interacting with the code generation system, there was no measurable productivity benefit in either speed or correctness of programming tasks [23].”
Background
An Empirical Evaluation of Competitive Programming AI: A Case Study of AlphaCode
Lertbanjongngam, S., Chinthanet, B., Ishio, T., Kula, R. G., Leelaprute, P., Manaskasemsak, B., Rungsawang, A., & Matsumoto, K. (2022) · IEEE International Workshop on Software Clones (IWSC) · doi:10.1109/iwsc55060.2022.00010 · Cited by 26 · cited 2× · see citing works
“Experiments illustrated the representational power of both these large language models (LLMs), and language models tailored specifically for code [6]–[9].”
Background
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Do Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2022) · arXiv · cited 1×
“Examining the security implications of using these systems, one study found that developers using an AI code assistant were more likely to believe that they wrote secure code but had in fact written less secure code than if they had not used the AI code assistant [24].”
Background

Flagged first, then by citations

Research works printed but not linked: 1

[14]14. "ChatGPT Software Testing Study: Zenodo Record." https://doi.org/10. 5281/zenodo.7700501, 2023.

Sorted by citations

What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature
Lo, C. K. (2023) · Education Sciences · doi:10.3390/educsci13040410 · Cited by 2,024 · see citing works
“Moreover, ChatGPT’s possible failure to generate a correct reference list (see Table 5) can be a telltale sign for instructors seeking to identify whether a student has used ChatGPT [50,51,56].”
Background
A survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941 · see citing works
ChatGPT for Education and Research: Opportunities, Threats, and Strategies
Rahman, M. M., & Watanobe, Y. (2023) · Applied Sciences · doi:10.3390/app13095783 · Cited by 1,080 · see citing works
“[33] studied the performance of ChatGPT in solving software testing curriculum questions.”
Background
The Potential of Generative Artificial Intelligence Across Disciplines: Perspectives and Future Directions
Ooi, K.-B., Tan, G. W.-H., Al-Emran, M., Al-Sharafi, M. A., Capatina, A., Chakraborty, A., Dwivedi, Y. K., Huang, T.-L., Kar, A. K., Lee, V.-H., et al. (2025) · Journal of Computer Information Systems · doi:10.1080/08874417.2023.2261010 · Cited by 746 · see citing works
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650 · see citing works
“For example, Jalil et al. [212] explored opportunities for (and issues with) ChatGPT in software testing education.”
Background
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396 · see citing works
“Since AI generated content is presented authoritatively (and is frequently correct), students are unlikely to question the content and may learn incorrect information [51, 96].”
Background
ChatGPT versus engineering education assessment: a multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity
Nikolic, S., Daniel, S., Haque, R., Belkina, M., Hassan, G. M., Grundy, S., Lyden, S., Neal, P., & Sandison, C. (2023) · European Journal of Engineering Education · doi:10.1080/03043797.2023.2213169 · Cited by 336 · see citing works
RETRACTED ARTICLE: The effect of ChatGPT on students’ learning performance, learning perception, and higher-order thinking: insights from a meta-analysis
Wang, J., & Fan, W. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04787-y · Cited by 322 · see citing works
“Many studies have claimed that ChatGPT can improve students ’ learning performance, learning perception, and higher-order thinking (Jalil et al. 2023; Kasneci et al. 2023; Lu et al. 2024), but other studies have found that it does not signi ﬁ cantly improve students ’ learning performance or…”
Retracted
Artificial intelligence in developing countries: The impact of generative artificial intelligence (AI) technologies for development
Mannuru, N. R., Shahriar, S., Teel, Z. A., Wang, T., Lund, B. D., Tijani, S., Pohboon, C. O., Agbaji, D. A., Alhassan, J., Galley, J., et al. (2023) · Information Development · doi:10.1177/02666669231200628 · Cited by 392 · see citing works
ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design
White, J., Hays, S., Fu, Q., Spencer-Smith, J. B., & Schmidt, D. C. (2024) · Springer eBooks · doi:10.1007/978-3-031-55642-5_4 · Cited by 344 · see citing works
An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258 · see citing works
ChatGPT Challenges Blended Learning Methodologies in Engineering Education: A Case Study in Mathematics
Ruiz, L. M. S., Moll-López, S., Nuñez-Pérez, A., Moraño, J.-A., & Vega-Fleitas, E. (2023) · Applied Sciences · doi:10.3390/app13106039 · Cited by 271 · see citing works
“Same issues found in mathematics Potential Issues [80,82,83] In agreement Limited accuracy and reliability.”
Background
RETRACTED: The application and challenges of ChatGPT in educational transformation: New demands for teachers' roles
Yu, H. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e24289 · Cited by 225 · see citing works
“The dialogue between educators and AI must be carefully managed to ensure that the technology is used to enhance, rather than undermine, educational outcomes [58].”
Background
Retracted
The use of ChatGPT in teaching and learning: a systematic review through SWOT analysis approach
Mai, D. T. T., da, C. A. N., & Hanh, N. V. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1328769 · Cited by 310 · see citing works
Empowering education development through AIGC: A systematic literature review
Chen, X., Hu, Z., & Wang, C. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12549-7 · Cited by 222 · see citing works
Would ChatGPT-facilitated programming mode impact college students’ programming behaviors, performances, and perceptions? An empirical study
Sun, D., Boudouaia, A., Zhu, C., & Li, Y. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00446-5 · Cited by 187 · see citing works
“As highlighted by this research study and previous studies (Jalil et al., 2023; Yilmaz & Yilmaz, 2023b), inaccuracies in the code generated by ChatGPT limit its effectiveness and may lead to instances of academic dishonesty such as plagiarism.”
BackgroundResultInfluential
CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631830 · Cited by 192 · see citing works
“LLMs have also been applied to other computing education tasks, such as writing tests [5, 15], and helping novices learn how to craft effective prompts [10].”
Background
Unveiling the landscape of generative artificial intelligence in education: A comprehensive taxonomy of applications, challenges, and future prospects
Samala, A. D., Rawas, S., Wang, T., Reed, J. M., Kim, J., Howard, N.-J., & Ertz, M. (2025) · Education and Information Technologies · doi:10.1007/s10639-024-12936-0 · Cited by 229 · see citing works
Evaluating and Improving ChatGPT for Unit Test Generation
Yuan, Z., Liu, M., Ding, S., Wang, K., Chen, Y., Peng, X., & Lou, Y. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3660783 · Cited by 171 · see citing works
“ChatGPT has attracted wide attention due to its outstanding capability of solving various tasks [30, 40, 47, 88], including the SE tasks [13, 25, 71, 77], e.g., program repair [71, 82], code generation [25, 32, 66].”
ChatGPT for Education and Research: Opportunities, Threats, and Strategies
Rahman, M. M., & Watanobe, Y. (2023) · doi:10.20944/preprints202303.0473.v1 · Cited by 1,080 · see citing works
Unleashing ChatGPT's Power: A Case Study on Optimizing Information Retrieval in Flipped Classrooms via Prompt Engineering
Wang, M., Wang, M., Xu, X., Yang, L., Cai, D., & Yin, M. (2023) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2023.3324714 · Cited by 176 · see citing works
“In contrast, ChatGPT’s performance in mathematics [51], sports science [52], and software testing [53] is notably poor.”
Background
ChatGPT: Challenges and Benefits in Software Programming for Higher Education
da Silva, C. A. G., Ramos, F. N., de Moraes, R. V., & dos Santos, E. L. (2024) · Sustainability · doi:10.3390/su16031245 · Cited by 179 · see citing works
“Jalil et al. suggest there is a potential promise or danger that ChatGPT might pose to traditional forms of instruction, especially to solve exams for software testing education [85].”
Background
Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses
Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600142 · Cited by 141 · see citing works
Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study
Lyu, W., Wang, Y., Chung, T. (., Sun, Y., & Zhang, Y. (2024) · Proceedings of the Eleventh ACM Conference on Learning @ Scale · doi:10.1145/3657604.3662036 · Cited by 131 · see citing works
“CurrentresearchontheuseofLLMsineducation hasprimarilylookedintotheirperformanceandcapabilities[40] comparedtohumans,suchasgeneratingcodeforprogramming tasks[24,39],answeringgeneralinquriries[38,44],addressing textbookquestions[20]andexamquestions[12].”
AI Chatbots in Education: Challenges and Opportunities
Davar, N. F., Dewan, M. A. A., & Zhang, X. (2025) · Information · doi:10.3390/info16030235 · Cited by 123 · see citing works
“Integrating ChatGPT with educational platforms can also empower learners with disabilities to overcome barriers by tailoring course content according to their speciﬁc needs [16].”
Towards an understanding of large language models in software engineering tasks
Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10602-0 · Cited by 149 · see citing works
“, 2023a), answering software testing questions (Jalil et al., 2023), and test case generation (Tang et al.”
Influential
Using ChatGPT for human–computer interaction research: a primer
Tabone, W., & de Winter, J. (2023) · Royal Society Open Science · doi:10.1098/rsos.231053 · Cited by 136 · see citing works
ChatGPT in Computer Science Curriculum Assessment: An analysis of Its Successes and Shortcomings
Qureshi, B. (2023) · International Conference e-Society, e-Learning and e-Technologies · doi:10.1145/3613944.3613946 · Cited by 111 · see citing works
“Jalil et al. [9] presented ChatGPT with textbook questions to evaluate its ability in software testing.”
Background
Let’s Have a Chat! A Conversation with ChatGPT: Technology, Applications, and Limitations
Shahriar, S., & Hayawi, K. (2023) · Artificial Intelligence and Applications · doi:10.47852/bonviewaia3202939 · Cited by 167 · see citing works
“[41] evaluated ChatGPT’s capabilities in software testing by presenting it with textbook questions.”
Method
The Security of Using Large Language Models: A Survey with Emphasis on ChatGPT
Zhou, W., Zhu, X., Han, Q., Li, L., Chen, X., Wen, S., & Xiang, Y. (2024) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.124983 · Cited by 101 · see citing works
Large Language Models in Computer Science Education: A Systematic Literature Review
Raihan, M. N., Siddiq, M. L., Santos, J. C. S., & Zampieri, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701863 · Cited by 125 · see citing works
“The second most used language is Java [120, 128], primarily taught as an object-oriented programming language.”
Does ChatGPT Help With Introductory Programming?An Experiment of Students Using ChatGPT in CS1
Xue, Y., Chen, H., Bai, G. R., Tairas, R., & Huang, Y. (2024) · Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training · doi:10.1145/3639474.3640076 · Cited by 96 · see citing works
“Jalil et al. evaluated the performance of ChatGPT in answering practice questions in a popular software testing curriculum [15].”
BackgroundInfluential
Learning by playing with generative AI: design and evaluation of a role-playing educational game with generative AI as scaffolding for instant feedback interaction
Chien, C.-C., Chan, H.-Y., & Hou, H.-T. (2025) · Journal of Research on Technology in Education · doi:10.1080/15391523.2024.2338085 · Cited by 90 · see citing works
“Jalil et al. (2023) explored the correctness of ChatGPT responses in a software testing session.”
Influential
ChatGPT-Supported Education in Primary Schools: The Potential of ChatGPT for Sustainable Practices
Uğraş, H., Uğraş, M., Papadakis, S., & Kalogiannakis, M. (2024) · Sustainability · doi:10.3390/su16229855 · Cited by 116 · see citing works
ChatGPT as an important tool in organizational management: A review of the literature
Ayinde, L., Wibowo, M. P., Ravuri, B., & Emdad, F. B. (2023) · Business Information Review · doi:10.1177/02663821231187991 · Cited by 74 · see citing works
“ChatGPT provided correct answers for specific contexts based on quizzes on software engineering, by answering 83% of coding questions (Jalil et al., 2023).”
Background
Patterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant
Sheese, B. E., Liffiton, M., Šavelka, J., & Denny, P. C. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636249 · Cited by 76 · see citing works
“Thebreadthofthisworkdemonstratesboththeflexibility andthecapabilitiesofLLMs.”
What factors will affect the effectiveness of using ChatGPT to solve programming problems? A quasi-experimental study
Jing, Y., Wang, H., Chen, X.-J., & Wang, C.-L. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-02751-w · Cited by 74 · see citing works
“In this context, given ChatGPT ’ s exceptional programming and precise answering capabilities (Luo et al., 2023), some scholars have noted that ChatGPT ’ s functions pose a threat to some programmers (Ouh et al., 2023; Jalil et al., 2023).”
ChatGPT: Systematic Review, Applications, and Agenda for Multidisciplinary Research
Singh, H., & Singh, A. (2023) · Journal of Chinese Economic and Business Studies · doi:10.1080/14765284.2023.2210482 · Cited by 77 · see citing works
“In education and competitive examination, ChatGPT offered factual answers to users (Jalil et al. 2023).”
BackgroundMethod
Students' reflections on their experience with <scp>ChatGPT</scp>
Šedlbauer, J., Činčera, J., Slavík, M., & Hartlová, A. (2024) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12967 · Cited by 71 · see citing works
Generative Artificial Intelligence for Software Engineering - a Research Agenda
NguyenDuc, A., Cabrero-Daniel, B., Arora, C., Przybylek, A., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K.-K., et al. (2023) · doi:10.2139/ssrn.4622517 · Cited by 71 · see citing works
Untitled work
(2024) · doi:10.1109/sp54263.2024.00182 · Cited by 62 · see citing works
ChatGPT in the Classroom. Exploring Its Potential and Limitations in a Functional Programming Course
Popovici, M.-D. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2269006 · Cited by 70 · see citing works
“There is a great deal of lively debate surrounding its potential uses, impact on research: Stokel-Walker (2023), science: Frieder et al. (2023), and education: Lambeets (2023), Elsen-Rooney (2023), Jalil et al. (2023), Lau and Guo (2023).”
Influential
How generative artificial intelligence has blurred notions of authorial identity and academic norms in higher education, necessitating clear university usage policies
Duah, J. E., & McGivern, P. (2024) · The International Journal of Information and Learning Technology · doi:10.1108/ijilt-11-2023-0213 · Cited by 67 · see citing works
AI-Tutoring in Software Engineering Education
Frankford, E., Sauerwein, C., Bassner, P., Krusche, S., & Breu, R. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640061 · Cited by 78 · see citing works
“However, regarding “Software Testing”, it was able to answer 55.6% of the questions partially correctly [11].”
BackgroundInfluential
Seeing ChatGPT Through Students’ Eyes: An Analysis of TikTok Data
Haensch, A., Ball, S. L., Herklotz, M., & Kreuter, F. (2023) · 2023 Big Data Meets Survey Science (BigSurv) · doi:10.1109/bigsurv59479.2023.10486710 · Cited by 62 · see citing works
“Jalil et al. [22] confronted ChatGPT with questions from a well-known textbook on software testing.”
Background
Generative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60 · see citing works
Exploring the Role of AI Assistants in Computer Science Education: Methods, Implications, and Instructor Perspectives
Wang, T., Vargas-Díaz, D., Brown, C., & Chen, Y. (2023) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc57772.2023.00018 · Cited by 68 · see citing works
“An increasing number of researchers have shown these AI assistants outperform students when solving complex computing tasks [3], and can circumvent plagiarism detection software [4].”
Background
Nuances are the Key: Unlocking ChatGPT to Find Failure-Inducing Tests with Differential Prompting
Li, T.-O., Zong, W., Wang, Y., Tian, H., Wang, Y., Cheung, S.-C., & Kramer, J. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00089 · Cited by 70 · see citing works
“There are also other works studying ChatGPT’s (or other LLMs’) capabilities for various software engineering tasks, including bug repair [25], [3], fuzzing [22], code generation [51], [52], code summarization [53], software testing education [54], and vulnerability detection [55], [56].”
Background
The Good and Bad of AI Tools in Novice Programming Education
Zviel-Girshin, R. (2024) · Education Sciences · doi:10.3390/educsci14101089 · Cited by 55 · see citing works
Bob or Bot: Exploring ChatGPT's Answers to University Computer Science Assessment
Richards, M., Waugh, K., Slaymaker, M., Petre, M., Woodthorpe, J., & Gooch, D. (2024) · ACM Transactions on Computing Education · doi:10.1145/3633287 · Cited by 52 · see citing works
“Each of these studies focusing on the academic performance of a LLM [ 30 , 19 , 20 , 33 ] used the research team or automated testing to assess the generated answers, rather than a marker who had no insight into the source of the answers.”
MethodResult
An In-Depth Review of ChatGPT’s Pros and Cons for Learning and Teaching in Education
Samala, A. D., Zhaı, X., Aoki, K., Bojić, L., & Žikić, S. (2024) · International Journal of Interactive Mobile Technologies (iJIM) · doi:10.3991/ijim.v18i02.46509 · Cited by 75 · see citing works
Evaluating the Quality of AI-Generated Digital Educational Resources for University Teaching and Learning
Huang, Q., Lv, C., Li, L., & Tu, S. (2025) · Systems · doi:10.3390/systems13030174 · Cited by 50 · see citing works
Ethical Aspects of ChatGPT in Software Engineering Research
Akbar, M. A., Khan, A. A., & Liang, P. (2023) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2023.3318183 · Cited by 52 · see citing works
“This can save time and resources by automating the process of generating test cases, allowing for rapid iteration and software performance evaluation [33].”
Influential
CHEAT: A Large-Scale Dataset for Detecting CHatGPT-writtEn AbsTracts
Yu, P., Chen, J., Feng, X., & Xia, Z. (2025) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2025.3536929 · Cited by 65 · see citing works
“(cid:2) CST : The ChatGPT-software-testing dataset [27] includes ChatGPT’s answers to 31 questions from the software testing book named “Introduction to Software Testing 2nd Edition”.”
transformative potential of Generative Artificial Intelligence (GenAI) in business
Cano-Marin, E. (2024) · ESIC Market · doi:10.7200/esicm.55.333 · Cited by 37 · see citing works
Large language models for qualitative research in software engineering: exploring opportunities and challenges
Bano, M., Hoda, R., Zowghi, D., & Treude, C. (2023) · Automated Software Engineering · doi:10.1007/s10515-023-00407-8 · Cited by 48 · see citing works
“Concurrently, Jalil et al. (2023) analyze ChatGPT’s role in software testing education, revealing its potential and the risks of overreliance due to varying response accuracy.”
Towards Autonomous Testing Agents via Conversational Large Language Models
Feldt, R., Kang, S., Yoon, J., & Yoo, S. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00148 · Cited by 49 · see citing works
“They also exhibit some ‘knowledge’ about testing methodologies and their steps and can adapt them to new situations [9].”
Background
Powerful or mediocre? Kindergarten teachers’ perspectives on using ChatGPT in early childhood education
Su, J., & Yang, W. (2023) · Interactive Learning Environments · doi:10.1080/10494820.2023.2266490 · Cited by 50 · see citing works
“Given the potential opportunities and challenges, sentiments in various levels of academia range from fear that students could use these AI tools to circumvent learning, to excitement over the potential of ChatGPT being used to unlock new learning opportunities (Jalil et al., 2023).”
Engineering Education in the Age of AI: Analysis of the Impact of Chatbots on Learning in Engineering
Sánchez, F. Á. B., & Cruz, J. (2024) · Education Sciences · doi:10.3390/educsci14050484 · Cited by 56 · see citing works
“For example, Jalil et.al., [29] found that although ChatGPT can respond to 77.5% of the ex-Educ amined questions, it provided correct or partially correct answers in only 56.6% of cases and correct or partially correct explanations in only 53% of cases.”
BackgroundInfluential
Software engineering education in the era of conversational AI: current trends and future directions
Sengul, C., Neykova, R., & Destefanis, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1436350 · Cited by 55 · see citing works
“The study showed no signiﬁcant improvement in students’ defect identiﬁcation skills, likely due to the inadequate capabilities of the custom CA. Jalil et al. (2023) examine the eﬀectiveness of ChatGPT in answering practice questions from standard software testing curricula.”
Background
Potentials of ChatGPT in Computer Programming: Insights from Programming Instructors
Husain, A. (2024) · Journal of Information Technology Education Research · doi:10.28945/5240 · Cited by 57 · see citing works
“…the potential roles of ChatGPT in coding and reported several useful applications, such as code generation (Kashefi & Mukerji, 2023), code optimization (Biswas, 2023), code documentation (Haleem et al., 2022), code review (Jalil et al., 2023), and debugging assistance (Surameery & Shakor, 2023).”
BackgroundResultInfluential
ChatGPT
Kazi, K. S. L. (2024) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3693-4268-8.ch001 · Cited by 38 · see citing works
Human-AI collaboration in software engineering: lessons learned from a hands-on workshop
Hamza, M., Siemon, D., Akbar, M. A., & Rahman, T. (2024) · Proceedings of the 7th ACM/IEEE International Workshop on Software-intensive Business · doi:10.1145/3643690.3648236 · Cited by 38 · see citing works
“Similarly, Sajed Jalil et al. [15] evaluate the performance of ChatGPT in a software testing curriculum, revealing that while it correctly or partially answers 55.6% of questions, its self-assessed confidence is often misaligned with the accuracy of responses, highlighting both the potential and…”
MethodInfluential
ChatGPT in Action: Analyzing Its Use in Software Development
Champa, A. I., Rabbi, M. F., Nachuma, C., & Zibran, M. F. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3645077 · Cited by 36 · see citing works
“Besides, some studies assessed the efficiency of ChatGPT in providing assistance [20, 30], while some researchers conducted comparative studies of Chat-GPT with other similar systems in assisting software development tasks [2, 23, 43].”
Result
Is ChatGPT Beneficial to Education? A Holistic Evaluation Framework Based on Intelligent Tutoring Systems
Virvou, M. K., & Tsihrintzis, G. A. (2023) · International Conference on Information, Intelligence, Systems & Applications (IISA) · doi:10.1109/iisa59645.2023.10345949 · Cited by 34 · see citing works
“In an evaluation experiment conducted by Jalil and colleagues [15], ChatGPT was evaluated in terms of accuracy and completeness of its domain knowledge of software testing.”
Method
An Evaluation of ChatGPT’s Proficiency in English Language Testing of The Vietnamese National High School Graduation ExaminationPreprint
Dao, X.-Q., Le, N.-B., Phan, X.-D., & Ngo, B. B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4473369 · Cited by 33 · see citing works
Risk management strategy for generative AI in computing education: how to handle the strengths, weaknesses, opportunities, and threats?
Humble, N. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00494-x · Cited by 40 · see citing works
“…et al., 2024; Cambaz & Zhang, 2024; Dwivedi et al., 2023; Finnie-Ansley et al., 2023; French et al., 2023; Gumina et al., 2023; Humble et al., 2024; Jalil et al., 2023; Karnalim et al., 2023; Li et al., 2023; Nguyen et al., 2024b; Nikolic et al., 2024; Pan et al., 2024; Quille et al., 2024;…”
Influential
Enhancing instructional design learning: a comparative study of scaffolding by a 5E instructional model-informed artificial intelligence chatbot and a human teacher
Bai, S., Lo, C. K., & Yang, C. C. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2024.2420184 · Cited by 37 · see citing works
“However, there were challenges during the “ evaluate ” phase, where the chatbots ’ ability to assess instructional designs sometimes diverged from the course assessment rubric, which was noted in the students ’ feedback and by previous studies (Jalil et al., 2023; Liu et al., 2023a).”
Eğitimde Yapay Zekâ Kullanımı: ChatGPT’nin KEFE ve PEST Analizi
Kutlucan, E., & Seferoğlu, S. S. (2024) · Türk Eğitim Bilimleri Dergisi · doi:10.37217/tebd.1368821 · Cited by 30 · see citing works
Enhancing Engineering Education in the Roblox Metaverse: Utilizing chatGPT for Game Development for Electrical Machine Course
Ho, W., & Lee, D.-H. (2023) · International Journal on Advanced Science Engineering and Information Technology · doi:10.18517/ijaseit.13.3.18458 · Cited by 30 · see citing works
Exploring the Latest Applications of OpenAI and ChatGPT: An In-Depth Survey
Hong, Z., & Shao, H. (2023) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2023.030649 · Cited by 34 · see citing works
ChatGPT: towards AI subjectivity
D’Amato, K. (2024) · AI & Society · doi:10.1007/s00146-024-01898-z · Cited by 29 · see citing works
“…public health (Biswas 2023), the medical industry (Kung et al. 2023), business and finance (AlAfan et al. 2023), law (Choi et al. 2023), creative writing (Cox and Tzoc 2023), software development (Jalil et al. 2023), marketing (Dwivedi et al. 2023), and scientific research (Salvagno et al. 2023).”
Recommendations to Create Programming Exercises to Overcome ChatGPT
Berrezueta-Guzman, J., & Krusche, S. (2023) · 2023 IEEE 35th International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet58097.2023.00031 · Cited by 27 · see citing works
“Jalil and his colleagues evaluated ChatGPT’s performance in answering common questions from a popular software testing curriculum [20].”
Background
Codepori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology
Rasheed, Z., Malik, S., Kemell, K.-K., Muhammad, W., Saari, M., Systä, K., & Abrahamsson, P. (2024) · doi:10.2139/ssrn.4979510 · Cited by 27 · see citing works
ChatGPT could be the reviewer of your next scientific paper. Evidence on the limits of AI-assisted academic reviews
Carabantes-Alarcón, D., González-Geraldo, J. L., & Olmeda, G. J. (2023) · El Profesional de la Informacion · doi:10.3145/epi.2023.sep.16 · Cited by 26 · see citing works
A Generative Artificial Intelligence (AI)-Based Human-Computer Collaborative Programming Learning Method to Improve Computational Thinking, Learning Attitudes, and Learning Achievement
Zhao, G., Yang, L., Hu, B., & Wang, J. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331251336154 · Cited by 27 · see citing works
“The rapid advancement of generative AI technologies has greatly improved the intelligence level of machines (OpenAI, 2022), and its unique advantages in language understanding, dialogue interaction and logical reasoning offer a novel approach to HCC programming learning (Jalil et al., 2023).”
Generative AI in the Software Modeling Classroom: An Experience Report With ChatGPT and Unified Modeling Language
Cámara, J., Troya, J., Montes-Torres, J., & Jaime, F. J. (2024) · IEEE Software · doi:10.1109/ms.2024.3385309 · Cited by 24 · see citing works
Hands-on analysis of using large language models for the auto evaluation of programming assignments
Mohamed, K. M., Yousef, M., Medhat, W., Mohamed, E. H., Khoriba, G. A., & Arafa, T. (2024) · Information Systems · doi:10.1016/j.is.2024.102473 · Cited by 26 · see citing works
Code Summarization without Direct Access to Code - Towards Exploring Federated LLMs for Software Engineering
Kumar, J., & Chimalakonda, S. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661210 · Cited by 23 · see citing works
“Further, the evolution of LLMs with newer models such as GPT family [1, 31], T5 family [23, 48], etc led to multiple adoptions and validation works in SE.”
Method
Generative AI-Enabled Conversational Interaction to Support Self-Directed Learning Experiences in Transversal Computational Thinking
Ouaazki, A., Bergram, K., Farah, J. C., Gillet, D., & Holzer, A. (2024) · International Conference on Conversational User Interfaces · doi:10.1145/3640794.3665542 · Cited by 28 · see citing works
“Similarly, Jalil et al. [26] investigated the capabilities of ChatGPT in the context of a traditional software testing course.”
Background
ChatGPT’s Impact on Education and Healthcare: Insights, Challenges, and Ethical Consideration
Hamad, Z. T., Jamil, N., & Belkacem, A. N. (2024) · IEEE Access · doi:10.1109/access.2024.3437374 · Cited by 25 · see citing works
“This real-time guidance fosters active learning, enabling students to take control of their educational journey and make continuous improvements [26].”
Background
Evaluating the performance of ChatGPT and Perplexity AI in Business Reference
Deike, M. (2024) · Journal of Business & Finance Librarianship · doi:10.1080/08963568.2024.2317534 · Cited by 22 · see citing works
“AI has been shown to frequently answer inquiries incorrectly in multiple disciplines (Jalil, Rafi, LaToza, Moran, & Lam, 2023).”
Background
ChatGPT in early childhood STEM education: Can it be an innovative tool to overcome challenges?
Uğraş, H., & Uğraş, M. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12960-0 · Cited by 23 · see citing works
“…emphasized in the literature that ChatGPT can produce false and fake information and that there are some doubts about its accuracy and reliability (Baidoo-Anu & Owusu Ansah, 2023; British University, Vietnam & Perkins, 2023 ; Jalil et al., 2023; King & chatGPT, 2023; Mogali, 2023; Szabo, 2023).”
Transforming Education with ChatGPT: Advancing Personalized Learning, Accessibility, and Ethical AI Integration
Asy’ari, M., & Шаров, С. (2024) · International Journal of Essential Competencies in Education · doi:10.36312/ijece.v3i2.2424 · Cited by 21 · see citing works
“Studies have highlighted its role in diverse fields, including athletic training (Jalil et al., 2023), software testing education (Alhammad, 2024), and literature appreciation (Forman et al., 2023).”
Large language model-based methodology for data-driven health prediction of Lithium-ion batteries
Tunçel, S., Çınar, H., Güçyetmez, M., & Erdoğan, N. (2025) · Energy Reports · doi:10.1016/j.egyr.2025.05.027 · Cited by 21 · see citing works
ChatGPT Uzaktan Eğitim Sınavlarında Başarılı Olabilir Mi Turizm Alanında Doğruluk ve Doğrulama Üzerine Bir Araştırma (Can ChatGPT Succeed in Distance Education Exams A Research on Accuracy and Verification in Tourism)
Göktaş, L. S. (2023) · Journal of Tourism and Gastronomy Studies · doi:10.21325/jotags.2023.1224 · Cited by 23 · see citing works
Affordances, constraints, and implications of ChatGPT in education from a social-ecological perspective: A data mining approach
Zhong, Y., Lian, J., Huang, H., & Deng, H. (2025) · Education and Information Technologies · doi:10.1007/s10639-024-13237-2 · Cited by 25 · see citing works
“Jalil et al. (2023) evaluated ChatGPT and found that the system was able to respond to 77.5% of the examined questions, and among these questions, it provided correct or partially correct answers in 55.6% of cases and correct or partially correct explanations of answers in 53.0% of cases.”
Influential
GPT-Powered Elicitation Interview Script Generator for Requirements Engineering Training
Görer, B., & Aydemir, F. B. (2024) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re59067.2024.00044 · Cited by 19 · see citing works
“As assistive systems, LLMs have been explored for various applications like aiding pair-programming learning processes [22], question-answering for software testing [23], and assisting with coding-related queries [24].”
Background
Who's the Best Detective? Large Language Models vs. Traditional Machine Learning in Detecting Incoherent Fourth Grade Math Answers
Urrutia, F., & Araya, R. (2023) · Journal of educational computing research · doi:10.1177/07356331231191174 · Cited by 20 · see citing works
An Automated Model of Software Requirement Engineering Using GPT-3.5
Yeow, J. S., Rana, M. E., & Majid, N. A. A. (2024) · ASU International Conference in Emerging Technologies for Sustainability and Intelligent Systems (ICETSIS) · doi:10.1109/icetsis61505.2024.10459458 · Cited by 18 · see citing works
“5 to be the answer users are looking for [28].”
Background
An Empirical Study on How Large Language Models Impact Software Testing Learning
Mezzaro, S., Gambi, A., & Fraser, G. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661273 · Cited by 17 · see citing works
“In the context of software testing education, instead, Jalil et al. investigated whether ChatGPT could answer questions and solve programming quizzes on basic software testing concepts taken from a widely used textbook [7].”
BackgroundMethod
Advancing Scientific Research in Computer Science by ChatGPT and LLaMA—A Review
Hassan, E., Bhatnagar, R., & Shams, M. Y. (2023) · Smart innovation, systems and technologies · doi:10.1007/978-981-99-6774-2_3 · Cited by 17 · see citing works
Exploring the impact of EU tendering operations on future AI governance and standards in pharmaceuticals
Pesqueira, A. M., Machado, A. D. B., Bolog, S., Pereira, R., & Sousa, M. J. (2024) · Computers & Industrial Engineering · doi:10.1016/j.cie.2024.110655 · Cited by 16 · see citing works
Enhancing Computer Programming Education using ChatGPT- A Mini Review
Deriba, F. G., Sanusi, I. T., & Sunday, A. O. (2023) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631848 · Cited by 15 · see citing works
“These studies have mostly shown that ChatGPT accurately generates programming solutions despite concerns with organization or issues specific to a programming language, among others [5] [16].”
Background
The effectiveness of Gen AI in assisting students’ knowledge construction in humanities and social sciences courses: learning behaviour analysis
He, S., & Lu, Y. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2024.2415444 · Cited by 17 · see citing works
“This involves closely observing the content generated by ChatGPT, directing students ’ learning process, and conducting secondary validation of the information provided by ChatGPT (Jalil et al., 2023).”
EXAMINING THE IMPACT OF AI-POWERED WRITING TOOLS ON INDEPENDENT WRITING SKILLS OF HEALTH SCIENCE GRADUATES
Milton, C., Vidhya, L., & Thiruvengadam, G. (2024) · Advanced Education · doi:10.20535/2410-8286.315068 · Cited by 16 · see citing works
“A study examining the ability of ChatGPT to furnish accurate responses to the questions fed by users found that it could only provide correct or partially correct answers in 57% of cases (Jalil et al., 2023).”
How Can ChatGPT Support Human Security Testers to Help Mitigate Supply Chain Attacks?
Zhang, Y., Song, W., Ji, Z., Yao, D. D., & Meng, N. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3632765 · Cited by 13 · see citing works
“Some research was recently conducted to explore LLMs’ capability in programming, coding assistance, or jailbreak attacks [65], [71], [82], [84], [86], [91], [92], [94], [100], [106], [107].”
La Incorporación del ChatGPT en la Educación Superior: Una Mirada desde el Paradigma de la Complejidad
Romero-Rodríguez, P. (2023) · 593 Digital Publisher CEIT · doi:10.33386/593dp.2023.5.1976 · Cited by 12 · see citing works
Assessing the Impact of ChatGPT in a PHP Programming Course
Alhajj, J., & Şah, M. (2023) · International Service Availability Symposium · doi:10.1109/isas60782.2023.10391549 · Cited by 11 · see citing works
“Different than the related works [1, 21, 26, 29], in our research, we divided the exam questions into different categories and measured the impact of ChatGPT as a whole and in different question categories in terms of scores and user attitudes.”
Background
An Exploratory Study on Upper-Level Computing Students’ Use of Large Language Models as Tools in a Semester-Long Project
Tanay, B. A., Arinze, L., Joshi, S. S., Davis, K., & Davis, J. C. (2024) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--46557 · Cited by 12 · see citing works

An Empirical Study on How Large Language Models Impact Software Testing Learning
Mezzaro, S., Gambi, A., & Fraser, G. (2024) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3661167.3661273 · Cited by 17 · see citing works
Randomized Controlled Trial in Vocational Education: Assessing ChatGPT’s Efficacy in Software Testing Skill Development through Multi-Stage Prompt EngineeringPreprint
Yang, Y., & Yang, Y. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5441864
Nurturing Code Quality: Leveraging Static Analysis and Large Language Models for Software Quality in Education
AlOmar, E. A. (2025) · ACM Transactions on Computing Education · doi:10.1145/3722229 · Cited by 11 · see citing works
Students' Perception of ChatGPT in Software Engineering: Lessons Learned from Five Courses
Baresi, L., Lucia, A. D., Marco, A. D., Penta, M. D., Ruscio, D. D., Mariani, L., Micucci, D., Palomba, F., Rossi, M. T., & Zampetti, F. (2025) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet66350.2025.00023 · Cited by 8 · see citing works
Large Language Models for Software Testing Education: an Experience ReportPreprint
Yang, P., Zhu, Y., Chang, C., Yu, S., Chen, Z., & Tang, Y. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2603.26329

Authors: Jalil, S., Rafi, S., LaToza, T. D., Moran, K., & Lam, W.
Affiliations: George Mason University (US)
ORCID: Jalil, Sajed
LaToza, Thomas D.
Venue: 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)
Publisher: IEEE
Published: 2023-04
Pages: 4130-4137
DOI: 10.1109/icstw58534.2023.00078
OpenAlex: W4378676756
Semantic Scholar: c1014f86c8d801c93b37eed58311da5f9c9da2e4
Cited by: 270
References: 23
Citations by year: 2026: 34 · 2025: 62 · 2024: 104 · 2023: 69 · 2022: 1
FWCI: 69.13
Type: conference paper
Language: en
Open access: green
arxiv.org
Licence: doi.org/10.15223/policy-029 (stm-asf)
doi.org/10.15223/policy-037 (stm-asf)
Locations: IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (publishedVersion)
arXiv (Cornell University) (submittedVersion)
arXiv (Cornell University)
Topics: Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords: ChatGPT, large language models, software testing education, artificial intelligence in education, chatbots, automatic question answering, academic dishonesty, prompting strategies, explanation quality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

ChatGPT and Software Testing Education: Promises & Perils

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Self-check

Cites

Cited by

Related

Record

About these checks