The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
ChatGPT and Software Testing Education: Promises & Perils
Open accessAbstract
Over the past decade, predictive language modeling for code has proven to be a valuable tool for enabling new forms of automation for developers. More recently, we have seen the ad-vent of general purpose "large language models", based on neural transformer architectures, that have been trained on massive datasets of human written text, which includes code and natural language. However, despite the demonstrated representational power of such models, interacting with them has historically been constrained to specific task settings, limiting their general applicability. Many of these limitations were recently overcome with the introduction of ChatGPT, a language model created by OpenAI and trained to operate as a conversational agent, enabling it to answer questions and respond to a wide variety of commands from end users.The introduction of models, such as ChatGPT, has already spurred fervent discussion from educators, ranging from fear that students could use these AI tools to circumvent learning, to excitement about the new types of learning opportunities that they might unlock. However, given the nascent nature of these tools, we currently lack fundamental knowledge related to how well they perform in different educational settings, and the potential promise (or danger) that they might pose to traditional forms of instruction. As such, in this paper, we examine how well ChatGPT performs when tasked with answering common questions in a popular software testing curriculum. We found that given its current capabilities, ChatGPT is able to respond to 77.5% of the questions we examined and that, of these questions, it is able to provide correct or partially correct answers in 55.6% of cases, provide correct or partially correct explanations of answers in 53.0% of cases, and that prompting the tool in a shared question context leads to a marginally higher rate of correct answers and explanations. Based on these findings, we discuss the potential promises and perils related to the use of ChatGPT by students and instructors.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In this method, researchers evaluate the agents by letting them conduct tasks such as software testing tasks, such as generating test cases, reproducing bugs, debugging code, and interacting with developers and external tools [166,168,169,170].Then, one can use metrics like test coverage and bug detection rate to measure the effectiveness of LLM-based agents.”
From A survey on large language model based autonomous agents (2024) · cites this paper as [169]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1×
“In parallel, the machine learning and natural language processing communities began building large-scale models centered around a specific type of neural architecture, the transformer [3]–[5], trained on massive datasets of text.”
BackgroundJakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 2×
“To enable this type of interaction, OpenAI made use of reinforcement learning from human feedback, refining methods from past work on InstructGPT [11], which trained LLMs with both unsupervised data and with supervision in the form of task instruction.”
MethodKung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 2×
“ChatGPT produces responses that are at or near the passing threshold for all three parts of the US Medical Licensing Exam without any additional information or prompt engineering [26].”
BackgroundResultTraining Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · cited 2×
“To enable this type of interaction, OpenAI made use of reinforcement learning from human feedback, refining methods from past work on InstructGPT [11], which trained LLMs with both unsupervised data and with supervision in the form of task instruction.”
MethodCodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 3×
“AlphaCode was found to generate code that was often similar to human-generated code [6] and achieved a simulated average ranking in the top 54% on Codeforces [17], a programming competition platform.”
BackgroundLearning to Represent Programs with GraphsPreprint
Allamanis, M., Brockschmidt, M., & Khademi, M. (2017) · International Conference on Learning Representations · doi:10.48550/arxiv.1711.00740 · Cited by 947
Evaluating large language models trained on code
Chen, M. (2024) · TIB Data Manager · doi:10.57702/wbv3e61b · Cited by 737
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660 · cited 1×
“Similarly, a second user study found that the productivity benefits of CoPilot were mixed: while it sometimes could make developers faster, it could also lead developers down timeconsuming rabbit holes debugging incorrect code [24].”
BackgroundIntroduction to Software Testing
Ammann, P., & Offutt, J. (2016) · Cambridge University Press eBooks · doi:10.1017/9781316771273 · Cited by 643
ChatGPT Goes to Law SchoolPreprint
Choi, J. H., Hickman, K. E., Monahan, A., & Schwarcz, D. B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4335905 · Cited by 565 · cited 2×
“Our results are in contrast to the results of applying ChatGPT in other domains, such as in medicine [26] or law [27] where ChatGPT is shown to pass certain parts of their exams.”
BackgroundResultThe Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463 · cited 2×
“One study found that Codex already performs better than most students on the code writing questions found in typical introductory programming exams [20] as well as more advanced exams on data structures and algorithms [21].”
BackgroundExplanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402 · cited 1×
“Including explanations may help reduce overreliance on potentially incorrect answers, but only in situations when the benefits of engaging with explanations outweigh the costs [25].”
BackgroundOn the naturalness of software
Hindle, A., Barr, E. T., Gabel, M., Su, Z., & Devanbu, P. (2016) · Communications of the ACM · doi:10.1145/2902362 · Cited by 307
Toward Deep Learning Software Repositories
White, M., Vendome, C., Linares‐Vásquez, M., & Poshyvanyk, D. (2015) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2015.38 · Cited by 222 · cited 1×
“As techniques for language modeling improved, researchers began to utilize Deep Learning (DL) architectures to learn rich, hierarchical representations of code that could then be used for various downstream tasks [2].”
MethodExamining Zero-Shot Vulnerability Repair with Large Language Models
Pearce, H. A., Tan, B., Ahmad, B., Karri, R., & Dolan-Gavitt, B. (2023) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp46215.2023.10179420 · Cited by 206
My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises
Finnie-Ansley, J., Denny, P., Luxton-Reilly, A., Santos, E. A., Prather, J., & Becker, B. A. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576134 · Cited by 186 · cited 1×
“One study found that Codex already performs better than most students on the code writing questions found in typical introductory programming exams [20] as well as more advanced exams on data structures and algorithms [21].”
BackgroundIn-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487569 · Cited by 154 · cited 1×
“One study found that, despite participants themselves enjoying interacting with the code generation system, there was no measurable productivity benefit in either speed or correctness of programming tasks [23].”
BackgroundAn Empirical Evaluation of Competitive Programming AI: A Case Study of AlphaCode
Lertbanjongngam, S., Chinthanet, B., Ishio, T., Kula, R. G., Leelaprute, P., Manaskasemsak, B., Rungsawang, A., & Matsumoto, K. (2022) · IEEE International Workshop on Software Clones (IWSC) · doi:10.1109/iwsc55060.2022.00010 · Cited by 33 · cited 2×
“Experiments illustrated the representational power of both these large language models (LLMs), and language models tailored specifically for code [6]–[9].”
BackgroundJi, Y., & Gao, S. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 10
Do Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2022) · arXiv · cited 1×
“Examining the security implications of using these systems, one study found that developers using an AI code assistant were more likely to believe that they wrote secure code but had in fact written less secure code than if they had not used the AI code assistant [24].”
Background
Research works printed but not linked: 1
- Puryear, 2022, Github CoPilot in the Classroom: Learning to Code with AI Assistance, CCSC
Cited by
Sorted by citations
What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature
Lo, C. K. (2023) · Education Sciences · doi:10.3390/educsci13040410 · Cited by 2,024
“Moreover, ChatGPT’s possible failure to generate a correct reference list (see Table 5) can be a telltale sign for instructors seeking to identify whether a student has used ChatGPT [50,51,56].”
BackgroundA survey on large language model based autonomous agents
Wang, L., Ma, C., Feng, X., Zhang, Z., Yang, H., Zhang, J., Chen, Z., Tang, J., Chen, X., Lin, Y., et al. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40231-1 · Cited by 1,941
ChatGPT for Education and Research: Opportunities, Threats, and Strategies
Rahman, M. M., & Watanobe, Y. (2023) · Applied Sciences · doi:10.3390/app13095783 · Cited by 1,080
“[33] studied the performance of ChatGPT in solving software testing curriculum questions.”
BackgroundOoi, K.-B., Tan, G. W.-H., Al-Emran, M., Al-Sharafi, M. A., Capatina, A., Chakraborty, A., Dwivedi, Y. K., Huang, T.-L., Kar, A. K., Lee, V.-H., et al. (2025) · Journal of Computer Information Systems · doi:10.1080/08874417.2023.2261010 · Cited by 746
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“For example, Jalil et al. [212] explored opportunities for (and issues with) ChatGPT in software testing education.”
Background
Metadata
- Authors
- Jalil, S., Rafi, S., LaToza, T. D., Moran, K., & Lam, W.
- Affiliations
- George Mason University (US)
- Venue
- 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)
- Publisher
- IEEE
- Published
- 2023-04
- Pages
- 4130-4137
- OpenAlex
- W4378676756
- Semantic Scholar
- c1014f86c8d801c93b37eed58311da5f9c9da2e4
- Cited by
- 270
- References
- 23
- Citations by year
- 2026: 34 · 2025: 62 · 2024: 104 · 2023: 69
- FWCI
- 69.13
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- ChatGPT, large language models, software testing education, artificial intelligence in education, chatbots, automatic question answering, academic dishonesty, prompting strategies, explanation quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
