CoChat
Sign Up
Cited by 275Artificial Intelligence in EducationSoftware Engineering ResearchTeaching and Learning Programming

The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.

ChatGPT and Software Testing Education: Promises & Perils

Open access
Jalil, Sajed, Rafi, Suzzana, LaToza, Thomas D., Moran, Kevin, Lam, Wing·2023·2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)
DOI 10.1109/icstw58534.2023.00078
Open this paper in CoChat — free accountOpen PDF

Abstract

Over the past decade, predictive language modeling for code has proven to be a valuable tool for enabling new forms of automation for developers. More recently, we have seen the ad-vent of general purpose "large language models", based on neural transformer architectures, that have been trained on massive datasets of human written text, which includes code and natural language. However, despite the demonstrated representational power of such models, interacting with them has historically been constrained to specific task settings, limiting their general applicability. Many of these limitations were recently overcome with the introduction of ChatGPT, a language model created by OpenAI and trained to operate as a conversational agent, enabling it to answer questions and respond to a wide variety of commands from end users.The introduction of models, such as ChatGPT, has already spurred fervent discussion from educators, ranging from fear that students could use these AI tools to circumvent learning, to excitement about the new types of learning opportunities that they might unlock. However, given the nascent nature of these tools, we currently lack fundamental knowledge related to how well they perform in different educational settings, and the potential promise (or danger) that they might pose to traditional forms of instruction. As such, in this paper, we examine how well ChatGPT performs when tasked with answering common questions in a popular software testing curriculum. We found that given its current capabilities, ChatGPT is able to respond to 77.5% of the questions we examined and that, of these questions, it is able to provide correct or partially correct answers in 55.6% of cases, provide correct or partially correct explanations of answers in 53.0% of cases, and that prompting the tool in a shared question context leads to a marginally higher rate of correct answers and explanations. Based on these findings, we discuss the potential promises and perils related to the use of ChatGPT by students and instructors.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/icstw58534.2023.00078

Cites

27 of 27 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 23 references

Research works printed but not linked: 1

  1. Puryear, 2022, Github CoPilot in the Classroom: Learning to Code with AI Assistance, CCSC

Cited by

Sorted by citations

Items 201–275 of 275

Related

See all

Metadata

Authors
Jalil, S., Rafi, S., LaToza, T. D., Moran, K., & Lam, W.
Affiliations
George Mason University (US)
ORCID
Venue
2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)
Publisher
IEEE
Published
2023-04
Pages
4130-4137
DOI
10.1109/icstw58534.2023.00078
OpenAlex
W4378676756
Semantic Scholar
c1014f86c8d801c93b37eed58311da5f9c9da2e4
Cited by
270
References
23
Citations by year
2026: 34 · 2025: 62 · 2024: 104 · 2023: 69
FWCI
69.13
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Education › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
ChatGPT, large language models, software testing education, artificial intelligence in education, chatbots, automatic question answering, academic dishonesty, prompting strategies, explanation quality

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review