Cited by 397Artificial Intelligence in EducationAcademic integrity and plagiarismArtificial Intelligence in Healthcare and Education

ChatGPT versus engineering education assessment: a multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity

Open access
Nikolic, Sasha, Daniel, Scott, Haque, Rezwanul, Belkina, Marina, Hassan, Ghulam M., Grundy, Sarah, Lyden, Sarah, Neal, Peter +1 more·2023·European Journal of Engineering Education

Abstract

ChatGPT, a sophisticated online chatbot, sent shockwaves through many sectors once reports filtered through that it could pass exams. In higher education, it has raised many questions about the authenticity of assessment and challenges in detecting plagiarism. Amongst the resulting frenetic hubbub, hints of potential opportunities in how ChatGPT could support learning and the development of critical thinking have also emerged. In this paper, we examine how ChatGPT may affect assessment in engineering education by exploring ChatGPT responses to existing assessment prompts from ten subjects across seven Australian universities. We explore the strengths and weaknesses of current assessment practice and discuss opportunities on how ChatGPT can be used to facilitate learning. As artificial intelligence is rapidly improving, this analysis sets a benchmark for ChatGPT’s performance as of early 2023 in responding to engineering education assessment prompts. ChatGPT did pass some subjects and excelled with some assessment types. Findings suggest that changes in current practice are needed, as typically with little modification to the input prompts, ChatGPT could generate passable responses to many of the assessments, and it is only going to get better as future versions are trained on larger data sets.

Integrity signals

Registry

Crossref · 10.1080/03043797.2023.2213169

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

The text we found for this paper looks like another document, so its claims and self-check weren’t run. The registry and citation checks still apply.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The text we found for this paper looks like another document, so the self-check wasn’t run.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Nikolic, S., Daniel, S., Haque, R., Belkina, M., Hassan, G. M., Grundy, S., Lyden, S., Neal, P., & Sandison, C.
Affiliations
  • University of Wollongong (AU)
  • University of Technology Sydney (AU)
  • University of the Sunshine Coast (AU)
  • Western Sydney University (AU)
  • The University of Western Australia (AU)
  • UNSW Sydney (AU)
  • University of Tasmania (AU)
  • Faculty of Engineering and Information Sciences, University of Wollongong, Wollongong, Australia
Corresponding author
Nikolic, Sasha
Venue
European Journal of Engineering Education
Published
2023-05-26
Volume
48
Issue
4
Pages
559-614
ISSN
  • 0304-3797 (ISSN-L)
  • 1469-5898
OpenAlex
W4378469471
Cited by
336
References
34
Citations by year
2026: 85 · 2025: 118 · 2024: 107 · 2023: 26
FWCI
78.87
Type
journal article
Language
en
Open access
Locations
  • European Journal of Engineering Education (publishedVersion, cc-by)
  • Research Online (University of Wollongong) (submittedVersion)
  • UWA Profiles and Research Repository (UWA) (submittedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Social Sciences › Safety Research › Academic integrity and plagiarism
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
ChatGPT, generative AI, engineering education, assessment integrity, academic integrity, plagiarism detection, critical thinking, higher education assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review