CoChat
Sign Up
Cited by 397Artificial Intelligence in EducationAcademic integrity and plagiarismArtificial Intelligence in Healthcare and Education

The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.

ChatGPT versus engineering education assessment: a multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity

Open access
Nikolic, Sasha, Daniel, Scott, Haque, Rezwanul, Belkina, Marina, Hassan, Ghulam M., Grundy, Sarah, Lyden, Sarah, Neal, Peter, Sandison, Caz·2023·European Journal of Engineering Education
DOI 10.1080/03043797.2023.2213169
Open this paper in CoChat — free accountOpen PDF

Abstract

ChatGPT, a sophisticated online chatbot, sent shockwaves through many sectors once reports filtered through that it could pass exams. In higher education, it has raised many questions about the authenticity of assessment and challenges in detecting plagiarism. Amongst the resulting frenetic hubbub, hints of potential opportunities in how ChatGPT could support learning and the development of critical thinking have also emerged. In this paper, we examine how ChatGPT may affect assessment in engineering education by exploring ChatGPT responses to existing assessment prompts from ten subjects across seven Australian universities. We explore the strengths and weaknesses of current assessment practice and discuss opportunities on how ChatGPT can be used to facilitate learning. As artificial intelligence is rapidly improving, this analysis sets a benchmark for ChatGPT’s performance as of early 2023 in responding to engineering education assessment prompts. ChatGPT did pass some subjects and excelled with some assessment types. Findings suggest that changes in current practice are needed, as typically with little modification to the input prompts, ChatGPT could generate passable responses to many of the assessments, and it is only going to get better as future versions are trained on larger data sets.

Overview

Integrity signals

Registry

Crossref · 10.1080/03043797.2023.2213169

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

The text we found for this paper looks like another document, so its claims and self-check weren’t run. The registry and citation checks still apply.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The text we found for this paper looks like another document, so the self-check wasn’t run.

Citations

Cites

Research works printed but not linked: 6

  1. Bubeck, S., V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, and S. Lundberg. 2023. Sparks of Artificial General Intelligence: Early Experiments with gpt-4. arXiv preprint arXiv:2303.12712 . · arXiv
  2. Merriam J., 1993, Engineering Mechanics: Volume 1, Statics
  3. Necesal P., 2012, Proceedings of the World Congress on Engineering and Computer Science 2012 Vol I WCECS 2012, 271
  4. Rudolph J., 2023, Journal of Applied Learning and Teaching, 6, 1
  5. Susnjak, T. 2022. ChatGPT: The End of Online Exam Integrity? arXiv preprint arXiv:2212.09292 . · arXiv
  6. White, J., Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering With Chatgpt. arXiv preprint arXiv:2302.11382 . · arXiv

Cited by

Sorted by citations

See all 397

Related

See all

Metadata

Authors
Nikolic, S., Daniel, S., Haque, R., Belkina, M., Hassan, G. M., Grundy, S., Lyden, S., Neal, P., & Sandison, C.
Affiliations
  • University of Wollongong (AU)
  • University of Technology Sydney (AU)
  • University of the Sunshine Coast (AU)
  • Western Sydney University (AU)
  • The University of Western Australia (AU)
  • UNSW Sydney (AU)
  • University of Tasmania (AU)
  • Faculty of Engineering and Information Sciences, University of Wollongong, Wollongong, Australia
Corresponding author
Nikolic, Sasha
ORCID
Venue
European Journal of Engineering Education
Publisher
Informa UK Limited
Published
2023-05-26
Volume
48
Issue
4
Pages
559-614
ISSN
  • 0304-3797 (ISSN-L)
  • 1469-5898
DOI
10.1080/03043797.2023.2213169
OpenAlex
W4378469471
Cited by
336
References
34
Citations by year
2026: 85 · 2025: 118 · 2024: 107 · 2023: 26
FWCI
78.87
Type
journal article
Language
en
Open access
Licence
Locations
  • European Journal of Engineering Education (publishedVersion, cc-by)
  • Research Online (University of Wollongong) (submittedVersion)
  • UWA Profiles and Research Repository (UWA) (submittedVersion, cc-by)
Topics
  • Social Sciences › Education › Artificial Intelligence in Education
  • Social Sciences › Education › Academic integrity and plagiarism
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
ChatGPT, generative AI, engineering education, assessment integrity, academic integrity, plagiarism detection, critical thinking, higher education assessment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review