The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index.
ChatGPT versus engineering education assessment: a multidisciplinary and multi-institutional benchmarking and analysis of this generative artificial intelligence tool to investigate assessment integrity
Open accessAbstract
ChatGPT, a sophisticated online chatbot, sent shockwaves through many sectors once reports filtered through that it could pass exams. In higher education, it has raised many questions about the authenticity of assessment and challenges in detecting plagiarism. Amongst the resulting frenetic hubbub, hints of potential opportunities in how ChatGPT could support learning and the development of critical thinking have also emerged. In this paper, we examine how ChatGPT may affect assessment in engineering education by exploring ChatGPT responses to existing assessment prompts from ten subjects across seven Australian universities. We explore the strengths and weaknesses of current assessment practice and discuss opportunities on how ChatGPT can be used to facilitate learning. As artificial intelligence is rapidly improving, this analysis sets a benchmark for ChatGPT’s performance as of early 2023 in responding to engineering education assessment prompts. ChatGPT did pass some subjects and excelled with some assessment types. Findings suggest that changes in current practice are needed, as typically with little modification to the input prompts, ChatGPT could generate passable responses to many of the assessments, and it is only going to get better as future versions are trained on larger data sets.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023) · JMIR Medical Education · doi:10.2196/45312 · Cited by 2,078
ChatGPT: five priorities for research
van Dis, E. A. M., Bollen, J., Zuidema, W., van Rooij, R., & Bockting, C. L. (2023) · Nature · doi:10.1038/d41586-023-00288-7 · Cited by 1,842 · cited 2×
“The size of training parameters is important because the software uses machine learning to autonomously learn (van Dis et al. 2023).”
MethodThe Role of the Laboratory in Undergraduate Engineering Education
Feisel, L. D., & Rosa, A. J. (2005) · Journal of Engineering Education · doi:10.1002/j.2168-9830.2005.tb00833.x · Cited by 1,692
An Overview of Chatbot Technology
Adamopoulou, E., & Moussiades, L. (2020) · IFIP advances in information and communication technology · doi:10.1007/978-3-030-49186-4_31 · Cited by 1,300
Exploring SWOT analysis – where are we now?
Helms, M. M., & Nixon, J. C. (2010) · Journal of strategy and management · doi:10.1108/17554251011064837 · Cited by 1,001
ChatGPT listed as author on research papers: many scientists disapprove
Stokel-Walker, C. (2023) · Nature · doi:10.1038/d41586-023-00107-z · Cited by 868
Mathematical Capabilities of ChatGPT
Frieder, S., Pinchetti, L., Chevalier, C., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P., & Berner, J. (2023) · neural information processing systems · doi:10.52202/075280-1205 · Cited by 754
Abstracts written by ChatGPT fool scientists
Else, H. (2023) · Nature · doi:10.1038/d41586-023-00056-7 · Cited by 611
ChatGPT Goes to Law SchoolPreprint
Choi, J. H., Hickman, K. E., Monahan, A., & Schwarcz, D. B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4335905 · Cited by 565
Assessment in the age of artificial intelligence
Swiecki, Z., Khosravi, H., Chen, G., Martinez-Maldonado, R., Lodge, J. M., Milligan, S., Selwyn, N., & Gašević, D. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100075 · Cited by 409 · cited 1×
“It has been suggested that AI technology can help educators with the design and implementation of assessment (Swiecki et al. 2022).”
Background
Research works printed but not linked: 6
- Bubeck, S., V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, and S. Lundberg. 2023. Sparks of Artificial General Intelligence: Early Experiments with gpt-4. arXiv preprint arXiv:2303.12712 . · arXiv
- Merriam J., 1993, Engineering Mechanics: Volume 1, Statics
- Necesal P., 2012, Proceedings of the World Congress on Engineering and Computer Science 2012 Vol I WCECS 2012, 271
- Rudolph J., 2023, Journal of Applied Learning and Teaching, 6, 1
- Susnjak, T. 2022. ChatGPT: The End of Online Exam Integrity? arXiv preprint arXiv:2212.09292 . · arXiv
- White, J., Q. Fu, S. Hays, M. Sandborn, C. Olea, H. Gilbert, A. Elnashar, J. Spencer-Smith, and D. C. Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering With Chatgpt. arXiv preprint arXiv:2302.11382 . · arXiv
Cited by
Sorted by citations
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083
Bahroun, Z., Anane, C., Ahmed, V., & Zacca, A. (2023) · Sustainability · doi:10.3390/su151712983 · Cited by 784
A Scoping Review on How Generative Artificial Intelligence Transforms Assessment in Higher Education
Xia, Q., Weng, X., Ouyang, F., Lin, T. J., & Chiu, T. K. F. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00468-z · Cited by 493
ChatGPT: The End of Online Exam Integrity?
Susnjak, T. (2022) · arXiv · doi:10.48550/arxiv.2212.09292 · Cited by 370
“Others [39–41] have recommended integrating multimodal approaches into exams which venture beyond just text-based questions and additionally combine visual components.”
Wamba, S. F., de Queiroz, M. M., Jabbour, C. J. C., & Shi, C. (2023) · International Journal of Production Economics · doi:10.1016/j.ijpe.2023.109015 · Cited by 364
Metadata
- Authors
- Nikolic, S., Daniel, S., Haque, R., Belkina, M., Hassan, G. M., Grundy, S., Lyden, S., Neal, P., & Sandison, C.
- Affiliations
- University of Wollongong (AU)
- University of Technology Sydney (AU)
- University of the Sunshine Coast (AU)
- Western Sydney University (AU)
- The University of Western Australia (AU)
- UNSW Sydney (AU)
- University of Tasmania (AU)
- Faculty of Engineering and Information Sciences, University of Wollongong, Wollongong, Australia
- Corresponding author
- Nikolic, Sasha
- ORCID
- Venue
- European Journal of Engineering Education
- Publisher
- Informa UK Limited
- Published
- 2023-05-26
- Volume
- 48
- Issue
- 4
- Pages
- 559-614
- ISSN
- 0304-3797 (ISSN-L)
- 1469-5898
- OpenAlex
- W4378469471
- Cited by
- 336
- References
- 34
- Citations by year
- 2026: 85 · 2025: 118 · 2024: 107 · 2023: 26
- FWCI
- 78.87
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- European Journal of Engineering Education (publishedVersion, cc-by)
- Research Online (University of Wollongong) (submittedVersion)
- UWA Profiles and Research Repository (UWA) (submittedVersion, cc-by)
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Social Sciences › Education › Academic integrity and plagiarism
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- ChatGPT, generative AI, engineering education, assessment integrity, academic integrity, plagiarism detection, critical thinking, higher education assessment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
