CoChat
Sign Up
Cited by 189

The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.

ChatGPT: The End of Online Exam Integrity?

Susnjak, Teo, McIntosh, Timothy·2024·Education Sciences
DOI 10.3390/educsci14060656
Open this paper in CoChat — free account

Abstract

This study addresses the significant challenge posed by the use of Large Language Models (LLMs) such as ChatGPT on the integrity of online examinations, focusing on how these models can undermine academic honesty by demonstrating their latent and advanced reasoning capabilities. An iterative self-reflective strategy was developed for invoking critical thinking and higher-order reasoning in LLMs when responding to complex multimodal exam questions involving both visual and textual data. The proposed strategy was demonstrated and evaluated on real exam questions by subject experts and the performance of ChatGPT (GPT-4) with vision was estimated on an additional dataset of 600 text descriptions of multimodal exam questions. The results indicate that the proposed self-reflective strategy can invoke latent multi-hop reasoning capabilities within LLMs, effectively steering them towards correct answers by integrating critical thinking from each modality into the final response. Meanwhile, ChatGPT demonstrated considerable proficiency in being able to answer multimodal exam questions across 12 subjects. These findings challenge prior assertions about the limitations of LLMs in multimodal reasoning and emphasise the need for robust online exam security measures such as advanced proctoring systems and more sophisticated multimodal exam questions to mitigate potential academic misconduct enabled by AI technologies.

Overview

Integrity signals

Registry

Crossref · 10.3390/educsci14060656

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Research works printed but not linked: 48

  1. Crawford, 2020, A systematic review of online examinations: A pedagogical innovation for scalable authentication and integrity, Comput. Educ., 159, 104024, 10.1016/j.compedu.2020.104024 · DOI
  2. Henderson, 2022, Factors associated with online examination cheating, Assess. Eval. High. Educ., 48, 980, 10.1080/02602938.2022.2144802 · DOI · Paper page
  3. Dumulescu, 2021, Academic leadership in the time of COVID-19—Experiences and perspectives, Front. Psychol., 12, 648344, 10.3389/fpsyg.2021.648344 · DOI
  4. Whisenhunt, 2022, Maximizing learning while minimizing cheating: New evidence and advice for online multiple-choice exams, Scholarsh. Teach. Learn. Psychol., 8, 140, 10.1037/stl0000242 · DOI
  5. Garg, 2022, A systematic literature review on online assessment security: Current challenges and integrity strategies, Comput. Secur., 113, 102544, 10.1016/j.cose.2021.102544 · DOI
  6. Arnold, 2016, Cheating at online formative tests: Does it pay off?, Internet High. Educ., 29, 98, 10.1016/j.iheduc.2016.02.001 · DOI
  7. Ahsan, 2021, Contract cheating in higher education: A systematic literature review and future research agenda, Assess. Eval. High. Educ., 47, 523, 10.1080/02602938.2021.1931660 · DOI
  8. Crook, 2021, How internet essay mill websites portray the student experience of higher education, Internet High. Educ., 48, 100775, 10.1016/j.iheduc.2020.100775 · DOI
  9. Allen, 2023, A systemic model of academic (mis) conduct to curb cheating in higher education, High. Educ., 87, 1529, 10.1007/s10734-023-01077-x · DOI
  10. Henderson, 2023, The temptation to cheat in online exams: Moving beyond the binary discourse of cheating and not cheating, Int. J. Educ. Integr., 19, 21, 10.1007/s40979-023-00143-2 · DOI
  11. Malik, 2023, Impact of academic cheating and perceived online learning effectiveness on academic performance during the COVID-19 pandemic among Pakistani students, Front. Psychol., 14, 1124095, 10.3389/fpsyg.2023.1124095 · DOI
  12. Newton, 2023, How common is cheating in online exams and did it increase during the COVID-19 pandemic? A systematic review, J. Acad. Ethics, 22, 323, 10.1007/s10805-023-09485-5 · DOI
  13. Handal, 2001, Academic dishonesty at the graduate level, Ethics Behav., 11, 287, 10.1207/S15327019EB1103_7 · DOI
  14. Newton, P., and Xiromeriti, M. (2023). ChatGPT performance on multiple choice question examinations in higher education. A pragmatic scoping review. Assess. Eval. High. Educ., 1–18. · DOI
  15. Ji, Z., Yu, T., Xu, Y., Lee, N., Ishii, E., and Fung, P. (2023). Towards Mitigating Hallucination in Large Language Models via Self-Reflection. arXiv. · DOI
  16. Farazouli, 2024, Hello GPT! Goodbye home examination? An exploratory study of AI chatbots impact on university teachers’ assessment practices, Assess. Eval. High. Educ., 49, 363, 10.1080/02602938.2023.2241676 · DOI · Paper page
  17. Richards, 2024, Bob or Bot: Exploring ChatGPT’s Answers to University Computer Science Assessment, ACM Trans. Comput. Educ., 24, 1, 10.1145/3633287 · DOI · Paper page
  18. Chan, 2023, Unproctored online exams provide meaningful assessment of student learning, Proc. Natl. Acad. Sci. USA, 120, e2302020120, 10.1073/pnas.2302020120 · DOI
  19. Martino, A., Iannelli, M., and Truong, C. (June, January 28). Knowledge injection to counter large language model (LLM) hallucination. Proceedings of the European Semantic Web Conference 2023, Hersonissos, Greece. · DOI
  20. Yao, J.Y., Ning, K.P., Liu, Z.H., Ning, M.N., and Yuan, L. (2023). LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examples. arXiv.
  21. Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., and Chen, Y. (2023). Siren’s Song in the AI Ocean: A Survey on Hallucination in Large Language Models. arXiv.
  22. Chang, Y.C., Wang, X., Wang, J., Wu, Y., Zhu, K., Chen, H., Yang, L., Yi, X., Wang, C., and Wang, Y. (2023). A Survey on Evaluation of Large Language Models. arXiv.
  23. McKenna, N., Li, T., Cheng, L., Hosseini, M.J., Johnson, M., and Steedman, M. (2023). Sources of Hallucination by Large Language Models on Inference Tasks. arXiv. · DOI · Paper page
  24. Liu, H., Ning, R., Teng, Z., Liu, J., Zhou, Q., and Zhang, Y. (2023). Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4. arXiv.
  25. Schultz, 2022, Perils and promise of online exams, Nat. Rev. Chem., 6, 299, 10.1038/s41570-022-00385-7 · DOI
  26. Alessio, 2017, Examining the effect of proctoring on online test scores, Online Learn., 21, 146, 10.24059/olj.v21i1.885 · DOI
  27. Han, 2023, Digital proctoring in higher education: A systematic literature review, Int. J. Educ. Manag., 38, 265
  28. AlSaad, 2023, Large language models in medical education: Opportunities, challenges, and future directions, JMIR Med. Educ., 9, e48291, 10.2196/48291 · DOI · Paper page
  29. Stribling, 2024, The model student: GPT-4 performance on graduate biomedical science exams, Sci. Rep., 14, 5670, 10.1038/s41598-024-55568-7 · DOI
  30. Zhang, 2024, M3exam A multilingual, multimodal, multilevel benchmark for examining large language models, Adv. Neural Inf. Process. Syst., 36, 5484
  31. Sadasivan, V.S., Kumar, A., Balasubramanian, S., Wang, W., and Feizi, S. (2023). Can AI-Generated Text be Reliably Detected?. arXiv.
  32. Orenstrakh, M.S., Karnalim, O., Suarez, C.A., and Liut, M. (2023). Detecting LLM-Generated Text in Computing Education: A Comparative Study for ChatGPT Cases. arXiv.
  33. Kumarage, T., Agrawal, G., Sheth, P., Moraffah, R., Chadha, A., Garland, J., and Liu, H. (2024). A Survey of AI-generated Text Forensic Systems: Detection, Attribution, and Characterization. arXiv.
  34. Pal, A., and Sankarasubbu, M. (2024). Gemini Goes to Med School: Exploring the Capabilities of Multimodal Large Language Models on Medical Challenge Problems & Hallucinations. arXiv. · DOI
  35. Nori, H., King, N., McKinney, S., Carignan, D., and Horvitz, E. (2023). Capabilities of GPT-4 on Medical Challenge Problems. arXiv.
  36. Stechly, K., Marquez, M., and Kambhampati, S. (2023). GPT-4 Doesn’t Know It’s Wrong: An Analysis of Iterative Prompting for Reasoning Problems. arXiv.
  37. Du, Y., Li, S., Torralba, A., Tenenbaum, J., and Mordatch, I. (2023). Improving Factuality and Reasoning in Language Models through Multiagent Debate. arXiv.
  38. Luo, J., Xiao, C., and Ma, F. (2023). Zero-Resource Hallucination Prevention for Large Language Models. arXiv.
  39. Creswell, A., and Shanahan, M. (2022). Faithful Reasoning Using Large Language Models. arXiv.
  40. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. (2023). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv.
  41. Yeadon, W., and Halliday, D.P. (2023). Exploring durham university physics exams with large language models. arXiv.
  42. Singla, A. (2023, January 7–11). Evaluating ChatGPT and GPT-4 for Visual Programming. Proceedings of the 2023 ACM Conference on International Computing Education Research, Chicago, IL, USA. · DOI
  43. Zheng, C., Liu, Z., Xie, E., Li, Z., and Li, Y. (2023). Progressive-Hint Prompting Improves Reasoning in Large Language Models. arXiv.
  44. Han, 2024, Inductive reasoning in humans and large language models, Cogn. Syst. Res., 83, 101155, 10.1016/j.cogsys.2023.101155 · DOI
  45. Luo, L., Lin, Z., Liu, Y., Shu, L., Zhu, Y., Shang, J., and Meng, L. (2023). Critique ability of large language models. arXiv.
  46. Feng, T.H., Denny, P., Wuensche, B., Luxton-Reilly, A., and Hooper, S. (February, January 29). More Than Meets the AI: Evaluating the performance of GPT-4 on Computer Graphics assessment questions. Proceedings of the 26th Australasian Computing Education Conference, Sydney, NSW, Australia.
  47. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017, January 4–9). Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems 30 (NIPS 2017), Long Beach, CA, USA.
  48. Melkonian, H., Bending, Z., and Tomossy, G. (2022, January 28–30). Viva Voce Assessment—Legal Education for the Real World. Proceedings of the 2022 Professional Legal Education Conference: LawTech, Newlaw and NetZero: Preparing for an Uncertain Future, Gold Coast, QLD, Australia. Conference Program.

Cited by

Sorted by citations

See all 189

Metadata

Authors
Susnjak, T., & McIntosh, T.
Affiliations
  • School of Mathematical and Computational Sciences, Massey University, Auckland 0632, New Zealand
  • School of Computing, Engineering and Mathematical Science, LaTrobe University, Melbourne, VIC 3086, Australia
ORCID
Venue
Education Sciences
Publisher
MDPI AG
Published
2024-06-17
Volume
14
Issue
6
Pages
656
ISSN
2227-7102
DOI
10.3390/educsci14060656
Cited by
189
References
14
Language
en
Licence
creativecommons.org/licenses/by/4.0/ (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review