Academics' perceptions of ChatGPT-generated written outputs: A practical application of Turing’s Imitation Game
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
Artificial intelligence (AI) technology, such as Chat Generative Pre-trained Transformer (ChatGPT), is evolving quickly and having a significant impact on the higher education sector. Although the impact of ChatGPT on academic integrity processes is a key concern, little is known about whether academics can reliably recognise texts that have been generated by AI. This qualitative study applies Turing’s Imitation Game to investigate 16 education academics’ perceptions of two pairs of texts written by either ChatGPT or a human. Pairs of texts, written in response to the same task, were used as the stimulus for interviews that probed academics’ perceptions of text authorship and the textual features that were important in their decision-making. Results indicated academics were only able to identify AI-generated texts half of the time, highlighting the sophistication of contemporary generative AI technology. Academics perceived the following categories as important for their decision-making: voice, word usage, structure, task achievement and flow. All five categories of decision-making were variously used to rationalise both accurate and inaccurate decisions about text authorship. The implications of these results are discussed with a particular focus on what strategies can be applied to support academics more effectively as they manage the ongoing challenge of AI in higher education. Implications for practice or policy: Experienced academics may be unable to distinguish between texts written by contemporary generative AI technology and humans. Academics are uncertain about the current capabilities of generative AI and need support in redesigning assessments that succeed in providing robust evidence of student achievement of learning outcomes. Institutions must assess the adequacy of their assessment designs, AI use policies, and AI-related procedures to enhance students’ capacity for effective and ethical use of generative AI technology.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 5×
“Turing’s Imitation Game Turing’s (1950) seminal article “ Computing Machinery and Intelligence ” presented a criterion for assessing the humanness of the communicative artefacts produced by computers.”
BackgroundInfluentialChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · cited 2×
“Current generative AI applications, underpinned by large language learning models, can complete various tasks, such as producing highly integrated human-like academic texts (Kasneci, et al. 2023).”
BackgroundMethodZawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-019-0171-0 · Cited by 7,242
GPT-3: Its Nature, Scope, Limits, and Consequences
Floridi, L., & Chiriatti, M. (2020) · Minds and Machines · doi:10.1007/s11023-020-09548-1 · Cited by 2,519
Exploring the impact of artificial intelligence on teaching and learning in higher education
Popenici, S. A. D., & Kerr, S. (2017) · Research and Practice in Technology Enhanced Learning · doi:10.1186/s41039-017-0062-8 · Cited by 2,019 · cited 1×
“Although relatively recent commentary predicted that generative AI would rapidly become an intrinsic component of higher education (Popenici & Kerr, 2017), the speed at which it has impacted the sector has been unanticipated (Lodge, Thompson, & Corrin, 2023).”
BackgroundChatGPT: Bullshit spewer or the end of traditional assessments in higher education?
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.9 · Cited by 1,806 · cited 1×
“One application of current relevance is Chat Generative Pre-trained Transformer (ChatGPT) (Kasneci et al., 2023; Lodge, Thompson, & Corrin, 2023; Rudolph et al., 2023), which is freely available to anyone with access to the internet (https://chat.openai.com).”
MethodGao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00819-6 · Cited by 1,099
ChatGPT in higher education: considerations for academic integrity and student learning
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.17 · Cited by 853 · cited 2×
“Although AI-mediated technology, such as ChatGPT, represents an important opportunity for higher education (Ouyang et al., 2022), it also represents a significant disruption (Dawson, 2021; Lodge, Thompson, & Corrin, 2023; Sullivan et al. 2023).”
Background
Cited by
Ahmed, Z., Shanto, S. S., Rime, M. H. K., Morol, M. K., Fahad, N., Hossen, M. J., & Abdullah-Al-Jubair, M. (2024) · IEEE Access · doi:10.1109/access.2024.3461874 · Cited by 123
Nikolic, S., Wentworth, I., Sheridan, L., Moss, S., Duursma, E., Jones, R. A., Ros, M., & Middleton, R. (2024) · Australasian Journal of Educational Technology · doi:10.14742/ajet.9643 · Cited by 102
“Essentially, AI is fast becoming part of the fabric of education (Matthews & Volpe, 2023, p. 82).”
Zhang, X., Zhang, P., Shen, Y., Liu, M., Wang, Q., Gašević, D., & Fan, Y. (2024) · Frontiers of Digital Education · doi:10.1007/s44366-024-0028-5 · Cited by 83
AI in Tertiary Education: Progress on Research and Practice
Thompson, K., Corrin, L., & Lodge, J. M. (2023) · Australasian Journal of Educational Technology · doi:10.14742/ajet.9251 · Cited by 31
Kangwa, D., Msafiri, M. M., & Fute, A. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.70109 · Cited by 22
Metadata
- Authors
- A Matthews, J., & Volpe, C. R.
- Venue
- Australasian Journal of Educational Technology
- Published
- 2023-12-22
- Volume
- 39
- Issue
- 5
- Pages
- 82-100
- ISSN
- 1449-3098 (ISSN-L)
- 1449-5554
- OpenAlex
- W4390099336
- Cited by
- 41
- References
- 15
- FWCI
- 9.63
- Type
- journal article
- Language
- en
- Open access
- diamond
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Academics' perceptions of ChatGPT-generated written outputs: A practical application of Turing’s Imitation Game. https://app.cochat.ai/papers/10.14742/ajet.8896
- The paper
- A Matthews, J., & Volpe, C. R. (2023). Academics' perceptions of ChatGPT-generated written outputs: A practical application of Turing’s Imitation Game. Australasian Journal of Educational Technology. https://doi.org/10.14742/ajet.8896
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.