The DOI matches its Crossref record; 23 of the 23 references it cites were found in a registry or an index; none of the 25 sources its claims cite is checked for support yet.
Detecting AI content in responses generated by ChatGPT, YouChat, and Chatsonic: The case of five AI content detection tools
Open accessAbstract
This paper set out to test the accuracy of five AI content tools, GPTZero, OpenAI Text Classifier, Writer.com’s AI Content Detector, Copyleaks AI Content Detector, and Giant Language model Test Room, to detect AI-generated content in the responses generated by ChatGPT, YouChat, and Chatsonic. The responses were generated from these three AI chatbots using English prompts related to applied English language studies. Then, the ChatGPT-generated responses were Google-translated into German, French, Spanish, Southern Sotho, and isiZulu, and inputted into GPTZero for it to detect the AI-generated content in them. Additionally, the ChatGPT-generated responses Google-translated into German, French and Spanish were inputted into Copyleaks AI Content Detector for it to detect the AI-generated content in them. For the ChatGPT-, YouChat-, and Chatsonic-generated responses, Copyleaks AI Content Detector emerged as the top-most performing AI content detector among the five AI content detectors. It was followed by OpenAI’s AI Text Classifier. Concerning the ChatGPT-generated responses that were Google-translated into five languages, GPTZero misidentified all of them as human-produced. For the ChatGPT-generated responses that were Google-translated into German, French and Spanish, Copyleaks AI Content Detector correctly identified three of the German-translated texts, five of the French-translated texts, and all the Spanish-translated texts as AI-generated. Thus, it is evident from this paper that all five AI content detectors seem not yet fully ready to accurately and convincingly detect AI-generated content from machine-generated texts in different contexts. This has dire consequences for AI-generated plagiarism in academic essay writing.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This exploratory study tested five free AI-content detectors on English responses generated by ChatGPT, YouChat, and Chatsonic, plus Google-translated versions of ChatGPT responses. Copyleaks AI Content Detector ranked first for correct English-text identification, while GPTZero misclassified all translated ChatGPT responses in five languages as human-written.
Key findings
- Across the three English response sets, the five detectors ranked by overall correct identification as: Copyleaks AI Content Detector (1), OpenAI AI Text Classifier (2), GPTZero (3), Writer.com’s AI Content Detector (4), and GLTR (5).
- For ChatGPT-generated English responses, Copyleaks correctly classified 5 texts and misclassified 1; GPTZero correctly classified 4 and was indecisive about 2; Writer.com’s detector classified 5 inaccurately as human-generated and 1 accurately; GLTR correctly classified 1 and misclassified 5.
- For YouChat-generated responses, Copyleaks correctly classified 5 and incorrectly classified 2; OpenAI AI Text Classifier correctly classified 3, misclassified 2, and was indecisive about 2; GPTZero correctly classified 1 and misclassified 6; GLTR misidentified all 7 as human-written.
- For Chatsonic-generated responses, Copyleaks correctly classified 5 and misclassified 3; GPTZero correctly classified 3, misclassified 4, and was indecisive about 1; OpenAI AI Text Classifier correctly identified 2, detected 1 incorrectly, and was indecisive about 5; GLTR misclassified 6 and was indecisive about 2.
- GPTZero incorrectly classified all ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and isiZulu as human-written; its highest perplexity scores were 2,478 (German), 221 (French), 361 (Spanish), 1,715 (Southern Sotho), and 938 (isiZulu). Copyleaks correctly classified 3 German-translated texts, 5 French-translated texts, and all Spanish-translated texts as AI-generated.
Methods
- Exploratory study design. Three sets of English prompts in applied English language studies were inputted into ChatGPT, YouChat, and Chatsonic; ChatGPT prompts were queried on 31 January 2023, and YouChat and Chatsonic prompts on 07 March 2023.
- Responses were inputted into five AI content detectors—GPTZero, OpenAI AI Text Classifier, Writer.com’s AI Content Detector, Copyleaks AI Content Detector, and Giant Language Model Test Room (GLTR)—in three phases from 30 March to 02 April 2023.
- Phase 1 fed English-only responses into all five detectors. Phase 2 fed ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and isiZulu into GPTZero. Phase 3 fed German, French, and Spanish translations into Copyleaks AI Content Detector. Analysis compared correct classifications, misclassifications, indecisive results, and reported perplexity or probability percentages.
Limitations
- The paper states a major limitation: it used free-to-use or non-premium versions of the AI content detectors; when data were collected, all five detectors had only free public versions.
- Evident from the text: the dataset is small and limited to prompts from applied English language studies, and only one detector (GPTZero) was used for the five translated languages while Copyleaks was used for three because it did not support Southern Sotho and isiZulu.
- Evident from the text: no texts were deliberately manipulated through editing, paraphrasing, or adding extra spaces, so the study did not test detector robustness against such evasion methods.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
57 Own result · 18 Own interpretation · 14 Not checked yet · 6 No basis given · 2 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The same kind of classification pattern was yielded by OpenAI AI Text Classifier.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
- Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
- Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Its lowest and highest percentages for human-generated content for these texts were 0% and 100%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
- Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
- Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Its lowest and highest percentages for humangenerated content were 1% and 99%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
- Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
- Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only one AI detector (OpenAI AI Text Classifier) had two undecided texts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“OpenAI AI Text Classifier had the most undecided texts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Additionally, the five AI content detectors were not able to distinguish, in clear-cut terms, between AI-generated texts and human-produced texts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPTZero incorrectly classified all the translated texts in all five languages as human-written.”
- Also stated in: § Discussion · plain
“As mentioned earlier, as regards the ChatGPT responses that were Google-translated into the five aforementioned languages, GPTZero misidentified all of them as humanproduced.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and (isi)Zulu remain AI-generated texts for the purpose of detection.
- Also stated in: § Discussion · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It was indecisive about one text.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Writer.com'sAI Content Detector recorded 1% and 99% as its lowest and highest percentages for human-generated content for two texts apiece.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“AI detectors with the joint-most indecisive texts were GPTZero and OpenAI AI Text Classifier.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 35
- Probably not a substantial claim. 17
- The quote only partly states this claim. 2
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Writer.com's AI Content Detector classified five texts inaccurately as human-generated, while its classification of one text was accurate.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
- Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
- Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, GPTZero correctly classified four texts as AI-generated, while it was indecisive about two texts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
- Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
- Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Its lowest and highest perplexity scores for all these texts were 40 and 167.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
- Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
- Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Its lowest and highest perplexity scores were 38 and 90.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
- Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
- Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Pertaining to YouChat-generated responses, GPTZero classified one text correctly as AI-generated, while it misclassified the six other texts as human-written.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
- Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
- Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Its lowest and highest probability percentages for AI-generated texts were 94% and 99.8%, while its probability percentage for humangenerated text was 19.5%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
- Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
- Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, Copyleaks AI Content Detector classified five texts accurately, but classified one text inaccurately.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
- Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
- Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For its part, Copyleaks AI Content Detector correctly classified five texts as AI-generated and incorrectly classified the other two texts as humangenerated.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
- Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
- Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Its lowest and highest probability percentages for AI generated texts were 99.6% and 99.9%, respectively, with three texts having a 99.9% tie.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
- Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
- Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Writer.com'sAI Content Detector correctly classified four texts as AI-generated, but misclassified three texts as human-generated.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
- Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
- Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For GLTR, it correctly classified one text as machine-generated, but misclassified five texts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
- Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
- Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“OpenAI AI Text Classifier classified three texts accurately as AI-generated but misclassified two texts.It was indecisive about two texts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
- Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
- Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Here, its lowest and highest probability percentages for AI-generated texts were 94.9% and 99.9%, with three texts having a tie at 99.9%.Its probability percentage for the human-generated text was 6.9%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and (isi)Zulu remain AI-generated texts for the purpose of detection.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Its lowest and highest probability percentages for these texts were 99. 5% and 99.9%, with two texts and four texts tied at 99.5% and 99.9%, respectively.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and (isi)Zulu remain AI-generated texts for the purpose of detection.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“It, then, correctly identified five French-translated texts as AI-generated but misidentified one text as humanwritten.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and (isi)Zulu remain AI-generated texts for the purpose of detection.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With reference to Chatsonic-generated responses, GPTZero correctly classified three texts as AI-generated, but misclassified four texts as human-generated.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Copyleaks AI Content Detector's highest probability percentage of 98.6% for human-generated text was the highest ever for these machine-generated texts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GLTR had the highest text misclassification rate and could classify only one text correctly.As such, it ranked last among the five AI content detectors.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Its lowest and highest probability percentages for AI-generated texts were 92.3% and 99.9%, with three texts having a 99.9% tie.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Again, in contrast to the other four AI detectors, GLTR misclassified six texts, while it was indecisive about two texts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Copyleaks AI Content Detector correctly classified five texts as AI-generated but misclassified three texts as human-written.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Its high and lowest perplexity scores for the texts translated into each of these languages were as follows: 110 and 2,478 (German); 57 and 221 (French); 108 and 361 (Spanish); 602 and 1,715 (Southern Sotho); and 651 and 938 ((isi)Zulu).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and (isi)Zulu remain AI-generated texts for the purpose of detection.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both Writer.com'sAI Content Detector and GLTR had the most joint misidentified texts, followed by GPTZero.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This AI content detector correctly classified three German-translated texts as AI-generated but misclassified three texts as human-written.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and (isi)Zulu remain AI-generated texts for the purpose of detection.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, Copyleaks AI Content Detector still had the most correct identifications of the texts in this set, but with GPTZero following it.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“OpenAI AI Text Classifier identified two texts correctly as AI-generated but detected one text incorrectly.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Writer.com'sAI Content Detector correctly identified two texts as AI-generated but misidentified six texts as human-written.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In relation to YouChat-generated responses, again, Copyleaks AI Content Detector had more correct identifications of the texts in this set than the other four AI detectors.It was followed by OpenAI AI Text Classifier.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Copyleaks AI Content Detector recorded 19.4% and 19.8% as its lowest and highest probability percentages for human-generated texts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Its lowest and highest probability percentages for AI-generated texts were 83.7% and 99.9%, while its lowest and highest probability percentages for human-generated texts were 14.8% and 53.4%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and (isi)Zulu remain AI-generated texts for the purpose of detection.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Writer.com'sAI Content Detector recorded the contrasting lowest and highest percentages of 2% and 100% for humangenerated content.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For ChatGPT-generated responses, Copyleaks AI Content Detector had more correct classifications of the texts than the other four AI detectors.It was followed by GPTZero and OpenAI AI Text Classifier, which were joint second.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Its lowest and highest probability percentages for human-generated texts were 19.4% and 19.8%.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
- Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
- Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In terms of misclassifications of texts (incorrect classifications of texts), GLTR topped the other four AI detectors with eighteen misclassifications; it was followed by Writer.com'sAI Content Detector with fourteen misclassifications.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The other AI detectors had more misclassifications of texts than the correct classifications, with GLTR racking up the most misclassified texts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Some of these AI content detectors, such as GPTZero and OpenAI AI Text Classifier, were specifically intended to detect AI-generated content from ChatGPT (Eliaçik, 2023a, b;Ismael, 2023;Iyer, 2023: Kirchner et al., 2023;Lim, 2023;Ofgang, 2033).”
What the sources say
- Not checked yetcites Eliaçik, 2023a, b
- Not checked yetcites Ismael, 2023
- Not checked yetcites Iyer, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The fact that the five AI content detectors recognised some of the AI-generated texts inputted to them as human-written points to their propensity to false negativity.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This, more than what has been said earlier, highlights the shakiness and, at times, the unreliability of a higher perplexity as an indicator of human-only-written texts.This is more so given that machine-translated texts can have inordinately higher perplexity scores, such as the ones for the five languages in this paper.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“In an instance in which seven AI text detectors, which included OpenAI AI Text Classifier, GPTZero and Copyleaks, were tested to detect AI-generated content created by Claude (a generative AI tool similar to ChatGPT), GPTZero was the top consistent performer.It was followed by ChatGPT and OpenAI AI Text Classifier.”
What the sources say
- Not checked yetcites Wiggers, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This means it identified the first text as 99% AI-generated, while it recognised the second text to be 1% AI-generated.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Again, here, Copyleaks AI Content Detector outperformed GPTZero, an outcome that contrasts with that of Wiggers' testing (2023) of seven AI content detectors in which GPTZero was a top performer.”
What the sources say
- Not checked yetcites Wiggers, 2023
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“If this is the case, this points to one of the inherent problems with current AI detection tools: their lack of reliability and consistency in accurately differentiating between AI-generated texts and human-produced texts.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“which, again, is a high score for texts that were machine-generated.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“As mentioned earlier, Aremu's (2023) paper that tested the detection capabilities of six AI detection tools found both ZeroGPT and GPTZero to have a higher level of deception robustness and resistance than the other four AI detection tools.”
What the sources say
- Not checked yetcites Aremu, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These are two extremely contrasting detection rates for these texts when considering that all the texts in this set were ChatGPT-generated.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As is the case with the previous instance, these are two diametrically opposed detection rates for texts that were ChatGPT-generated.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Again, these are high probability percentages for machine-generated texts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“For instance, Cai and Cui (2023) found that effecting a mere single space, what they call an extra space, results in text detection evasion.”
What the sources say
- Not checked yetcites Cai & Cui, 2023
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“which is a bit high for texts that were exclusively machine-generated.”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“which is a very high score for texts that were machine-generated.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In addition, Guo et al. (2023) point out that by removing indicating words such as Nope, My take is, and Hmm, from human-written content and by removing I regret to hear that, I'm an AI assistant, and Here are steps to follow, from an AI-generated content, most AI detection tools are likely to be tricked in their detection capability.”
What the sources say
- Not checked yetcites Guo et al., 2023
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These inherent algorithmic shortcomings are likely to be there in the premium versions of these AI detection tools as well.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But it seems implausible that the premiums versions of these AI detection tools are free of the two algorithmic shortcomings mentioned above.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
The paper relies on this without support.
The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
“The prompts for these three AI chatbots were based on some of the aspects of applied English language studies (AELS).”
“The three sets of AI-generated English responses and the ChatGPT-generated English responses that were Google-translated into the five languages specified above constituted the datasets for this study.”
The paper relies on this without support.
The Google Translate translations of the ChatGPT-generated English responses into German, French, Spanish, Southern Sotho, and isiZulu accurately preserve the content such that detection results on the translated texts reflect the tools' ability to detect AI content in those languages.
“In the second phase, the ChatGPT-generated responses were machinetranslated into five languages using Google Translate and inputted into GPTZero.”
The paper relies on this without support.
The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
“All of the five AI content detectors reviewed above were employed in this paper in their free-to-use or non-premium versions.”
The paper relies on this without support.
The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
“The idea of classified is a proxy for predicted as these tools attempt to predict whether a given text response is AI-or human-generated more than just classifying a given text.”
The paper relies on this without support.
The ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and (isi)Zulu remain AI-generated texts for the purpose of detection.
“All the ChatGPTgenerated responses were subjected to GPTZero for it to detect if they were AI-generated or human-written (see Table 2).”
“all the ChatGPT-generated responses, which were Google-translated into the three languages as mentioned above, were subjected to Copyleaks AI Content Detector for it to detect whether they were AI-generated or not (see Table 3).”
The paper relies on this without support.
The accuracy and correctness of the Google-translated texts is not relevant to testing the detectors' ability to classify them as machine-generated.
“In this section, what is at issue is not the accuracy and correctness of the Google-translated texts for all five languages but rather GPTZero's ability to classify them correctly as machine-generated texts.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Despite advancements in AI detection tools, the effectiveness of conventional anti-plagiarism software, such as iThenticate and Turnitin, remains questionable against AI-generated content (Perkins, 2023;Chaka, 2023Chaka, , 2024;;Hassoulas et al., 2023;Ifelebuegu, 2023;Mohammadkarimi et al., 2023;Sullivan et al., 2023).”
From Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania (2024) · cites this paper as Chaka, 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The detectors were chosen based on a review of existing studies (Chaka, 2023a; Gao etal., 2022; Krishna etal., 2023; Orenstrakh etal., 2023; Perkins, 2023; Weber-Wulff etal., 2023) as follows:”
From Simple techniques to bypass GenAI text detectors: implications for inclusive education (2024) · cites this paper as Chaka, 2023a
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Predominantly, contemporary detectors designed for LLM-generated text primarily target monolingual applications, often neglecting to evaluate and optimize performance across multiple languages.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Chaka (2023)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Wang et al. (2023b) and Chaka (2023) noted the lack of control observed in multilingual LLM-generated text detectors across various languages, despite the existence of certain language migration capabilities.”
From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Chaka (2023)
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“However, claims of accuracy are contradicted by studies which demonstrate the varied levels of the detectors' ability to distinguish accurately between AI and human-generated content. (Chaka, 2023a; Gao etal., 2022; Krishna etal., 2023; Orenstrakh etal., 2023; Perkins etal., 2023; Walters, 2023; Weber-Wulff etal., 2023).”
From Simple techniques to bypass GenAI text detectors: implications for inclusive education (2024) · cites this paper as Chaka, 2023a
In the citing paper
“Turnitin and other similar plagiarism detection tools, such as ZeroGPT, GPTZero, and Copyleaks, have since evolved, incorporating enhanced techniques to detect AI-generated text, despite the possibility of false positives, as noted in different studies that have found these tools still not yet fully ready to accurately and reliably identify AI-generated text [10, 51]”
From A systematic literature review of empirical research on ChatGPT in education (2024) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Discussion
“GPTZero had the highest perplexity score of 167, which, again, is a high score for texts that were machine-generated.”
§ Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors
“It recorded the lowest and highest perplexity scores for these eight texts as 32 and 118.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Discussion
“For Copyleaks AI Content Detector, its probability percentage for a humangenerated text was 19.4%, which is a bit high for texts that were exclusively machine-generated.”
§ Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors
“Its lowest and highest probability percentages for AI-generated texts were 94% and 99.8%, while its probability percentage for humangenerated text was 19.5%.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Detection by GPTZero of the ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and (isi)Zulu
“Its high and lowest perplexity scores for the texts translated into each of these languages were as follows: 110 and 2,478 (German); 57 and 221 (French); 108 and 361 (Spanish); 602 and 1,715 (Southern Sotho); and 651 and 938 ((isi)Zulu).”
§ Discussion
“This is particularly the case with the German, Southern Sotho and (isi)Zulu texts, whose highest perplexity scores were 2,478, 1,715 and 938, respectively.”
Citations
Cites
Chatting and cheating: Ensuring academic integrity in the era of ChatGPT
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2190148 · Cited by 3,087
Chatting and Cheating. Ensuring academic integrity in the era of ChatGPT
Cotton, D., Cotton, P., & Shipway, J. R. (2023) · doi:10.35542/osf.io/mrz8h · Cited by 3,079
ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.9 · Cited by 1,806
ChatGPT in higher education: considerations for academic integrity and student learning
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.17 · Cited by 853
Perkins, M. (2023) · Journal of University Teaching and Learning Practice · doi:10.53761/1.20.02.07 · Cited by 839
The role of ChatGPT in higher education: Benefits, challenges, and future research directions
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.29 · Cited by 561
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.23 · Cited by 421
How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection
Guo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y., Yue, J., & Wu, Y. (2023) · arXiv · doi:10.48550/arxiv.2301.07597 · Cited by 318
Will ChatGPT get you caught? Rethinking of Plagiarism Detection
Khalil, M., & Er, E. (2023) · doi:10.35542/osf.io/fnh48 · Cited by 259
The death of the short-form physics essay in the coming AI revolution
Yeadon, W., Inyang, O.-O., Mizouri, A., Peach, A., & Testrow, C. P. (2023) · Physics Education · doi:10.1088/1361-6552/acc5cf · Cited by 199
Research works printed but not linked: 2
- Gerring, J., Mahomey, J., & Elman, C. (2020). Introduction. In C. Elman, J. Gerring & J. Mahoney (Eds.), The production of knowledge: Enhancing progress in social science (pp. 1-14). Cambridge University Press.
- Chaka Chaka. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. org/10.1038/d41586-022-04397-7. · DOI · Paper page
Cited by
Sorted by citations
Generative AI Literacy: Twelve Defining Competencies
Annapureddy, R., Fornaroli, A., & Gatica-Perez, D. (2025) · Digital Government: Research and Practice · doi:10.1145/3685680 · Cited by 333
“In recent years, there has been a surge in software and AI models to detect AI-generated content, like text ( ZeroGPT , 9 GPTZero , 10 PlagiarismCheck , 11 GPT Radar , 12 and Writer.com , 13 etc.) [18, 66, 97], images ( AI or Not 14 or Is it AI?”
Perkins, M., Furze, L., Roe, J., & MacVaugh, J. (2024) · Journal of University Teaching and Learning Practice · doi:10.53761/q3azde36 · Cited by 291
A systematic literature review of empirical research on ChatGPT in education
Albadarin, Y., Saqr, M., Pope, N., & Tukiainen, M. (2024) · Discover Education · doi:10.1007/s44217-024-00138-2 · Cited by 191
A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions
Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025) · Computational Linguistics · doi:10.1162/coli_a_00549 · Cited by 212
Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania
Rudolph, J., Mohamed Ismail, F. M., & Popenici, S. (2024) · Journal of University Teaching and Learning Practice · doi:10.53761/54fs5e77 · Cited by 189
“…in AI detection tools, the effectiveness of conventional anti-plagiarism software, such as iThenticate and Turnitin, remains questionable against AI-generated content (Perkins, 2023; Chaka, 2023, 2024; Hassoulas et al., 2023; Ifelebuegu, 2023; Mohammadkarimi et al., 2023; Sullivan et al., 2023).”
Method
Metadata
- Authors
- Chaka, C.
- Affiliations
- University of South Africa (ZA)
- Corresponding author
- Chaka Chaka
- ORCID
- Chaka Chaka
- Venue
- Journal of Applied Learning & Teaching
- Publisher
- Kaplan Business School, Australia
- Published
- 2023-07-24
- Volume
- 6
- Issue
- 2
- ISSN
- 2591-801X (ISSN-L)
- OpenAlex
- W4385156842
- Semantic Scholar
- 3c5f627391edb49a0a24e83f27eebb5a63a64d04
- Cited by
- 132
- References
- 22
- Citations by year
- 2026: 20 · 2025: 43 · 2024: 60 · 2023: 9
- FWCI
- 33.10
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Journal of Applied Learning & Teaching (publishedVersion, cc-by)
- Topics
- Social Sciences › Education › Academic integrity and plagiarism
- Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- AI-generated text detection, ChatGPT, GPTZero, multilingual translation, machine translation, academic plagiarism, English language teaching
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-09, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-09 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
