CoChat
Sign Up
Cited by 136Academic integrity and plagiarismAuthorship Attribution and ProfilingArtificial Intelligence in Education

Detecting AI content in responses generated by ChatGPT, YouChat, and Chatsonic: The case of five AI content detection tools

Open access
Chaka Chaka·2023·Journal of Applied Learning & Teaching
DOI 10.37074/jalt.2023.6.2.12
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper set out to test the accuracy of five AI content tools, GPTZero, OpenAI Text Classifier, Writer.com’s AI Content Detector, Copyleaks AI Content Detector, and Giant Language model Test Room, to detect AI-generated content in the responses generated by ChatGPT, YouChat, and Chatsonic. The responses were generated from these three AI chatbots using English prompts related to applied English language studies. Then, the ChatGPT-generated responses were Google-translated into German, French, Spanish, Southern Sotho, and isiZulu, and inputted into GPTZero for it to detect the AI-generated content in them. Additionally, the ChatGPT-generated responses Google-translated into German, French and Spanish were inputted into Copyleaks AI Content Detector for it to detect the AI-generated content in them. For the ChatGPT-, YouChat-, and Chatsonic-generated responses, Copyleaks AI Content Detector emerged as the top-most performing AI content detector among the five AI content detectors. It was followed by OpenAI’s AI Text Classifier. Concerning the ChatGPT-generated responses that were Google-translated into five languages, GPTZero misidentified all of them as human-produced. For the ChatGPT-generated responses that were Google-translated into German, French and Spanish, Copyleaks AI Content Detector correctly identified three of the German-translated texts, five of the French-translated texts, and all the Spanish-translated texts as AI-generated. Thus, it is evident from this paper that all five AI content detectors seem not yet fully ready to accurately and convincingly detect AI-generated content from machine-generated texts in different contexts. This has dire consequences for AI-generated plagiarism in academic essay writing.

AI snapshot

Written from the paper’s full text.

TL;DR

This exploratory study tested five free AI-content detectors on English responses generated by ChatGPT, YouChat, and Chatsonic, plus Google-translated versions of ChatGPT responses. Copyleaks AI Content Detector ranked first for correct English-text identification, while GPTZero misclassified all translated ChatGPT responses in five languages as human-written.

Key findings

  • Across the three English response sets, the five detectors ranked by overall correct identification as: Copyleaks AI Content Detector (1), OpenAI AI Text Classifier (2), GPTZero (3), Writer.com’s AI Content Detector (4), and GLTR (5).
  • For ChatGPT-generated English responses, Copyleaks correctly classified 5 texts and misclassified 1; GPTZero correctly classified 4 and was indecisive about 2; Writer.com’s detector classified 5 inaccurately as human-generated and 1 accurately; GLTR correctly classified 1 and misclassified 5.
  • For YouChat-generated responses, Copyleaks correctly classified 5 and incorrectly classified 2; OpenAI AI Text Classifier correctly classified 3, misclassified 2, and was indecisive about 2; GPTZero correctly classified 1 and misclassified 6; GLTR misidentified all 7 as human-written.
  • For Chatsonic-generated responses, Copyleaks correctly classified 5 and misclassified 3; GPTZero correctly classified 3, misclassified 4, and was indecisive about 1; OpenAI AI Text Classifier correctly identified 2, detected 1 incorrectly, and was indecisive about 5; GLTR misclassified 6 and was indecisive about 2.
  • GPTZero incorrectly classified all ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and isiZulu as human-written; its highest perplexity scores were 2,478 (German), 221 (French), 361 (Spanish), 1,715 (Southern Sotho), and 938 (isiZulu). Copyleaks correctly classified 3 German-translated texts, 5 French-translated texts, and all Spanish-translated texts as AI-generated.

Methods

  • Exploratory study design. Three sets of English prompts in applied English language studies were inputted into ChatGPT, YouChat, and Chatsonic; ChatGPT prompts were queried on 31 January 2023, and YouChat and Chatsonic prompts on 07 March 2023.
  • Responses were inputted into five AI content detectors—GPTZero, OpenAI AI Text Classifier, Writer.com’s AI Content Detector, Copyleaks AI Content Detector, and Giant Language Model Test Room (GLTR)—in three phases from 30 March to 02 April 2023.
  • Phase 1 fed English-only responses into all five detectors. Phase 2 fed ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and isiZulu into GPTZero. Phase 3 fed German, French, and Spanish translations into Copyleaks AI Content Detector. Analysis compared correct classifications, misclassifications, indecisive results, and reported perplexity or probability percentages.

Limitations

  • The paper states a major limitation: it used free-to-use or non-premium versions of the AI content detectors; when data were collected, all five detectors had only free public versions.
  • Evident from the text: the dataset is small and limited to prompts from applied English language studies, and only one detector (GPTZero) was used for the five translated languages while Copyleaks was used for three because it did not support Southern Sotho and isiZulu.
  • Evident from the text: no texts were deliberately manipulated through editing, paraphrasing, or adding extra spaces, so the study did not test detector robustness against such evasion methods.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.37074/jalt.2023.6.2.12

Cites

23 of 23 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 25 cited sources checked · 0 supported · 0 flagged

  • Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    For ChatGPT-generated responses, OpenAI AI Text Classifier yielded the same classification pattern as GPTZero.

    “The same kind of classification pattern was yielded by OpenAI AI Text Classifier.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
    • Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
    • Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
  • Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    Writer.com's AI Content Detector's lowest and highest percentages for human-generated content for YouChat-generated texts were 0% and 100%.

    “Its lowest and highest percentages for human-generated content for these texts were 0% and 100%.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
    • Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
    • Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
  • Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    Writer.com's AI Content Detector's lowest and highest percentages for human-generated content for ChatGPT-generated texts were 1% and 99%.

    “Its lowest and highest percentages for humangenerated content were 1% and 99%.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
    • Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
    • Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.
  • Findingown data
    § Discussion

    Only one AI detector (OpenAI AI Text Classifier) had two undecided texts for YouChat-generated responses.

    “Only one AI detector (OpenAI AI Text Classifier) had two undecided texts.”
    Quote in the paperNot a substantial claim
  • Findingown data
    § Discussion

    OpenAI AI Text Classifier had the most undecided texts for Chatsonic responses.

    “OpenAI AI Text Classifier had the most undecided texts.”
    Quote in the paperNot a substantial claim
  • May combine two claims. 35
  • Classified as probably not a substantial claim. 17
  • The quote only partly states this claim. 2

  • May combine two claims.

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    For ChatGPT-generated responses, Writer.com's AI Content Detector classified five texts inaccurately as human-generated, while its classification of one text was accurate.

    “Writer.com's AI Content Detector classified five texts inaccurately as human-generated, while its classification of one text was accurate.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
    • Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
    • Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.

  • May combine two claims.

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    For ChatGPT-generated responses, GPTZero correctly classified four texts as AI-generated and was indecisive about two texts.

    “For example, GPTZero correctly classified four texts as AI-generated, while it was indecisive about two texts.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
    • Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
    • Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    GPTZero's lowest and highest perplexity scores for YouChat-generated texts were 40 and 167.

    “Its lowest and highest perplexity scores for all these texts were 40 and 167.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
    • Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
    • Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    GPTZero's lowest and highest perplexity scores for ChatGPT-generated texts were 38 and 90.

    “Its lowest and highest perplexity scores were 38 and 90.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
    • Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
    • Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.

  • May combine two claims.

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    For YouChat-generated responses, GPTZero classified one text correctly as AI-generated, while it misclassified the six other texts as human-written.

    “Pertaining to YouChat-generated responses, GPTZero classified one text correctly as AI-generated, while it misclassified the six other texts as human-written.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
    • Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
    • Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.

  • May combine two claims.

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    Copyleaks AI Content Detector's lowest and highest probability percentages for AI-generated ChatGPT texts were 94% and 99.8%, while its probability percentage for human-generated text was 19.5%.

    “Its lowest and highest probability percentages for AI-generated texts were 94% and 99.8%, while its probability percentage for humangenerated text was 19.5%.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
    • Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
    • Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.

  • May combine two claims.

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    For ChatGPT-generated responses, Copyleaks AI Content Detector classified five texts accurately, but classified one text inaccurately.

    “In contrast, Copyleaks AI Content Detector classified five texts accurately, but classified one text inaccurately.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
    • Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
    • Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.

  • May combine two claims.

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    For YouChat-generated responses, Copyleaks AI Content Detector correctly classified five texts as AI-generated and incorrectly classified the other two texts as human-generated.

    “For its part, Copyleaks AI Content Detector correctly classified five texts as AI-generated and incorrectly classified the other two texts as humangenerated.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
    • Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
    • Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.

  • May combine two claims.

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    Copyleaks AI Content Detector's lowest and highest probability percentages for AI-generated YouChat texts were 99.6% and 99.9%, respectively, with three texts having a 99.9% tie.

    “Its lowest and highest probability percentages for AI generated texts were 99.6% and 99.9%, respectively, with three texts having a 99.9% tie.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
    • Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
    • Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.

  • May combine two claims.

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    For YouChat-generated responses, Writer.com's AI Content Detector correctly classified four texts as AI-generated, but misclassified three texts as human-generated.

    “Writer.com'sAI Content Detector correctly classified four texts as AI-generated, but misclassified three texts as human-generated.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
    • Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
    • Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.

  • May combine two claims.

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    For ChatGPT-generated responses, GLTR correctly classified one text as machine-generated, but misclassified five texts.

    “For GLTR, it correctly classified one text as machine-generated, but misclassified five texts.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
    • Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
    • Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.

  • May combine two claims.

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    For YouChat-generated responses, OpenAI AI Text Classifier classified three texts accurately as AI-generated but misclassified two texts, and was indecisive about two texts.

    “OpenAI AI Text Classifier classified three texts accurately as AI-generated but misclassified two texts.It was indecisive about two texts.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
    • Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
    • Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.

  • May combine two claims.

    Findingown data
    § Detection by Copyleaks AI Content Detector of the ChatGPT responses Google-translated into German, French, and Spanish

    Copyleaks' lowest and highest probability percentages for AI-generated French-translated texts were 94.9% and 99.9%, with three texts tied at 99.9%; its probability for the human-generated text was 6.9%.

    “Here, its lowest and highest probability percentages for AI-generated texts were 94.9% and 99.9%, with three texts having a tie at 99.9%.Its probability percentage for the human-generated text was 6.9%.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and (isi)Zulu remain AI-generated texts for the purpose of detection.

  • May combine two claims.

    Findingown data
    § Detection by Copyleaks AI Content Detector of the ChatGPT responses Google-translated into German, French, and Spanish

    Copyleaks' lowest and highest probability percentages for Spanish-translated texts were 99.5% and 99.9%, with two texts and four texts tied at 99.5% and 99.9%, respectively.

    “Its lowest and highest probability percentages for these texts were 99. 5% and 99.9%, with two texts and four texts tied at 99.5% and 99.9%, respectively.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and (isi)Zulu remain AI-generated texts for the purpose of detection.

  • May combine two claims.

    Findingown data
    § Detection by Copyleaks AI Content Detector of the ChatGPT responses Google-translated into German, French, and Spanish

    Copyleaks correctly identified five French-translated ChatGPT texts as AI-generated but misidentified one as human-written.

    “It, then, correctly identified five French-translated texts as AI-generated but misidentified one text as humanwritten.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and (isi)Zulu remain AI-generated texts for the purpose of detection.

  • May combine two claims.

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    For Chatsonic-generated responses, GPTZero correctly classified three texts as AI-generated but misclassified four texts as human-generated.

    “With reference to Chatsonic-generated responses, GPTZero correctly classified three texts as AI-generated, but misclassified four texts as human-generated.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given
    § Discussion

    Copyleaks AI Content Detector's highest probability percentage of 98.6% for human-generated text was the highest ever for these machine-generated texts.

    “Copyleaks AI Content Detector's highest probability percentage of 98.6% for human-generated text was the highest ever for these machine-generated texts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Discussion

    GLTR had the highest text misclassification rate and could classify only one text correctly, and ranked last among the five.

    “GLTR had the highest text misclassification rate and could classify only one text correctly.As such, it ranked last among the five AI content detectors.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    Copyleaks AI Content Detector's lowest and highest probability percentages for AI-generated Chatsonic texts were 92.3% and 99.9%, with three texts having a 99.9% tie.

    “Its lowest and highest probability percentages for AI-generated texts were 92.3% and 99.9%, with three texts having a 99.9% tie.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    For Chatsonic-generated responses, GLTR misclassified six texts and was indecisive about two texts.

    “Again, in contrast to the other four AI detectors, GLTR misclassified six texts, while it was indecisive about two texts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    For Chatsonic-generated responses, Copyleaks AI Content Detector correctly classified five texts as AI-generated but misclassified three texts as human-written.

    “Copyleaks AI Content Detector correctly classified five texts as AI-generated but misclassified three texts as human-written.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Detection by GPTZero of the ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and (isi)Zulu

    GPTZero's highest and lowest perplexity scores for the translated texts were: German 110 and 2,478; French 57 and 221; Spanish 108 and 361; Southern Sotho 602 and 1,715; (isi)Zulu 651 and 938.

    “Its high and lowest perplexity scores for the texts translated into each of these languages were as follows: 110 and 2,478 (German); 57 and 221 (French); 108 and 361 (Spanish); 602 and 1,715 (Southern Sotho); and 651 and 938 ((isi)Zulu).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and (isi)Zulu remain AI-generated texts for the purpose of detection.

  • May combine two claims.

    Findingown data
    § Discussion

    Both Writer.com's AI Content Detector and GLTR had the most joint misidentified texts for Chatsonic responses, followed by GPTZero.

    “Both Writer.com'sAI Content Detector and GLTR had the most joint misidentified texts, followed by GPTZero.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Detection by Copyleaks AI Content Detector of the ChatGPT responses Google-translated into German, French, and Spanish

    Copyleaks correctly classified three German-translated ChatGPT texts as AI-generated but misclassified three as human-written.

    “This AI content detector correctly classified three German-translated texts as AI-generated but misclassified three texts as human-written.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and (isi)Zulu remain AI-generated texts for the purpose of detection.

  • May combine two claims.

    Findingown data
    § Discussion

    Concerning Chatsonic-generated responses, Copyleaks had the most correct identifications, with GPTZero following it.

    “For example, Copyleaks AI Content Detector still had the most correct identifications of the texts in this set, but with GPTZero following it.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    For Chatsonic-generated responses, OpenAI AI Text Classifier identified two texts correctly as AI-generated but detected one text incorrectly.

    “OpenAI AI Text Classifier identified two texts correctly as AI-generated but detected one text incorrectly.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    For Chatsonic-generated responses, Writer.com's AI Content Detector correctly identified two texts as AI-generated but misidentified six texts as human-written.

    “Writer.com'sAI Content Detector correctly identified two texts as AI-generated but misidentified six texts as human-written.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Discussion

    For YouChat-generated responses, Copyleaks had more correct identifications than the other four AI detectors, followed by OpenAI AI Text Classifier.

    “In relation to YouChat-generated responses, again, Copyleaks AI Content Detector had more correct identifications of the texts in this set than the other four AI detectors.It was followed by OpenAI AI Text Classifier.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data
    § Discussion

    Copyleaks AI Content Detector recorded 19.4% and 19.8% as its lowest and highest probability percentages for human-generated YouChat texts.

    “Copyleaks AI Content Detector recorded 19.4% and 19.8% as its lowest and highest probability percentages for human-generated texts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Detection by Copyleaks AI Content Detector of the ChatGPT responses Google-translated into German, French, and Spanish

    Copyleaks' lowest and highest probability percentages for AI-generated German-translated texts were 83.7% and 99.9%, and for human-generated texts were 14.8% and 53.4%.

    “Its lowest and highest probability percentages for AI-generated texts were 83.7% and 99.9%, while its lowest and highest probability percentages for human-generated texts were 14.8% and 53.4%.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and (isi)Zulu remain AI-generated texts for the purpose of detection.

  • The quote only partly states this claim.

    Findingown data
    § Discussion

    Writer.com's AI Content Detector recorded the contrasting lowest and highest percentages of 2% and 100% for human-generated Chatsonic texts.

    “Writer.com'sAI Content Detector recorded the contrasting lowest and highest percentages of 2% and 100% for humangenerated content.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Discussion

    For ChatGPT-generated responses, Copyleaks had more correct classifications than the other four AI detectors, followed by GPTZero and OpenAI AI Text Classifier joint second.

    “For ChatGPT-generated responses, Copyleaks AI Content Detector had more correct classifications of the texts than the other four AI detectors.It was followed by GPTZero and OpenAI AI Text Classifier, which were joint second.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Detection of the ChatGPT-, YouChat-, and Chatsonicgenerated English responses by five AI content detectors

    Copyleaks AI Content Detector's lowest and highest probability percentages for human-generated YouChat texts were 19.4% and 19.8%.

    “Its lowest and highest probability percentages for human-generated texts were 19.4% and 19.8%.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.
    • Rests on an assumption: The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.
    • Rests on an assumption: The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.

  • May combine two claims.

    Findingown data
    § Discussion

    In misclassifications of texts, GLTR topped the other four AI detectors with eighteen misclassifications, followed by Writer.com's AI Content Detector with fourteen misclassifications.

    “In terms of misclassifications of texts (incorrect classifications of texts), GLTR topped the other four AI detectors with eighteen misclassifications; it was followed by Writer.com'sAI Content Detector with fourteen misclassifications.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Discussion

    For YouChat-generated responses, the other AI detectors had more misclassifications than correct classifications, with GLTR racking up the most misclassified texts.

    “The other AI detectors had more misclassifications of texts than the correct classifications, with GLTR racking up the most misclassified texts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Implications and limitations

    Some of these AI content detectors, such as GPTZero and OpenAI AI Text Classifier, were specifically intended to detect AI-generated content from ChatGPT.

    “Some of these AI content detectors, such as GPTZero and OpenAI AI Text Classifier, were specifically intended to detect AI-generated content from ChatGPT (Eliaçik, 2023a, b;Ismael, 2023;Iyer, 2023: Kirchner et al., 2023;Lim, 2023;Ofgang, 2033).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Eliaçik, 2023a, b

      Not checked in the free run.

    • cites Ismael, 2023

      Not checked in the free run.

    • cites Iyer, 2023

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data
    § Discussion

    The five AI content detectors recognised some AI-generated texts as human-written, pointing to their propensity to false negativity.

    “The fact that the five AI content detectors recognised some of the AI-generated texts inputted to them as human-written points to their propensity to false negativity.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § Discussion

    Higher perplexity as an indicator of human-only-written texts is shaky and at times unreliable, especially given that machine-translated texts can have inordinately higher perplexity scores.

    “This, more than what has been said earlier, highlights the shakiness and, at times, the unreliability of a higher perplexity as an indicator of human-only-written texts.This is more so given that machine-translated texts can have inordinately higher perplexity scores, such as the ones for the five languages in this paper.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § Discussion

    In an instance where seven AI text detectors, including OpenAI AI Text Classifier, GPTZero and Copyleaks, were tested to detect AI-generated content created by Claude, GPTZero was the top consistent performer, followed by ChatGPT and OpenAI AI Text Classifier.

    “In an instance in which seven AI text detectors, which included OpenAI AI Text Classifier, GPTZero and Copyleaks, were tested to detect AI-generated content created by Claude (a generative AI tool similar to ChatGPT), GPTZero was the top consistent performer.It was followed by ChatGPT and OpenAI AI Text Classifier.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wiggers, 2023

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data
    § Discussion

    Writer.com's AI Content Detector identified the first ChatGPT text as 99% AI-generated and the second as 1% AI-generated.

    “This means it identified the first text as 99% AI-generated, while it recognised the second text to be 1% AI-generated.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data
    § Discussion

    Copyleaks outperformed GPTZero on the translated texts, contrasting with Wiggers' testing where GPTZero was a top performer.

    “Again, here, Copyleaks AI Content Detector outperformed GPTZero, an outcome that contrasts with that of Wiggers' testing (2023) of seven AI content detectors in which GPTZero was a top performer.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Wiggers, 2023

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation
    § Discussion

    If text manipulation reduces detection efficacy, this points to one of the inherent problems with current AI detection tools: their lack of reliability and consistency in accurately differentiating between AI-generated texts and human-produced texts.

    “If this is the case, this points to one of the inherent problems with current AI detection tools: their lack of reliability and consistency in accurately differentiating between AI-generated texts and human-produced texts.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § Discussion

    A perplexity score of 167 is a high score for texts that were machine-generated.

    “which, again, is a high score for texts that were machine-generated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Discussion

    Aremu's (2023) paper found both ZeroGPT and GPTZero to have a higher level of deception robustness and resistance than the other four AI detection tools.

    “As mentioned earlier, Aremu's (2023) paper that tested the detection capabilities of six AI detection tools found both ZeroGPT and GPTZero to have a higher level of deception robustness and resistance than the other four AI detection tools.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Aremu, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data
    § Discussion

    These are two extremely contrasting detection rates for ChatGPT-generated texts.

    “These are two extremely contrasting detection rates for these texts when considering that all the texts in this set were ChatGPT-generated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data
    § Discussion

    These are two diametrically opposed detection rates for texts that were ChatGPT-generated.

    “As is the case with the previous instance, these are two diametrically opposed detection rates for texts that were ChatGPT-generated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § Discussion

    These are high probability percentages for machine-generated texts.

    “Again, these are high probability percentages for machine-generated texts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Discussion

    Cai and Cui (2023) found that effecting a mere single space, what they call an extra space, results in text detection evasion.

    “For instance, Cai and Cui (2023) found that effecting a mere single space, what they call an extra space, results in text detection evasion.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Cai & Cui, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data
    § Discussion

    A probability percentage of 19.4% for a human-generated text is a bit high for texts that were exclusively machine-generated.

    “which is a bit high for texts that were exclusively machine-generated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given
    § Discussion

    A perplexity score of 167 is very high for texts that were machine-generated.

    “which is a very high score for texts that were machine-generated.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Discussion

    Guo et al. (2023) point out that by removing indicating words from human-written content and from AI-generated content, most AI detection tools are likely to be tricked in their detection capability.

    “In addition, Guo et al. (2023) point out that by removing indicating words such as Nope, My take is, and Hmm, from human-written content and by removing I regret to hear that, I'm an AI assistant, and Here are steps to follow, from an AI-generated content, most AI detection tools are likely to be tricked in their detection capability.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Guo et al., 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning
    § Discussion

    These inherent algorithmic shortcomings are likely to be there in the premium versions of these AI detection tools as well.

    “These inherent algorithmic shortcomings are likely to be there in the premium versions of these AI detection tools as well.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning
    § Implications and limitations

    It seems implausible that the premium versions of these AI detection tools are free of the two algorithmic shortcomings mentioned above.

    “But it seems implausible that the premiums versions of these AI detection tools are free of the two algorithmic shortcomings mentioned above.”
    Quote in the paperNot a substantial claim
53 of 53

  • The paper relies on this without support.

    The AI-generated responses from ChatGPT, YouChat, and Chatsonic, elicited by the specific prompts used, are representative of AI-generated academic content in general.

    “The prompts for these three AI chatbots were based on some of the aspects of applied English language studies (AELS).”
    “The three sets of AI-generated English responses and the ChatGPT-generated English responses that were Google-translated into the five languages specified above constituted the datasets for this study.”

  • The paper relies on this without support.

    The Google Translate translations of the ChatGPT-generated English responses into German, French, Spanish, Southern Sotho, and isiZulu accurately preserve the content such that detection results on the translated texts reflect the tools' ability to detect AI content in those languages.

    “In the second phase, the ChatGPT-generated responses were machinetranslated into five languages using Google Translate and inputted into GPTZero.”

  • The paper relies on this without support.

    The free-to-use or non-premium versions of the five AI content detectors are equivalent to their premium versions for the purpose of evaluating their accuracy.

    “All of the five AI content detectors reviewed above were employed in this paper in their free-to-use or non-premium versions.”

  • The paper relies on this without support.

    The classification outputs of the AI content detection tools (e.g., AI-generated vs human-generated, perplexity scores, probability percentages) are valid indicators of their accuracy in detecting AI-generated content.

    “The idea of classified is a proxy for predicted as these tools attempt to predict whether a given text response is AI-or human-generated more than just classifying a given text.”

  • The paper relies on this without support.

    The ChatGPT-generated responses Google-translated into German, French, Spanish, Southern Sotho, and (isi)Zulu remain AI-generated texts for the purpose of detection.

    “All the ChatGPTgenerated responses were subjected to GPTZero for it to detect if they were AI-generated or human-written (see Table 2).”
    “all the ChatGPT-generated responses, which were Google-translated into the three languages as mentioned above, were subjected to Copyleaks AI Content Detector for it to detect whether they were AI-generated or not (see Table 3).”

  • The paper relies on this without support.

    The accuracy and correctness of the Google-translated texts is not relevant to testing the detectors' ability to classify them as machine-generated.

    “In this section, what is at issue is not the accuracy and correctness of the Google-translated texts for all five languages but rather GPTZero's ability to classify them correctly as machine-generated texts.”
6 of 6

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The effectiveness of conventional anti-plagiarism software, such as iThenticate and Turnitin, remains questionable against AI-generated content.

    “Despite advancements in AI detection tools, the effectiveness of conventional anti-plagiarism software, such as iThenticate and Turnitin, remains questionable against AI-generated content (Perkins, 2023;Chaka, 2023Chaka, , 2024;;Hassoulas et al., 2023;Ifelebuegu, 2023;Mohammadkarimi et al., 2023;Sullivan et al., 2023).”

    From Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania (2024) · cites this paper as Chaka, 2023

    The Literature review (page unavailable) reports that a study including Turnitin found AI-detection tools unsuitable as evidence of academic misconduct and vulnerable to paraphrasing and translation, supporting doubts about their effectiveness. But it also says Turnitin ranked first for detection accuracy; it does not assess iThenticate or establish that conventional plagiarism-checking functions are ineffective against AI-generated content. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Seven AI text detectors were selected for testing because prior literature identified them as somewhat effective at identifying GenAI content.

    “The detectors were chosen based on a review of existing studies (Chaka, 2023a; Gao etal., 2022; Krishna etal., 2023; Orenstrakh etal., 2023; Perkins, 2023; Weber-Wulff etal., 2023) as follows:”

    From Simple techniques to bypass GenAI text detectors: implications for inclusive education (2024) · cites this paper as Chaka, 2023a

    The Detection section (page unavailable) reports that Chaka tested five AI content detectors, and the Discussion reports that some performed better than others. This makes the paper relevant to a literature-based choice of detectors, but it does not establish that all seven detectors in the citing study were somewhat effective or explain that study’s selection decision. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Contemporary detectors designed for LLM-generated text primarily target monolingual applications, often neglecting to evaluate and optimize performance across multiple languages.

    “Predominantly, contemporary detectors designed for LLM-generated text primarily target monolingual applications, often neglecting to evaluate and optimize performance across multiple languages.”

    From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Chaka (2023)

    The 'ChatGPT's prompts' section says GPTZero was the only detector in the study that recognized all five tested languages, while Copyleaks did not support Southern Sotho or isiZulu. This supports limited language coverage among the tested tools, but the paper does not establish that contemporary detectors predominantly target monolingual applications or that their developers neglect multilingual evaluation and optimization. The paper itself evaluates detection across several languages. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Multilingual LLM-generated text detectors lack control across various languages, despite the existence of certain language migration capabilities.

    “Wang et al. (2023b) and Chaka (2023) noted the lack of control observed in multilingual LLM-generated text detectors across various languages, despite the existence of certain language migration capabilities.”

    From A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions (2025) · cites this paper as Chaka (2023)

    The GPTZero detection section reports that it misclassified all ChatGPT responses translated into five languages as human-written. The Copyleaks detection section reports mixed results across three languages, including correct classification of every Spanish text. These findings support concerns about cross-language detection, but the paper does not establish a general lack of control across multilingual detectors or discuss “language migration capabilities.” (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Claims of accuracy by AI text detector providers are contradicted by studies demonstrating varied levels of detectors' ability to distinguish accurately between AI and human-generated content.

    “However, claims of accuracy are contradicted by studies which demonstrate the varied levels of the detectors' ability to distinguish accurately between AI and human-generated content. (Chaka, 2023a; Gao etal., 2022; Krishna etal., 2023; Orenstrakh etal., 2023; Perkins etal., 2023; Walters, 2023; Weber-Wulff etal., 2023).”

    From Simple techniques to bypass GenAI text detectors: implications for inclusive education (2024) · cites this paper as Chaka, 2023a

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Chatsonic perplexity maximum changes from 118 to 167

    Minor

    The results report 118 as GPTZero's highest perplexity score for Chatsonic texts, whereas the discussion reports 167. The results assign 167 to the YouChat texts instead.

  • Conflicting Copyleaks percentage for a ChatGPT-generated text

    Minor

    The results give 19.5% for Copyleaks' human-generated classification in the ChatGPT set, but the discussion gives 19.4%.

  • Translated-text perplexity limits are reversed

    Minor

    The results present 2,478 as the German texts' lowest score by listing it second after "high and lowest," but the discussion explicitly identifies 2,478 as their highest score. The same reversal applies to the Southern Sotho and (isi)Zulu figures.

3 of 3

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Gerring, J., Mahomey, J., & Elman, C. (2020). Introduction. In C. Elman, J. Gerring & J. Mahoney (Eds.), The production of knowledge: Enhancing progress in social science (pp. 1-14). Cambridge University Press.
  2. Chaka Chaka. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms. org/10.1038/d41586-022-04397-7. · DOI · Paper page

Cited by

Sorted by citations

Related

Authors
Chaka, C.
Affiliations
University of South Africa (ZA)
Corresponding author
Chaka Chaka
ORCID
Chaka Chaka
Venue
Journal of Applied Learning & Teaching
Publisher
Kaplan Business School, Australia
Published
2023-07-24
Volume
6
Issue
2
ISSN
2591-801X (ISSN-L)
DOI
10.37074/jalt.2023.6.2.12
OpenAlex
W4385156842
Semantic Scholar
3c5f627391edb49a0a24e83f27eebb5a63a64d04
Cited by
132
References
22
Citations by year
2026: 20 · 2025: 43 · 2024: 60 · 2023: 9
FWCI
33.10
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
Journal of Applied Learning & Teaching (publishedVersion, cc-by)
Topics
  • Social Sciences › Safety Research › Academic integrity and plagiarism
  • Physical Sciences › Computer Science › Artificial Intelligence › Authorship Attribution and Profiling
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
Keywords
AI-generated text detection, ChatGPT, GPTZero, multilingual translation, machine translation, academic plagiarism, English language teaching

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review