CoChat
Sign Up
Cited by 157Academic integrity and plagiarismArtificial Intelligence in EducationArtificial Intelligence in Healthcare and Education

Perception, performance, and detectability of conversational artificial intelligence across 32 university courses

Open access
Ibrahim, Hazem, Liu, Fengyuan, Asim, Rohail, Battu, Balaraju, Benabderrahmane, Sidahmed, Alhafni, Bashar, Adnan, Wifag, Alhanai, Tuka +31 more·2023·Scientific Reports
DOI 10.1038/s41598-023-38964-3PMC PMC10449897
Open this paper in CoChat — free account

Abstract

The emergence of large language models has led to the development of powerful tools such as ChatGPT that can produce text indistinguishable from human-generated work. With the increasing accessibility of such technology, students across the globe may utilize it to help with their school work—a possibility that has sparked ample discussion on the integrity of student evaluation processes in the age of artificial intelligence (AI). To date, it is unclear how such tools perform compared to students on university-level courses across various disciplines. Further, students’ perspectives regarding the use of such tools in school work, and educators’ perspectives on treating their use as plagiarism, remain unknown. Here, we compare the performance of the state-of-the-art tool, ChatGPT, against that of students on 32 university-level courses. We also assess the degree to which its use can be detected by two classifiers designed specifically for this purpose. Additionally, we conduct a global survey across five countries, as well as a more in-depth survey at the authors’ institution, to discern students’ and educators’ perceptions of ChatGPT’s use in school work. We find that ChatGPT’s performance is comparable, if not superior, to that of students in a multitude of courses. Moreover, current AI-text classifiers cannot reliably detect ChatGPT’s use in school work, due to both their propensity to classify human-written answers as AI-generated, as well as the relative ease with which AI-generated text can be edited to evade detection. Finally, there seems to be an emerging consensus among students to use the tool, and among educators to treat its use as plagiarism. Our findings offer insights that could guide policy discussions addressing the integration of artificial intelligence into educational frameworks.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper compares ChatGPT against students across 32 university courses, surveys student and educator perceptions in five countries and at NYUAD, and tests whether AI-text classifiers can detect ChatGPT use. It finds ChatGPT often performs comparably or superior to students, current detectors are unreliable, and a simple paraphrasing attack evades detection almost entirely.

Key findings

  • ChatGPT’s performance was comparable or superior to students on nine of 32 courses in the Results (Data Structures; Introduction to Public Policy; Quantitative Synthetic Biology; Cyberwarfare; Object Oriented Programming; Structure and Properties of Civil Engineering Materials; Biopsychology; Climate/Change; Management and Organizations); the Discussion states 12 (38%) of 32. The largest gaps favoring students were for math-related and trick questions, while ChatGPT outperformed students only on factual-knowledge questions.
  • Detection was poor: OpenAI’s Text Classifier misclassified 5% of student submissions as AI-generated and 49% of ChatGPT submissions as human-generated; GPTZero had an 18% false-positive rate and 32% false-negative rate. After the Quillbot attack, false-negative rates rose from 49% to 98% for OpenAI’s classifier and from 32% to 95% for GPTZero.
  • In the global survey (N=1601 across Brazil, India, Japan, UK, US), 74% of students said they would use ChatGPT for their studies; main reasons were improving skills and saving time, while non-users mainly cited not knowing how or no need. OLS results showed students in Brazil and India were significantly more likely, and students in Japan significantly less likely, to use ChatGPT than those in the USA; prior use was associated with future use, and poor/working-class students were more likely to indicate they would use it.
  • At NYUAD, 57% of surveyed students planned to use ChatGPT for assignments and 64% expected peers to do so; 61% believed it should be used and 55% expected peers to believe so. Among professors, 69% planned to treat ChatGPT use as plagiarism and 71% expected others to do so; 72% believed it should be treated as plagiarism and 73% expected peers to believe so.

Methods

  • Course comparison: NYUAD faculty provided 10 text-based questions per course and three randomly chosen student answers per question; ChatGPT generated three distinct answers per question. Submissions were randomized and graded by three graders per course, with faculty annotating questions by Anderson and Krathwohl taxonomy dimensions and question type.
  • Surveys: a global survey recruited participants from Brazil, India, Japan, the UK, and the US, targeting at least 200 students and 100 educators per country (N=1601). A second NYUAD survey included 151 students (7.2% of 2103) and 60 professors (15.2% of 393), with OLS regression used to explore factors associated with students’ decision to use ChatGPT.
  • Detection: GPTZero and OpenAI’s AI text classifier were tested on 280 questions after excluding 40 math-only questions; 47 code-snippet questions were excluded from the Quillbot attack. Normalized confusion matrices quantified false-positive and false-negative rates before and after paraphrasing.

Limitations

  • Paper-stated: student and educator sentiments were measured in the two months after ChatGPT’s release and may evolve; large language models and detection classifiers will continue to improve, so future work should examine newer models and lower educational levels.
  • Evident from text: inter-rater reliability was below 0.6 for six of 32 courses; four were essay-based and subjective, while two were not, though excluding the two non-essay courses gave qualitatively similar results.
  • Evident from text: the survey relies on self-reports, and the NYUAD response rates were low (7.2% of students and 15.2% of professors). The text also reports nine comparable/superior courses in Results but 12 (38%) in the Discussion, an inconsistency.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41598-023-38964-3

Cites

34 of 34 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 17 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Simply hearing about ChatGPT is not significantly associated with students' decision to use it to assist with their studies.

    “On the contrary, simply hearing about ChatGPT is not significantly associated with the students’ decision to use it to assist with their studies.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Students who have used ChatGPT are significantly more likely to use it again.

    “As for prior experience with ChatGPT, those who have used it are significantly more likely to use it again.”
    Quote in the paperNot a substantial claim
  • Findingown data

    The nine courses on which ChatGPT's performance was comparable to or surpassed students were Data Structures, Introduction to Public Policy, Quantitative Synthetic Biology, Cyberwarfare, Object Oriented Programming, Structure and Properties of Civil Engineering Materials, Biopsychology, Climate/Change, and Management and Organizations.

    “These courses are: (i) Data Structures; (ii) Introduction to Public Policy; (iii) Quantitative Synthetic Biology; (iv) Cyberwarfare; (v) Object Oriented Programming; (vi) Structure and Properties of Civil Engineering Materials; (vii) Biopsychology; (viii) Climate/Change; and (ix) Management and Organizations.”
    Quote in the paperSubstantial claim
  • Findingown data

    There is a consensus among survey respondents that using ChatGPT in school work should be acknowledged.

    “Starting with panel a (ethics), there seems to be a consensus that using ChatGPT in school work should be acknowledged.”
    Quote in the paperNot a substantial claim
  • Findingown data

    ChatGPT's performance is comparable to or superior to that of students on nine out of the 32 university-level courses examined.

    “We find that ChatGPT’s performance is comparable, if not superior, to that of students in a multitude of courses.”
    • Also stated in: § Introduction · plain
      “We find that ChatGPT’s performance is comparable, or even superior, to that of students on nine out of the 32 courses.”
    • Also stated in: § Results · plain
      “Apart from Mathematics and Economics, each discipline has at least one course on which ChatGPT’s performance is comparable to, or even surpasses, that of students.”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 21
  • May combine two claims. 28
  • The quote only partly states this claim. 6

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    There is a consensus that ChatGPT will increase the competitiveness of non-native English students.

    “In terms of panel c (inequality), there seems to be a consensus that ChatGPT will increase the competitiveness of non-native English students.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    74% of students indicate they would use ChatGPT to assist with their studies, while both educators and students underestimated this share.

    “As can be seen in the fourth row, representing the average response across the five countries, 74% of students indicate that they would use ChatGPT (black line), while both educators and students underestimated this share.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Students in all five surveyed countries believe they can outsource mundane tasks to ChatGPT, and educators in Brazil and India agree with this statement.

    “Moving on to panel b (jobs), students in all five countries believe they can outsource mundane tasks to ChatGPT, and the educators in Brazil and India seem to agree with this statement.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Opinions vary on whether using ChatGPT in homework is unethical and whether it should be prohibited in school work; students in India and the US believe it is unethical and should be prohibited, while those in Brazil believe otherwise.

    “In contrast, opinions vary when it comes to determining whether the use of ChatGPT in homework is unethical and whether it should be prohibited in school work, e.g., students in India and the US believe it is unethical and should be prohibited, while those in Brazil believe otherwise.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Educators in Brazil and Japan agree that ChatGPT will reduce inequality in education, while those in the remaining three countries disagree.

    “When it comes to whether ChatGPT will reduce inequality in education, educators in Brazil and Japan (the two non-English speaking countries in our sample) agree with this statement, while those in the remaining three countries disagree.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Students from Brazil and India are significantly more likely, and students from Japan are significantly less likely, to use ChatGPT than those from the USA.

    “As can be seen, students from Brazil and India are significantly more likely, and students from Japan are significantly less likely, to use ChatGPT than those from the USA.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    India is the only country where educators believe ChatGPT is needed to increase their competitiveness in their job, and students therein agree the most with this statement.

    “India is the only country where educators believe ChatGPT is needed to increase their competitiveness in their job, and the students therein agree the most with this statement.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The largest performance gap between ChatGPT and students was for math-related questions, followed by trick questions.

    “Again, we find that the largest performance gap between ChatGPT and students was for math-related questions, followed by trick questions.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    The majority of NYUAD students plan to use ChatGPT to assist with their assignments (57%).

    “The majority of students plan to use ChatGPT to assist with their assignments (57%), and expect their peers to use it for this purpose (64%).”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    Most NYUAD professors expect others to treat the use of ChatGPT as plagiarism (71%).

    “Most professors plan to treat the use of ChatGPT as plagiarism (69%), and expect others to do so (71%).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Professors agreed more, or disagreed less, with all statements in the ethics category compared to students, while students agreed more, or disagreed less, with all statements in the jobs and inequality categories.

    “As can be seen, professors agreed more, or disagreed less, with all statements in category (i) compared to students (notice how all red data points fall below the diagonal) while students agreed more, or disagreed less, with all statements in categories (ii) and (iii).”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    The majority of NYUAD students believe ChatGPT should be used to assist with assignments (61%).

    “Moreover, the majority believe ChatGPT should be used (61%), and expect that their peers believe it should be used (55%), to assist with assignments.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPTZero has a higher false positive rate (18%) but a lower false-negative rate (32%) than OpenAI's Text Classifier.

    “GPTZero has a higher false positive rate (18%), but a lower false-negative rate (32%).”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    The majority of NYUAD students expect their peers to use ChatGPT to assist with assignments (64%).

    “The majority of students plan to use ChatGPT to assist with their assignments (57%), and expect their peers to use it for this purpose (64%).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Quillbot attack is successful on all courses, often increasing the false-negative rate to 100%.

    “Importantly, the Quillbot attack is successful on all courses, often increasing the false-negative rate to 100%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    OpenAI's Text Classifier misclassifies 5% of student submissions as AI-generated and 49% of ChatGPT's submissions as human-generated.

    “As can be seen, OpenAI’s Text Classifier misclassifies 5% of student submissions as AI-generated, and 49% of ChatGPT’s submissions as human-generated.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Professors and students at NYUAD agree that ChatGPT's use should be acknowledged, and neither believes that it will take their future job.

    “Despite these differences, professors and students seem to agree that ChatGPT’s use should be acknowledged, and neither believes that it will take their future job.”
    Quote in the paperNot a substantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    In each discipline, there are courses for which students' submissions are misclassified as AI-generated, as well as courses for which ChatGPT submissions are misclassified as human-generated.

    “As the figure illustrates, in each discipline, there are courses for which students’ submissions are misclassified, as well as courses for which ChatGPT submissions are misclassified.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    The majority of NYUAD students from all GPA brackets indicated that they would use ChatGPT.

    “As for GPA, disregarding those who preferred not to disclose their GPA, the majority of students from all GPA brackets into which students fell indicated that they would use the tool.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Students in Brazil think the use of ChatGPT in homework is ethical and should be allowed.

    “while those in Brazil think it is ethical and should be allowed”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey samples are representative of students and educators in the surveyed countries.

  • May combine two claims.

    Findingown data

    Students in India think the use of ChatGPT in homework is unethical and should not be allowed.

    “students in India think the use of ChatGPT in homework is unethical and should not be allowed”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The survey samples are representative of students and educators in the surveyed countries.

  • May combine two claims.

    Interpretationown data

    Humans outperform ChatGPT on math-related and trick questions.

    “For the time being, humans seem to outperform ChatGPT in these areas.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given

    The detectability of ChatGPT-generated solutions to homework remains unknown.

    “Finally, the detectability of ChatGPT-generated solutions to homework remains unknown.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Pursnani et al. showed that ChatGPT's performance varied across different sections of the US Fundamentals of Engineering exam, performing exceptionally in Ethics and Professional practice while under-performing in Hydrology.

    “Similarly, Pursnani et al. evaluated ChatGPT’s performance in the context of Engineering, testing the model on the US Fundamentals of Engineering exam33. In their study, they showed that the model’s performance varied across different sections of the exam, performing exceptionally in areas such as Ethics and Professional practice, while under-performing in others such as Hydrology.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] Pursnani, V., Sermet, Y. & Demir, I. Performance of ChatGPT on the US fundamentals of engineering exam: Comprehensive assessment of proficiency and potential implications for professional environmental engineering practice. arXiv preprintarXiv:2304.12198 (2023). source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    An obfuscation attack renders detection algorithms futile, failing to detect 95% of ChatGPT answers.

    “Worse still, an obfuscation attack renders these algorithms futile, failing to detect 95% of ChatGPT answers.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    There is an emerging consensus among students to use ChatGPT and among educators to treat its use as plagiarism.

    “Finally, there seems to be an emerging consensus among students to use the tool, and among educators to treat its use as plagiarism.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    ChatGPT performed at or near a passing level on all three stages of the United States Medical Licensing Exam without additional specialized training or reinforcement.

    “For instance, Kung et al. evaluated the performance of ChatGPT on the United States Medical Licensing Exam, where they show that ChatGPT performed at or near a passing level on all three stages of the test, without the need for additional specialized training or reinforcement21.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [21] Kung, T. H. et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS Digital Health 2, e0000198 (2023). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    The perspectives of students and educators around the globe regarding the usage of such technology remain unclear.

    “Moreover, the perspectives of students and educators around the globe regarding the usage of such technology remain unclear.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Abstracts written by ChatGPT have been shown to be indistinguishable from human-generated work.

    “which is unsurprising given that abstracts written by ChatGPT have been shown to be indistinguishable from human-generated work17.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [17] Else, H. Abstracts written by ChatGPT fool scientists. Nature 613, 423 (2023). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Studies have shown the validity of using SurveyMonkey to conduct surveys used in academic research.

    “Studies have shown the validity of using this platform to conduct surveys used in academic research, in addition to the numerous studies which use such platforms for conducting surveys globally56–60.”
    Quote in the paperNot a substantial claim0 of 5 sources checked
    • cites [56] Bentley, F. R., Daskalova, N. & White, B. Comparing the reliability of Amazon Mechanical Turk and Survey Monkey to traditional market research surveys. In Conf. Hum. Factors Comput. Syst. - Proc., 1092–1099 (2017). source

      Not checked in the free run.

    • cites [57] Mielke, J., Vermaßen, H. & Ellenbeck, S. Ideals, practices, and future prospects of stakeholder involvement in sustainability science. Proc. Natl. Acad. Sci. U.S.A. 114, E10648–E10657 (2017). source

      Not checked in the free run.

    • cites [58] Parsa, S. et al. Obstacles to integrated pest management adoption in developing countries. Proc. Natl. Acad. Sci. U.S.A. 111, 3889–3894 (2014). source

      Not checked in the free run.

    • cites [59] Evans, R. R. et al. Developing valid and reliable online survey instruments using commercial software programs. J. Consum. Health Internet 13, 42–52 (2009). source

      Not checked in the free run.

    • cites [60] Péloquin, K. & Lafontaine, M.-F. Measuring empathy in couples: Validity and reliability of the interpersonal reactivity index for couples. J. Pers. Assess. 92, 146–157 (2010). source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The survey was piloted in the United States on Prolific with open-ended questions to ensure respondents did not find anything odd or surprising and that no new topics emerged about ChatGPT that the opinion statements covered.

    “The survey was piloted in the United States on Prolific with open-ended questions to investigate that respondents did not find anything odd or surprising about survey questions, and also to ensure no new topics emerged about ChatGPT that the opinion statements covered.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    Inter-rater reliability for the majority of courses was greater than 0.6, but six of the 32 courses did not meet this threshold.

    “While the inter-rater reliability for the majority of courses was greater than 0.6, six of the 32 courses did not meet this threshold.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Both students' and ChatGPT's answers were compiled into a single document in random order, labelled as "Submission 1" to "Submission 6".

    “Both students’ and ChatGPT’s answers were then compiled into a single document in random order, labelled as “Submission 1” to “Submission 6”.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given

    The participating faculty vetted the graders' output and ensured that the grading is in line with the rubrics.

    “The participating faculty vetted the graders’ output and ensured that the grading is in line with the rubrics.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Four of the six courses with inter-rater reliability below threshold were essay-based and inherently subjective, which may explain the discrepancy.

    “Four of these six courses were essay-based and inherently subjective which may explain this discrepancy.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Student participants who indicated that they were aware of the study and faculty members who coauthored the paper were excluded.

    “We excluded student participants who indicated that they aware of our study, as well as faculty members who coauthored the paper.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Informed consent was obtained from all participants in every segment of this study.

    “Informed consent was obtained from all participants in every segment of this study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All research was performed in accordance with relevant guidelines and regulations.

    “All research was performed in accordance with relevant guidelines and regulations.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Treating the use of ChatGPT as plagiarism is likely to be the most common approach to counter its usage.

    “Treating the use of ChatGPT as plagiarism is likely to be the most common approach to counter its usage.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    Educational institutions are facing a serious threat with regards to current student evaluation frameworks.

    “These findings, coupled with the clear ease with which students can avoid detection by current AI-text classifiers as demonstrated in our study, suggests that educational institutions are facing a serious threat with regards to current student evaluation frameworks.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The 32 NYUAD courses are representative of university-level courses generally.
    • Rests on an assumption: The AI-text classifiers’ classifications are valid indicators of whether text was generated by AI.

  • May combine two claims.

    Interpretationreasoning

    The inherent conflict between students' use of ChatGPT and professors' treatment of it as plagiarism could create an environment where students have incentives to hide their use while professors are eager to detect it.

    “The inherent conflict between these two could create an environment where students’ incentives suggest they should hide the use of ChatGPT while professors are eager to detect its use (and may expect their colleagues to do the same).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Students’ self-reported intentions to use ChatGPT predict their actual use.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The analytical sample only contains current students and educators who are either teachers or professors based on respondents' self-reports.

    “In addition, we ensured that the analytical sample only contains current students and educators who are either teachers or professors based on respondents’ self-reports.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    SurveyMonkey utilizes email and location verification for fraud detection and ID exclusions, thereby guarding against duplicate and bot-submitted responses.

    “The platform utilizes email and location verification for fraud detection and ID exclusions, thereby guarding against duplicate and bot-submitted responses.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Academic misconduct in school is highly correlated to dishonesty and with similar decision-making patterns in the workplace.

    “studies have demonstrated that academic misconduct in school is highly correlated to dishonesty and with similar decision-making patterns in the workplace39,40”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [39] Harding, T. S., Passow, H. J., Carpenter, D. D. & Finelli, C. J. An examination of the relationship between academic dishonesty and professional behavior. In 33rd Annual frontiers in education, 2003. FIE 2003., vol. 3, S2A–6 (IEEE, 2003). source

      Not checked in the free run.

    • cites [40] Nonis, S. & Swift, C. O. An examination of the relationship between academic dishonesty and workplace dishonesty: A multicampus investigation. J. Educ. Bus. 77, 69–77 (2001). source

      Not checked in the free run.

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    The participating faculty was instructed not to inform the graders that some of the submissions are AI-generated.

    “The participating faculty was instructed not to inform the graders that some of the submissions are AI-generated.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The research was approved by the Institutional Review Board of New York University Abu Dhabi (HRPP-2023-5).

    “The research was approved by the Institutional Review Board of New York University Abu Dhabi (HRPP-2023-5).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The NYUAD survey includes responses from 151 students (out of a total of 2103, 7.2%) and 60 professors (out of a total of 393, 15.2%) collected between January 23rd and February 3rd, 2023.

    “As for the NYUAD survey, it includes responses from 151 students (out of a total of 2103, 7.2%) and 60 professors (out of a total of 393, 15.2%) collected between January 23rd and February 3rd, 2023.”
    Quote in the paperNot a substantial claim
47 of 47

  • The paper relies on this without support.

    The global survey respondents are representative of students and educators in Brazil, India, Japan, the UK, and the US.

    “To understand how the use of ChatGPT is perceived by educators and students, we fielded a global survey in five countries, namely Brazil, India, Japan, the UK, and the US, targeting a minimum of 100 educators and 200 students per country; see Methods for more details.”

  • The paper relies on this without support.

    The NYUAD survey respondents are representative of NYUAD students and professors.

    “Additionally, we survey 151 undergraduate students and 60 professors more extensively at the authors’ institution to explore differences in perceptions of ChatGPT across disciplines.”

  • The paper relies on this without support.

    The Quillbot attack can be readily implemented by students without requiring any technical know-how.

    “Our “Quillbot attack” was chosen since it can be readily implemented by students, without requiring any technical know-how; see Methods for a detailed description of the attack, and see Fig. 4b for a stylized example of how it works.”

  • The paper relies on this without support.

    The value of a degree is in part linked to the validity of the assessment process.

    “The value of the degree granted is in part linked to the validity of the assessment process.”

  • The paper relies on this without support.

    The 32 NYUAD courses are representative of university-level courses generally.

    “When comparing ChatGPT’s performance to that of students across university-level courses, the results indicate a clear need to take “AI-plagiarism” seriously.”

  • The paper relies on this without support.

    The survey samples are representative of students and educators in the surveyed countries.

    “Participants were recruited from Brazil, India, Japan, United Kingdom, and United States, with a minimum of 200 students and 100 educators per country”

  • The paper relies on this without support.

    The AI-text classifiers’ classifications are valid indicators of whether text was generated by AI.

    “Once all student and ChatGPT submissions were collected for all courses, they were used to test two classifiers designed specifically to determine whether a body of text has been generated using AI, namely, GPTZero35, and OpenAI’s own AI text classifier36.”

  • The paper relies on this without support.

    Students’ self-reported intentions to use ChatGPT predict their actual use.

    “the vast majority of surveyed students (94%) indicated that they would use it to assist with their homework and assignments in the coming semester”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • A recent study observed that there is an emerging consensus among students to use the tool, while educators tend to view its use as plagiarism, highlighting the need for policy discussions on integrating AI into educational frameworks.

    “A recent study [24] observed that there is an emerging consensus among students to use the tool, while educators tend to view its use as plagiarism, highlighting the need for policy discussions on integrating AI into educational frameworks.”

    From ChatGPT for Academic Purposes: Survey Among Undergraduate Healthcare Students in Malaysia (2024) · cites this paper as [24]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Number of courses where ChatGPT matched or exceeded students changes

    Major

    The Discussion reports 12 of 32 courses, but the Results identify nine such courses and say the same nine remain after standardizing grades. This changes a central performance finding.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 3

  1. Vincent, J. Top AI conference bans use of ChatGPT and AI language tools to write academic papers (2023). https://www.theverge.com/2023/1/5/23540291/chatgpt-ai-writing-tool-banned-writing-academic-icml-paper.
  2. Qureshi, B. Exploring the use of chatgpt as a tool for learning and assessment in undergraduate computer science curriculum: Opportunities and challenges. arXiv preprintarXiv:2304.11214 (2023).
  3. Bicchieri, C. Norms in the Wild (Oxford University Press, Oxford, 2016).

Cited by

Sorted by citations

Related

Authors
Ibrahim, H., Liu, F., Asim, R., Battu, B., Benabderrahmane, S., Alhafni, B., Adnan, W., Alhanai, T., AlShebli, B., Baghdadi, R., Bélanger, J. J., Beretta, E., Celik, K., Chaqfeh, M., Daqaq, M. F., Bernoussi, Z. E., Fougnie, D., Garcia de Soto, B., Gandolfi, A., Gyorgy, A., Habash, N., Harris, J. A., Kaufman, A., Kirousis, L., Kocak, K., Lee, K., Lee, S. S., Malik, S., Maniatakos, M., Melcher, D., Mourad, A., Park, M., Rasras, M., Reuben, A., Zantout, D., Gleason, N. W., Makovi, K., Rahwan, T., & Zaki, Y.
Affiliations
New York University Abu Dhabi (AE)
Corresponding author
  • Rahwan, Talal
  • Zaki, Yasir
ORCID
Venue
Scientific Reports
Publisher
Springer Science and Business Media LLC
Published
2023-08-24
Volume
13
Issue
1
Pages
12187-12187
ISSN
2045-2322 (ISSN-L)
DOI
10.1038/s41598-023-38964-3
PMC
PMC10449897
OpenAlex
W4386136032
Semantic Scholar
1ffc9d5ebfbc9596aa6c4b9072c60dc12858a718
Cited by
154
References
32
Citations by year
2026: 19 · 2025: 58 · 2024: 69 · 2023: 7
FWCI
38.28
Type
journal article
Language
en
Update notices
correction (2023-10-10)
Open access
Licence
Locations
  • Scientific Reports (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Iris (University of Trento) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Funders
  • New York University Abu Dhabi (CG001)
  • Tamkeen (CG001)
Topics
  • Social Sciences › Safety Research › Academic integrity and plagiarism
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
ChatGPT, large language models, AI-generated text, academic integrity, plagiarism, AI detection, student perceptions, educator perceptions, university courses, generative AI in education
MeSH terms
Artificial Intelligence, Communication, Humans, Perception, Schools, Universities

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review