ChatGPT for Education and Research: Opportunities, Threats, and Strategies
Open accessPublished version of a preprint
Abstract
In recent years, the rise of advanced artificial intelligence technologies has had a profound impact on many fields, including education and research. One such technology is ChatGPT, a powerful large language model developed by OpenAI. This technology offers exciting opportunities for students and educators, including personalized feedback, increased accessibility, interactive conversations, lesson preparation, evaluation, and new ways to teach complex concepts. However, ChatGPT poses different threats to the traditional education and research system, including the possibility of cheating on online exams, human-like text generation, diminished critical thinking skills, and difficulties in evaluating information generated by ChatGPT. This study explores the potential opportunities and threats that ChatGPT poses to overall education from the perspective of students and educators. Furthermore, for programming learning, we explore how ChatGPT helps students improve their programming skills. To demonstrate this, we conducted different coding-related experiments with ChatGPT, including code generation from problem descriptions, pseudocode generation of algorithms from texts, and code correction. The generated codes are validated with an online judge system to evaluate their accuracy. In addition, we conducted several surveys with students and teachers to find out how ChatGPT supports programming learning and teaching. Finally, we present the survey results and analysis.
AI snapshot
Written from the paper’s full text.
TL;DR
This paper examines ChatGPT’s opportunities and threats for education and research, with a focus on programming learning and teaching. It combines coding experiments on an online judge system with surveys of students and teachers, finding generally useful support but also accuracy, integrity, and over-reliance concerns.
Key findings
- In code-generation experiments, ChatGPT’s generated code was correct about 95.83% of the time on a basic compiler and about 75% on the Aizu Online Judge (AOJ), with an average accuracy of 85.42%.
- Among surveyed students, 78.8% had used ChatGPT for programming problems, 86.7% said its suggestions helped solve programming problems, and 92.9% found it useful for learning programming; most student satisfaction ratings were 4/5 (36.6%) or 5/5 (20%).
- About 60% of surveyed teachers had used ChatGPT for teaching programming, but about 50% rated satisfaction as 3/5, suggesting limited trust; one teacher said ChatGPT “is still not perfect to answer some exams of my Java class.”
- The paper identifies threats including cheating on online exams and assignments, diminished critical thinking, difficulty evaluating AI-generated text/code, ethical implications and bias, and blind reliance on generative AI; proposed strategies include clear assignment instructions, plagiarism detection, proctoring, comparing responses, and follow-up questions.
Methods
- The authors conducted coding-related experiments with ChatGPT: code generation from problem descriptions, pseudocode generation, code correction/debugging, and code optimization.
- For code generation, they used eight random “Algorithms and Data Structures” problems from AOJ, generated code three times per problem, and validated results on both a basic compiler and the AOJ platform.
- They also conducted surveys with undergraduate, Master’s, and doctoral students and with teachers about ChatGPT’s support for programming learning and teaching; the paper reports percentages but not the total number of respondents.
Limitations
- The paper states that experiments were conducted on ChatGPT during active development, so results may vary with new ChatGPT versions, different questions, different problem descriptions, and different solution codes for optimization.
- The paper notes that although ChatGPT generated about 85.42% correct code on average, adjustments may still be necessary for error-free compilation and acceptance, and it failed an elementary counting task involving the digit “8.”
- Evident from the text: the survey sample size is not reported, and the coding experiments are limited to eight AOJ problems with three generations each, which restricts generalizability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 9 cited sources checked · 0 supported · 0 flagged
- Findingown data
In the authors' experiment, the correctness rate of ChatGPT-generated code based on a basic compiler was approximately 95.83%.
“the correctness rate of the generated code based on the basic compiler is approximately 95.83%”
Quote in the paperSubstantial claim - Findingown data
ChatGPT generates mostly correct code considering long problem descriptions, including constraints, algorithms, and input and output formatting.
“ChatGPT generates mostly correct code considering long problem descriptions, including constraints, algorithms, and input and output formatting.”
Quote in the paperSubstantial claim - Findingown data
In the authors' experiment, the correctness rate of ChatGPT-generated code based on AOJ compilation was approximately 75%.
“the correctness rate based on AOJ compilation is approximately 75%”
Quote in the paperNot a substantial claim - Findingown data
In the authors' experiment, ChatGPT solved all mathematical derivative equations correctly, with 100% correctness.
“In addition, we experimented with the ChatGPT model to find the derivatives of mathematical equations, and it solved all equations correctly.In this case, the correctness is 100%.”
Quote in the paperNot a substantial claim - Findingown data
There was a significant difference between ChatGPT-generated quizzes of different difficulty levels on the same topic.
“We also asked ChatGPT about quizzes of other levels, such as medium and easy, and we found that there was a significant difference between quizzes of different levels.”
Quote in the paperNot a substantial claim
- Findingown data
ChatGPT can help optimize code by suggesting ways to reduce memory usage and time complexity.
“ChatGPT can help optimize codes by suggesting ways to reduce the memory usage and time complexity.”
Quote in the paperSubstantial claim - Findingown data
When asked to prepare a detailed outline for an Algorithms and Data Structures course, ChatGPT created a complete table of contents showing topics, breakdowns, and learning objectives.
“For example, we asked ChatGPT to “prepare a detailed outline for the Algorithms and Data Structures course”, as depicted in Figure 2. It creates a complete table of contents for the Algorithms and Data Structures course, showing topics and a breakdown of each topic, along with learning objectives.”
Quote in the paperNot a substantial claim - Findingown data
Most surveyed students rated their programming experience between 1/5 and 3/5.
“We found that most students rated their programming experiences between 1/5 and 3/5.”
Quote in the paperNot a substantial claim - Findingown data
The average accuracy of the code generated by ChatGPT is 85.42%.
“The average accuracy of the code generated by ChatGPT is 85.42%.”
Quote in the paperSubstantial claim - Findingown data
About 86.7% of surveyed students answered 'Yes' to whether ChatGPT's suggestions help them in solving programming problems.
“We found that about 86.7% of the students answered “Yes”.”
Quote in the paperSubstantial claim - Findingown data
Most teachers were satisfied with the suggestions provided by ChatGPT for programming teaching and research.
“For questions about programming teaching and research, most teachers were satisfied with the suggestions provided by ChatGPT.”
Quote in the paperNot a substantial claim - Findingown data
About 50% of teachers gave a rating of 3/5 on satisfaction with ChatGPT.
“About 50% of teachers gave a rating of 3/5.”
Quote in the paperNot a substantial claim - Findingown data
78.8% of surveyed students answered 'Yes' to having taken help/support from ChatGPT for solving programming problems.
“In response to the question “Have you taken help/support from ChatGPT for solving programming problems?”,78.8% of the students answered “Yes” and 21.2% answered “No”.”
Quote in the paperSubstantial claim - Findingown data
The distribution of student satisfaction ratings with ChatGPT was: 1/5 from 3.3%, 2/5 from 10%, 3/5 from 30%, 4/4 from 36.6%, and 5/5 from 20%.
“received a rating of 1/5 from 3.3% of students, 2/5 from 10%, 3/5 from 30%, 4/4 from 36.6%, and 5/5 from 20%”
Quote in the paperSubstantial claim - Findingown data
ChatGPT can provide clues and research ideas by analyzing a specific topic.
“It can also provide clues and research ideas by analyzing a specific topic.”
Quote in the paperNot a substantial claim - Interpretationreasoning
Various constraints (time, memory, etc.) and output formatting on the AOJ platform may be a reason for the lower accuracy on the AOJ platform than the basic compiler.
“when running a submitted code on the AOJ platform, various constraints (time, memory, etc.) and output formatting are taken into account, which may be a reason for the lower accuracy on the AOJ platform than the basic compiler”
Quote in the paperNot a substantial claim - Interpretationown data
ChatGPT can generate easy-to-understand explanations and pseudocode that are useful for learners to understand algorithmic concepts.
“The results demonstrate that ChatGPT can generate easy-to-understand explanations and pseudocode that are useful for learners to understand algorithmic concepts.”
Quote in the paperSubstantial claim- Rests on an assumption: The authors' judgment that ChatGPT-generated explanations and pseudocode are easy to understand and useful reflects learners' actual understanding.
- Interpretationreasoning
ChatGPT can be a powerful tool to assist programmers in their programming-learning process.
“It appears that ChatGPT can be a powerful tool to assist programmers in their programming-learning process.”
Quote in the paperSubstantial claim- Rests on an assumption: The eight randomly selected AOJ problems and three code generations per problem are representative of ChatGPT's general code-generation performance for programming learning.
- Premisereasoning
Restrictions have been placed on the use of ChatGPT in educational institutions.
“As a result, restrictions have been placed on the use of ChatGPT in educational institutions [63].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [63] Kalhan, R. (2023, March 12). ChatGPT Banned from New York City Public Schools’ Devices and Networks. Available online: https://www.nbcnews.com/tech/tech-news/new-york-city-public-schools-ban-chatgpt-devices-networks-rcna64446.
Not checked in the free run.
- Premisecitation
ChatGPT has exhibited top performance in many application domains, such as coherent content and essay generation, chatbot responses, language translation, question answering, and programming code.
“ChatGPT has exhibited top performance in many application domains, such as coherent content and essay generation, chatbot responses, language translation, question answering, and programming code [9,10].”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [9] Kasneci, 2023, ChatGPT for good? On opportunities and challenges of large language models for education, Learn. Individ. Differ. source
Not checked in the free run.
- cites [10] Qadir, J. (2022). Engineering Education in the Era of ChatGPT: Promise and Pitfalls of Generative AI for Education. TechRxiv Prepr. source
Not checked in the free run.
- Interpretationown data
A large number of students have taken the support of ChatGPT for their programming learning.
“These results show that a large number of students have taken the support of ChatGPT for their programming learning.”
Quote in the paperNot a substantial claim- Rests on an assumption: The survey respondents are representative of students and teachers who use ChatGPT for programming learning and teaching.
- Rests on an assumption: Students' self-reported helpfulness of ChatGPT suggestions reflects actual improvement in solving programming problems.
- Premisecitation
ChatGPT can be used to grade assignments and quizzes.
“In addition, the model can be used to grade assignments and quizzes [9].”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [9] Kasneci, 2023, ChatGPT for good? On opportunities and challenges of large language models for education, Learn. Individ. Differ. source
Not checked in the free run.
- Interpretationown data
ChatGPT has clear limitations.
“It also shows the clear limitation of ChatGPT.”
Quote in the paperSubstantial claim - Interpretationown data
ChatGPT and other AI LLMs can be supporting tools for educational and research work.
“ChatGPT and other AI LLMs have the potential and can be supporting tools for educational and research work.”
Quote in the paperSubstantial claim - Interpretationown data
ChatGPT has not yet gained enough trust in programming education.
“This result also indicates that ChatGPT has not yet gained enough trust in programming education.”
Quote in the paperSubstantial claim - Interpretationreasoning
In some cases, adjustments to the generated code may be necessary to achieve error-free compilation and acceptance.
“In some cases, adjustments to the generated code may be necessary to achieve error-free compilation and acceptance.”
Quote in the paperNot a substantial claim - Interpretationown data
Most students were satisfied with the programming support provided by ChatGPT.
“It can be seen that most students were satisfied with the programming support provided by ChatGPT.”
Quote in the paperSubstantial claim - Interpretationown data
ChatGPT has limitations in generating code based on the description.
“it still has the limitations of generating code based on the description.”
Quote in the paperNot a substantial claim - Interpretationreasoning
The indistinguishability of ChatGPT-generated answers and texts poses a challenge to educators and researchers.
“This poses a challenge to educators and researchers [57,[60][61][62].”
Quote in the paperSubstantial claim - Interpretationown data
There is a risk of blindly trusting ChatGPT.
“There is a risk of blindly trusting ChatGPT.”
Quote in the paperSubstantial claim - Interpretationreasoning
Existing plagiarism-detection tools are finding it increasingly difficult to distinguish between AI-and human-generated texts.
“Existing plagiarismdetection tools are finding it increasingly difficult to distinguish between AI-and humangenerated texts.”
Quote in the paperSubstantial claim - Interpretationreasoning
A heavy reliance on generative AI tools such as ChatGPT can negatively affect education and research.
“A heavy reliance on generative AI tools such as ChatGPT can negatively affect education and research.”
Quote in the paperSubstantial claim - Premisecitation
ChatGPT has been authorized and credited as author on published papers and preprints.
“Recently, ChatGPT has authorized and credited published papers and preprints [58].”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [58] 2023, ChatGPT listed as author on research papers: Many scientists disapprove, Nature source
Not checked in the free run.
- May combine two claims. 18
- Classified as probably not a substantial claim. 31
May combine two claims.
Findingown dataThe suggestions ChatGPT provided for debugging erroneous AOJ codes were interesting to programmers and helpful in resolving the code errors.
“The suggestions proved to be interesting to the programmers and helpful in resolving the code errors.”
Quote in the paperSubstantial claim- Rests on an assumption: The authors' assessment that ChatGPT's debugging suggestions were helpful reflects actual programmer benefit.
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown dataOf the participating students, 61.3% were undergraduate students, 32.2% were Master's students, and 6.5% were second-year doctoral students.
“Of the participating students, 61.3% (35.5% first year, 16.1% second year, 6.5% third year, and 3.2% fourth year) were undergraduate students, 32.2% (16.1% first year and 16.1% second year) were Master's students, and 6.5% were second-year doctoral students.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataChatGPT can identify errors in code and provide potential suggestions and code snippets.
“ChatGPT can identify errors in code and provide potential suggestions and code snippets.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
Findingown dataWhen asked to provide pseudocode for a selection sort and an explanation, ChatGPT generated the pseudocode and explanation.
“For example, when we asked ChatGPT to “provide a pseudocode for a selection sort and an explanation”, it generated the pseudocode and explanation, as depicted in Figure 7.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataThe optimized code generated by ChatGPT for the Bubble Sort problem passed all test cases (10/10) and was accepted on the AOJ platform.
“The generated optimized code was validated on the AOJ platform, and it passed all test cases (10/10) and was accepted for the Bubble Sort problem”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone givenChatGPT can generate topic-specific quiz questions tailored to the subject matter and difficulty level.
“ChatGPT can also be used to generate topic-specific quiz questions tailored to the subject matter and difficulty level.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Findingnone givenChatGPT can generate nearly accurate answers to technical questions from a wide range of topics and correct or partially correct programming code based on problem descriptions, algorithm and problem names, etc.
“ChatGPT can generate nearly accurate answers to technical questions from a wide range of topics and correct or partially correct programming code based on problem descriptions, algorithm and problem names, etc.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataThe accuracy of the generated code is 95.83% on the basic compiler and 75% on AOJ.
“the experimental results show that the accuracy of the generated code is 95.83% on the basic compiler and 75% on AOJ.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Findingnone givenChatGPT can be used to create lesson plans for specific courses, providing topic-specific illustrations, activities, and exercises.
“ChatGPT can be used to create lesson plans for specific courses, such as math, chemistry, physics, computer science, civil engineering, language, and literature.ChatGPT provides topic-specific illustrations, activities, and exercises to help educators better teach their students.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataChatGPT is useful not only for programming education but also for education and research.
“Our surveys and experimental results show that ChatGPT is useful not only for programming education but also for education and research.”
Quote in the paperSubstantial claimMay combine two claims.
Findingown dataDuring the experiments, we obtained significant results in code generation, error checking and debugging, and optimization of the solution code.
“During the experiments, we obtained significant results in code generation, error checking and debugging, and optimization of the solution code.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationown dataChatGPT optimized and fixed the erroneous code for the Bubble Sort problem, though this is a rare case.
“This is a rare case (or an example of erroneous code), but ChatGPT has optimized and fixed it.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
InterpretationreasoningGeneration of harmful content by ChatGPT can lead to social unrest, reputation damage, and even physical harm.
“This can lead to social unrest, reputation damage, and even physical harm.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Interpretationnone givenChatGPT can maintain humanlike conversations and generate human-like text for any natural language query that is nearly indistinguishable.
“ChatGPT is a revolutionary LLM that can maintain humanlike conversations and generate human-like text for any natural language query that is nearly indistinguishable.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
InterpretationreasoningSimply acquiring answers and code from ChatGPT can be a barrier to improving learners' critical thinking and problem-solving skills.
“Simply acquiring answers and code from ChatGPT can be a barrier to improving learners' critical thinking and problem-solving skills.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Premisenone givenThe quality and suitability of ChatGPT applications for programming learning remain unclear.
“However, the quality and suitability of these applications for programming learning remain unclear.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
InterpretationreasoningSolution codes generated by AI models can be used for academic coding exams and competitions.
“the solution codes generated by AI models can be used for academic coding exams and competitions”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenChatGPT can provide personalized learning support for students by suggesting customized resources and learning activities based on a student's needs and learning style.
“Educators can use ChatGPT to provide personalized learning support for their students.Depending on a student's needs and learning style, ChatGPT can suggest customized resources and learning activities.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Interpretationnone givenThe ease of obtaining answers, problem-solving strategies, and scientific text generation can limit critical thinking and problem-solving skills.
“This is because the ease of obtaining answers, problem-solving strategies, and scientific text generation can limit critical thinking and problem-solving skills.”
Quote in the paperNot a substantial claimMay combine two claims.
Interpretationnone givenChatGPT can be used to answer questions, write essays, solve problems, explain complex topics, provide virtual tutoring, practice languages, learn programming, teach, and support research.
“The model can be used to answer questions, write essays, solve problems, explain complex topics, provide virtual tutoring, practice languages, learn programming, teach, and support research.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).
Interpretationnone givenThe ethical implications of ChatGPT are complex and multifaceted.
“the ethical implications (e.g., bias and discrimination, privacy and security, misuse of technology, accountability, transparency, and social impact) of ChatGPT are complex and multifaceted”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).
InterpretationreasoningIt can be difficult to hold anyone accountable for the responses generated by ChatGPT.
“it can be difficult to hold anyone accountable for the responses generated”
Quote in the paperNot a substantial claim- Rests on an assumption: ChatGPT generates responses without human intervention.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationCotton et al. [57] presented several strategies for recognizing texts from LLMs such as ChatGPT, including language inconsistencies, a lack of proper citations, factual errors, ambiguity, and poor context awareness.
“Cotton et al. [57] presented several strategies for recognizing texts from LLMs such as ChatGPT, including language inconsistencies, a lack of proper citations, factual errors, ambiguity, and poor context awareness.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [57] Cotton, D.R., Cotton, P.A., and Shipway, J.R. (2023). Chatting and Cheating. Ensuring academic integrity in the era of ChatGPT. Innov. Educ. Teach. Int., 1–12. source
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Interpretationnone givenChatGPT and other generative models involve the collection and processing of personal data, which raises concerns about privacy and data security.
“ChatGPT and other generative models involve the collection and processing of personal data, which raises concerns about privacy and data security.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenIf the answer is identical to answers generated by ChatGPT, it may be a sign that the answer was not generated by a human.
“If the answer is identical to answers generated by ChatGPT, it may be a sign that the answer was not generated by a human.”
Quote in the paperNot a substantial claimClassified as probably not a substantial claim.
PremisecitationThe answers and texts generated by ChatGPT are becoming indistinguishable from human-generated answers and texts.
“The answers and texts generated by ChatGPT are becoming indistinguishable from human-generated answers and texts.”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [57] Cotton, D.R., Cotton, P.A., and Shipway, J.R. (2023). Chatting and Cheating. Ensuring academic integrity in the era of ChatGPT. Innov. Educ. Teach. Int., 1–12. source
Not checked in the free run.
- cites [60] Elkins, 2020, Can GPT-3 pass a Writer’s turing test?, J. Cult. Anal.
Not checked in the free run.
- cites [61] Gao, C.A., Howard, F.M., Markov, N.S., Dyer, E.C., Ramesh, S., Luo, Y., and Pearson, A.T. (2022). Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. bioRxiv. source
Not checked in the free run.
Classified as probably not a substantial claim.
InterpretationreasoningChatGPT threatens the fairness and validity of online exams and assignments.
“In short, ChatGPT threatens the fairness and validity of online exams and assignments.”
Quote in the paperSubstantial claim- Rests on an assumption: ChatGPT can generate human-like text for academic topics.
May combine two claims.
Interpretationnone givenChatGPT can solve technical and non-technical problems.
“Furthermore, the ChatGPT model can be used to solve technical (e.g., engineering and computer programming) and non-technical (e.g., language and literature) problems.”
Quote in the paperSubstantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
Interpretationnone givenChatGPT can effectively support the writing process of research by finding and correcting typographical errors, improving grammatical inconsistencies, providing advanced vocabulary, and recommending improvement strategies.
“First, it can effectively support the writing process of research.At its most basic, it can improve writing by finding and correcting typographical errors, improving grammatical inconsistencies, providing advanced vocabulary, and recommending improvement strategies.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Interpretationnone givenChatGPT and other AI language models can be used to generate fake news, hate speech, and other harmful content.
“ChatGPT and other AI language models can be used to generate fake news, hate speech, and other harmful content.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
InterpretationreasoningPlagiarism-detection tools can help detect cases of fraud.
“This can help detect cases of fraud.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenChatGPT responses typically have certain characteristics, such as a lack of personalization or a rather generic tone.
“ChatGPT responses typically have certain characteristics, such as a lack of personalization or a rather generic tone.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.
InterpretationreasoningChatGPT can answer learners' queries and provide explanations and examples on a particular topic.
“Educators can receive the help of ChatGPT to answer learners' questions.Furthermore, if educators can ask ChatGPT for explanations and examples on a particular topic, it will surely increase the effectiveness of teaching.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
InterpretationreasoningIf the answers appear disjointed or nonsensical, this may indicate that they were generated by ChatGPT or another AI model.
“If the answers appear disjointed or nonsensical, this may indicate that they were generated by ChatGPT or another AI model.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenChatGPT responses may not have a consistent or logical flow, especially when it is generating answers to complex questions.
“ChatGPT responses may not have a consistent or logical flow, especially when it is generating answers to complex questions.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenChatGPT can summarize published work on a particular topic, helping researchers understand the work.
“The model can also summarize published work on a particular topic, which helps researchers understand the work.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based).
Premisenone givenThere are no such tools that can recognize code generated by AI models.
“To the best of our knowledge, there are no such tools that can recognize code generated by AI models”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S4 (based). May combine two claims.
Interpretationnone givenChatGPT has certain limitations, such as a lack of common sense, potential bias, difficulty with complex reasoning, and inability to process visual information.
“it still has certain limitations, such as a lack of common sense, potential bias, difficulty with complex reasoning, and inability to process visual information.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Interpretationnone givenChatGPT can be used to check submitted assignments for plagiarism.
“The model can be used to check submitted assignments for plagiarism.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenOur experiments were conducted on ChatGPT, which is currently in an active development phase.
“It is worth noting that our experiments were conducted on ChatGPT, which is currently in an active development phase.”
Quote in the paperNot a substantial claim
The paper relies on this without support.
The eight randomly selected AOJ problems and three code generations per problem are representative of ChatGPT's general code-generation performance for programming learning.
“We leveraged the problem descriptions of eight random problems and generated codes three times for each problem on the basis of the same description.”
The paper relies on this without support.
The survey respondents are representative of students and teachers who use ChatGPT for programming learning and teaching.
“We conducted a comprehensive survey on how ChatGPT supports programming learning and teaching among undergraduate, Master's, and doctoral students and teachers.”
The paper relies on this without support.
Students' self-reported helpfulness of ChatGPT suggestions reflects actual improvement in solving programming problems.
“By asking the question “Do ChatGPT's suggestions help you in solving programming problems?”,we tried to assess how ChatGPT suggestions help students.We found that about 86.7% of the students answered “Yes”.”
The paper relies on this without support.
The authors' judgment that ChatGPT-generated explanations and pseudocode are easy to understand and useful reflects learners' actual understanding.
“The results demonstrate that ChatGPT can generate easy-to-understand explanations and pseudocode that are useful for learners to understand algorithmic concepts.”
The paper relies on this without support.
The authors' assessment that ChatGPT's debugging suggestions were helpful reflects actual programmer benefit.
“The suggestions proved to be interesting to the programmers and helpful in resolving the code errors.”
The paper relies on this without support.
ChatGPT can generate human-like text for academic topics.
“As ChatGPT can generate human-like text for academic topics”
The paper relies on this without support.
LLMs rely heavily on training data, and when the data contain biases or anomalies, it could lead to unfair results.
“Typically, LLMs rely heavily on training data, and when the data contain biases or anomalies, it could lead to unfair results.”
The paper relies on this without support.
The internal mechanisms and processes of ChatGPT are not sufficiently open and transparent to users about how they work.
“Furthermore, the internal mechanisms and processes are not sufficiently open and transparent to users about how they work.”
The paper relies on this without support.
ChatGPT generates responses without human intervention.
“Because ChatGPT generates responses without human intervention”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Students that are more experienced using ChatGPT tend to be less concerned about its impact on academic integrity and more optimistic about its role in self-study.
“Particularly, students that are more experienced using ChatGPT tend to be less concerned about its impact on academic integrity and more optimistic about its role in self-study (Nisar and Aslam 2023;Rahman and Watanobe 2023).”
From Widely used but barely trusted: understanding student perceptions on the use of generative AI in higher education (2025) · cites this paper as Rahman and Watanobe 2023
Section 5 reports students’ programming experience, ChatGPT use, and satisfaction, but does not compare attitudes by experience using ChatGPT. Section 6 discusses academic-integrity risks without reporting whether more experienced users are less concerned. Searches for experience, self-study, and integrity found no passage establishing the claimed relationship. (LLM, conf=0.76)
Experimental — accuracy not yet measuredChatGPT has been highlighted for fostering students' performance, engagement, and motivation and enhancing teaching practices.
“foster students’ performance, engagement and motivation [14, 44, 58], and enhance teaching practices [17, 18, 64, 74].”
From A systematic literature review of empirical research on ChatGPT in education (2024) · cites this paper as [58]
Fostering students' performance, engagement, and motivation leads to increased students' academic success.
“fostering students' performance, engagement, and motivation, leading to increase students' academic success [14, 44, 58]”
From A systematic literature review of empirical research on ChatGPT in education (2024) · cites this paper as [58]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Table 1 names a different lead author for reference 25
MinorTable 1 attributes reference 25 to Simon et al., while the reference list identifies Frieder as its lead author.
“Simon et al. [25]”
“25. Frieder, S.; Pinchetti, L.; Griffiths, R.R.; Salvatori, T.; Lukasiewicz, T.; Petersen, P.C.; Chevalier, A.; Berner, J. Mathematical capabilities of chatgpt.”
Cites
1 retracted
Retracted: Chat With ChatGPT on Intelligent Vehicles: An IEEE TIV Perspective
Du, H., Teng, S., Chen, H., Ma, J., Wang, X., Gou, C., Li, B., Ma, S., Miao, Q., Na, X., et al. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3253281 · Cited by 164 · cited 1× · see citing works
“Moreover, ChatGPT has recently been applied to tasks in different domains such as software engineering [36], medical education [37], data augmentation [38], fifth industrial revolution [39], code generation [40], intelligent vehicles [41], and solving AI tasks [42].”
MethodRetractedVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 1× · see citing works
“Therefore, the Transformer architecture was introduced [4] to alleviate this problem.”
MethodLanguage Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1× · see citing works
“Generative pretrained transformer (GPT)-3 is a large language model (LLM) based on the Transformer architecture [5,6] that has achieved significant success in NLP tasks.”
BackgroundKaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,695 · see citing works
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552 · see citing works
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · cited 3× · see citing works
“Furthermore, several studies [9–12 Frieder et al. [25] tested the mathematical capabilities of ChatGPT on GHOSTS and handcrafted datasets.”
BackgroundMethodTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · cited 2× · see citing works
“Finally, the resulting model is called InstructGPT [8].”
BackgroundMethodKung, T. H., Cheatham, M., ChatGPT, Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., et al. (2022) · doi:10.1101/2022.12.19.22283643 · Cited by 4,115 · see citing works
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115 · cited 1× · see citing works
“Kung et al. [26] evaluated the performance of ChatGPT on the US Medical Licensing Exam, which includes three exams: Step 1, Step 2CK, and Step 3.”
BackgroundChatting and cheating: Ensuring academic integrity in the era of ChatGPT
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2190148 · Cited by 3,087 · cited 3× · see citing works
“Students can be given clear instructions on assignments and online exams on how to structure their assignments and answer their questions online [57].”
BackgroundChatting and Cheating. Ensuring academic integrity in the era of ChatGPT
Cotton, D., Cotton, P., & Shipway, J. R. (2023) · doi:10.35542/osf.io/mrz8h · Cited by 3,079 · see citing works
GPT-3: Its Nature, Scope, Limits, and Consequences
Floridi, L., & Chiriatti, M. (2020) · Minds and Machines · doi:10.1007/s11023-020-09548-1 · Cited by 2,519 · see citing works
ChatGPT: Bullshit spewer or the end of traditional assessments in higher education?
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.9 · Cited by 1,806 · see citing works
Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2022) · doi:10.1101/2022.12.23.521610 · Cited by 1,099 · see citing works
Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00819-6 · Cited by 1,099 · see citing works
ChatGPT for Education and Research: Opportunities, Threats, and Strategies
Rahman, M. M., & Watanobe, Y. (2023) · doi:10.20944/preprints202303.0473.v1 · Cited by 1,080 · see citing works
Self-citationCompetition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 1× · see citing works
“Alphacode [30] is a state-of-the-art LLM developed and trained specifically to support competitive-level programming.”
BackgroundPavlik, J. V. (2023) · Journalism & Mass Communication Educator · doi:10.1177/10776958221149577 · Cited by 972 · cited 1× · see citing works
“To address this issue, it is important for students, educators, and researchers to be aware of the limitations of LLMs, and these tools can only be used as supportive tools to enhance research and learning [59].”
BackgroundChatGPT listed as author on research papers: many scientists disapprove
Stokel-Walker, C. (2023) · Nature · doi:10.1038/d41586-023-00107-z · Cited by 868 · cited 1× · see citing works
“Recently, ChatGPT has authorized and credited published papers and preprints [58].”
BackgroundChatGPT for (Finance) research: The Bananarama Conjecture
Dowling, M., & Lucey, B. M. (2023) · Finance research letters · doi:10.1016/j.frl.2023.103662 · Cited by 611 · see citing works
The rise of <scp>ChatGPT</scp> : Exploring its potential in medical education
Lee, H. (2024) · Anatomical Sciences Education · doi:10.1002/ase.2270 · Cited by 596 · cited 1× · see citing works
“Moreover, ChatGPT has recently been applied to tasks in different domains such as software engineering [36], medical education [37], data augmentation [38], fifth industrial revolution [39], code generation [40], intelligent vehicles [41], and solving AI tasks [42].”
MethodAntaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023) · doi:10.1101/2023.01.22.23284882 · Cited by 563 · see citing works
Evaluating the Performance of ChatGPT in Ophthalmology
Antaki, F., Touma, S., Milad, D., El-Khoury, J., & Duval, R. (2023) · Ophthalmology Science · doi:10.1016/j.xops.2023.100324 · Cited by 563 · cited 1× · see citing works
“[44] 11 Ophthalmology question answering”
Are We There Yet? - A Systematic Literature Review on Chatbots in Education
Wollny, S., Schneider, J., Di Mitri, D., Weidlich, J., Rittberger, M., & Drachsler, H. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.654924 · Cited by 557 · cited 1× · see citing works
“Numerous studies have been conducted to explore the application of artificial intelligence (AI) in education, including chatbots [17], programming support [18,19], language models [2,20], and NLP tools [21].”
BackgroundAI bot ChatGPT writes smart essays — should professors worry?
Stokel-Walker, C. (2022) · Nature · doi:10.1038/d41586-022-04397-7 · Cited by 543 · see citing works
ChatGPT Goes to Law SchoolPreprint
Choi, J. H., Hickman, K. E., Monahan, A., & Schwarcz, D. B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4335905 · Cited by 522 · cited 1× · see citing works
“[48] 66 Examine the performance for law school exams”
BackgroundHuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
Shen, Y., Song, K., Tan, X., Li, D., Lu, W., & Zhuang, Y. (2023) · arXiv · doi:10.48550/arxiv.2303.17580 · Cited by 477 · cited 1× · see citing works
“Moreover, ChatGPT has recently been applied to tasks in different domains such as software engineering [36], medical education [37], data augmentation [38], fifth industrial revolution [39], code generation [40], intelligent vehicles [41], and solving AI tasks [42].”
MethodOpenAI ChatGPT Generated Literature Review: Digital Twin in Healthcare
Aydın, Ö., & Karaarslan, E. (2022) · doi:10.2139/ssrn.4308687 · Cited by 391 · see citing works
Mathematical Capabilities of ChatGPT
Frieder, S., Pinchetti, L., Chevalier, A., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, P. C., & Berner, J. (2023) · arXiv · doi:10.48550/arxiv.2301.13867 · Cited by 385 · cited 2× · see citing works
“[25] tested the mathematical capabilities of ChatGPT on GHOSTS and handcrafted datasets.”
MethodChatGPT: The End of Online Exam Integrity?
Susnjak, T. (2022) · arXiv · doi:10.48550/arxiv.2212.09292 · Cited by 370 · cited 2× · see citing works
“ChatGPT has made significant progress; however, there are concerns about misuse [12–14].”
BackgroundAn Analysis of the Automatic Bug Fixing Performance of ChatGPT
Sobania, D., Briesch, M., Hanna, C., & Petke, J. (2023) · 2023 IEEE/ACM International Workshop on Automated Program Repair (APR) · doi:10.1109/apr59189.2023.00012 · Cited by 338 · cited 2× · see citing works
“Nevertheless, ChatGPT has significantly outperformed traditional code-repair approaches [32].”
MethodIs ChatGPT A Good Translator? Yes With GPT-4 As The Engine
Jiao, W., Wang, W., Huang, J.-T., Wang, X., Shi, S., & Tu, Z. (2023) · arXiv · doi:10.48550/arxiv.2301.08745 · Cited by 317 · see citing works
Alshater, M. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4312358 · Cited by 304 · see citing works
Plagiarism in the age of massive generative pre-trained transformers (GPT-3)
Dehouche, N. (2021) · Ethics in Science and Environmental Politics · doi:10.3354/esep00195 · Cited by 296 · cited 1× · see citing works
“This poses a challenge to educators and researchers [57,60–62].”
BackgroundMalinka, K., Peresíni, M., Firc, A., Hujnák, O., & Janus, F. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588827 · Cited by 287 · see citing works
ChatGPT and Software Testing Education: Promises & Perils
Jalil, S., Rafi, S., LaToza, T. D., Moran, K., & Lam, W. (2023) · 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw58534.2023.00078 · Cited by 270 · cited 2× · see citing works
“[33] studied the performance of ChatGPT in solving software testing curriculum questions.”
BackgroundEngineering Education in the Era of ChatGPT: Promise and Pitfalls of Generative AI for Education
Qadir, J. (2022) · doi:10.36227/techrxiv.21789434.v1 · Cited by 264 · see citing works
Evaluating ChatGPT as an Adjunct for Radiologic Decision-Making
Rao, A., Kim, J., Kamineni, M., Pang, M., Lie, W., & Succi, M. D. (2023) · doi:10.1101/2023.02.02.23285399 · Cited by 257 · cited 1× · see citing works
“[45] 11 Adjunct for radiologic decision-making”
Biochemistry and Molecular Biology Education
Biology, I. U. O. B. A. M. (2007) · Biochemistry and Molecular Biology Education · doi:10.1002/(issn)1539-3429 · Cited by 204 · see citing works
Can GPT-3 pass a writer’s turing test?
Elkins, K., & Chun, J. (2020) · Journal of Cultural Analytics · doi:10.22148/001c.17212 · Cited by 202 · cited 1× · see citing works
“This poses a challenge to educators and researchers [57,60–62].”
BackgroundGilson, A., Safranek, C., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2022) · doi:10.1101/2022.12.23.22283901 · Cited by 172 · cited 1× · see citing works
“[27] investigated the performance of ChatGPT on the medical licensing exam.”
BackgroundAutomated Essay Writing: An AIED Opinion
Sharples, M. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-022-00300-7 · Cited by 171 · cited 1× · see citing works
“Sharples [51] encourages educators and learners to take advantage of the available capabilities of AI tools such as ChatGPT rather than forego their use.”
BackgroundA Survey of Learning-based Automated Program Repair
Zhang, Q., Fang, C., Ma, Y., Sun, W., & Chen, Z. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3631974 · Cited by 153 · see citing works
Chat with ChatGPT on Industry 5.0: Learning and Decision-Making for Intelligent Industries
Wang, F., Yang, J., Wang, X., Li, J., & Han, Q. (2023) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2023.123552 · Cited by 147 · cited 1× · see citing works
“Moreover, ChatGPT has recently been applied to tasks in different domains such as software engineering [36], medical education [37], data augmentation [38], fifth industrial revolution [39], code generation [40], intelligent vehicles [41], and solving AI tasks [42].”
MethodChatAug: Leveraging ChatGPT for Text Data AugmentationPreprint
Dai, H.-X., Liu, Z., Liao, W., Huang, X.-K., Wu, Z., Zhao, L., Liu, W., Liu, N.-H., Li, S., Zhu, D.-J., et al. (2023) · arXiv.org · doi:10.48550/arxiv.2302.13007 · Cited by 147 · cited 1× · see citing works
“Moreover, ChatGPT has recently been applied to tasks in different domains such as software engineering [36], medical education [37], data augmentation [38], fifth industrial revolution [39], code generation [40], intelligent vehicles [41], and solving AI tasks [42].”
MethodAre ChatGPT and AlphaCode going to replace programmers?
Castelvecchi, D. (2022) · Nature · doi:10.1038/d41586-022-04383-z · Cited by 138 · cited 2× · see citing works
“ChatGPT is a more general conversation engine, whereas Alphacode is more specialized for programming [31], even though these two systems use “virtually the same architecture” [31].”
BackgroundNatural Language Processing for Enhancing Teaching and Learning
Litman, D. J. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.9879 · Cited by 123 · cited 1× · see citing works
“Numerous studies have been conducted to explore the application of artificial intelligence (AI) in education, including chatbots [17], programming support [18,19], language models [2,20], and NLP tools [21].”
BackgroundChatGPT: Unlocking the future of NLP in finance
Zaremba, A., & Demir, E. (2023) · Modern Finance · doi:10.61351/mf.v1i1.43 · Cited by 121 · see citing works
A Bidirectional LSTM Language Model for Code Evaluation and Repair
Rahman, M. M., Watanobe, Y., & Nakamura, K. (2021) · Symmetry · doi:10.3390/sym13020247 · Cited by 103 · cited 3× · see citing works
“However, an RNN model has limitations and cannot handle extremely long-range dependencies in natural language texts [2].”
BackgroundMethodSelf-citationChatGPT FOR PROGRAMMING NUMERICAL METHODS
Kashefi, A., & Mukerji, T. (2023) · Journal of Machine Learning for Modeling and Computing · doi:10.1615/jmachlearnmodelcomput.2023048492 · Cited by 100 · see citing works
Welsh, M. (2022) · Communications of the ACM · doi:10.1145/3570220 · Cited by 95 · cited 2× · see citing works
“Some experts have expressed concern about the future of some common practices such as programming in the era of ChatGPT [13].”
BackgroundChoose your programming copilot
Sobania, D., Briesch, M., & Rothlauf, F. (2022) · Genetic and Evolutionary Computation Conference (GECCO) · doi:10.1145/3512290.3528700 · Cited by 91 · cited 1× · see citing works
“, CodeBERT, Codex, and PyMT5) have achieved state-of-the-art results for various programming tasks [55].”
BackgroundEthics in Science and Environmental Politics
(2026) · Ethics in Science and Environmental Politics · doi:10.3354/esep · Cited by 90 · see citing works
Rahman, M. M., Watanobe, Y., & Nakamura, K. (2020) · Applied Sciences · doi:10.3390/app10082973 · Cited by 71 · cited 2× · see citing works
“To assist programmers, deep-learning-based tools are introduced for code repair, completion, error detection, optimization, verification, and classification [1,2,19,20,53,54].”
BackgroundMethodSelf-citationWenzlaff, K., & Spaeth, S. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4302443 · Cited by 68 · cited 1× · see citing works
“Karsten [46] 8 Explain alternative and community finance and crowdfunding”
BackgroundChatGPT: Unlocking the Future of NLP in FinancePreprint
Zaremba, A., & Demir, E. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4323643 · Cited by 65 · cited 1× · see citing works
“Adam and Demir [47] 17 Financial application”
An Analysis of the Automatic Bug Fixing Performance of ChatGPT
(2023) · doi:10.48550/arxiv.2301.08653 · Cited by 49 · cited 2× · see citing works
“Nevertheless, ChatGPT has significantly outperformed traditional code-repair approaches [32].”
MethodA Neural Network Based Intelligent Support Model for Program Code Completion
Rahman, M. M., Watanobe, Y., & Nakamura, K. (2020) · Scientific Programming · doi:10.1155/2020/7426461 · Cited by 40 · cited 2× · see citing works
“Researchers have also used recurrent neural network (RNN) models in many applications because of their recurrent structure to remember dependencies in texts [1–3].”
BackgroundMethodSelf-citationImpact of Practical Skills on Academic Performance: A Data-Driven Analysis
Rahman, M. M., Watanobe, Y., Kiran, R. U., Thang, T. C., & Paik, I. (2021) · IEEE Access · doi:10.1109/access.2021.3119145 · Cited by 40 · cited 1× · see citing works
“Experienced programmers have demonstrated improvement in both professional and academic fields [52].”
BackgroundSelf-citationOnline Judge System: Requirements, Architecture, and Experiences
Watanobe, Y., Rahman, M. M., Matsumoto, T., Rage, U. K., & Ravikumar, P. (2022) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194022500346 · Cited by 40 · cited 1× · see citing works
“In the experiment, we used the “ Algorithms and Data Structures ” problem description from the Aizu Online Judge (AOJ) system [56] and verified the correctness of the codes generated by ChatGPT on the same platform and in a basic compiler.”
MethodSelf-citationWhite, J., Hays, S., Fu, Q., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.07839 · Cited by 39 · cited 1× · see citing works
“Moreover, ChatGPT has recently been applied to tasks in different domains such as software engineering [36], medical education [37], data augmentation [38], fifth industrial revolution [39], code generation [40], intelligent vehicles [41], and solving AI tasks [42].”
MethodIdentifying algorithm in program code based on structural features using CNN classification model
Watanobe, Y., Rahman, M. M., Amin, M. F. I., & Kabir, R. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04078-y · Cited by 30 · cited 1× · see citing works
“AI-based models have achieved plausible success on a variety of programming tasks, including code repair, summarization, completion, correction, classification, and generation [20,29].”
BackgroundSelf-citationUsing ChatGPT to produce code for a typical college‐level assignment
Wollowski, M. (2023) · AI Magazine · doi:10.1002/aaai.12086 · Cited by 16 · see citing works
Commentary: How the internet is changing the way we think, read and remember
Parslow, G. R. (2011) · Biochemistry and Molecular Biology Education · doi:10.1002/bmb.20514 · Cited by 13 · cited 1× · see citing works
“In the past, questions have been raised about Google and how this tool will change the way people think, read, and memorize [15].”
BackgroundA Novel Rule-Based Online Judge Recommender System to Promote Computer Programming Education
Rahman, M. M., Watanobe, Y., Rage, U. K., & Nakamura, K. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-79463-7_2 · Cited by 13 · cited 1× · see citing works
“Numerous studies have been conducted to explore the application of artificial intelligence (AI) in education, including chatbots [17], programming support [18,19], language models [2,20], and NLP tools [21].”
BackgroundSelf-citationChatGPT for Programming Numerical MethodsPreprint
Kashefi, A., & Mukerji, T. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.12093 · Cited by 8 · cited 1× · see citing works
“In the study [35], the programming capabilities of ChatGPT for solving numerical problems were investigated.”
MethodA Stacked Bidirectional LSTM Model for Classifying Source Codes Built in MPLs
Rahman, M. M., Watanobe, Y., Kiran, R. U., & Kabir, R. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-93733-1_5 · Cited by 7 · cited 3× · see citing works
“AI-based models have achieved plausible success on a variety of programming tasks, including code repair, summarization, completion, correction, classification, and generation [20,29].”
BackgroundMethodSelf-citationA Survey of Learning-based Automated Program RepairPreprint
Zhang, Q., Fang, C., Ma, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.03270 · Cited by 5 · cited 1× · see citing works
“To assist programmers, deep-learning-based tools are introduced for code repair, completion, error detection, optimization, verification, and classification [1,2,19,20,53,54].”
MethodEvaluation of Source Codes Using Bidirectional LSTM Neural Network
Rahman, M. M., Watanobe, Y., & Nakamura, K. (2020) · IEEE International Conference on Knowledge Innovation and Invention (ICKII) · doi:10.1109/ickii50300.2020.9318916 · Cited by 4 · cited 1× · see citing works
“To assist programmers, deep-learning-based tools are introduced for code repair, completion, error detection, optimization, verification, and classification [1,2,19,20,53,54].”
MethodSelf-citationChatGPT and Software Testing Education: Promises & Perils
Jalil, S., Rafi, S., LaToza, T. D., Moran, K., & Lam, W. (2023) · arXiv · doi:10.48550/arxiv.2302.03287 · cited 2×
“[33] studied the performance of ChatGPT in solving software testing curriculum questions.”
BackgroundMalinka, K., Perešíni, M., Firc, A., Hujňák, O., & Januš, F. (2023) · arXiv · doi:10.48550/arxiv.2303.11146
Flagged first, then by citations
Research works printed but not linked: 3
- Rahman, M.M. (2022). Data Analysis and Code Assessment Using Machine Learning Techniques for Programming Activities. [Ph.D. Thesis, The University of Aizu].
- Avila-Chauvet, L., Mejía, D., and Acosta Quiroz, C.O. (2023). Chatgpt as a Support Tool for Online Behavioral Task Programming. SSRN, Available online: https://ssrn.com/abstract=4329020.
- Jiao, W., Wang, W., Huang, J.t., Wang, X., and Tu, Z. (2023). Is ChatGPT a good translator? A preliminary study. arXiv.
Cited by
Sorted by citations
A survey on large language model (LLM) security and privacy: The Good, The Bad, and The Ugly
Yao, Y., Duan, J., Xu, K., Cai, Y., Sun, Z., & Zhang, Y. (2024) · High-Confidence Computing · doi:10.1016/j.hcc.2024.100211 · Cited by 1,083 · see citing works
Grassini, S. (2023) · Education Sciences · doi:10.3390/educsci13070692 · Cited by 1,048 · see citing works
“…authors have attempted to evaluate possible possibilities and problems related to the advent of AI technologies in the sphere of education [11,12,23–25], and the UNESCO has also published a report attempting to discuss the main challenges and the emerging ethical implications of AI in higher…”
BackgroundChallenges and Opportunities of Generative AI for Higher Education as Explained by ChatGPT
Michel-Villarreal, R., Vilalta-Perdomo, E., Salinas-Navarro, D. E., Thierry-Aguilera, R., & Gerardou, F. S. (2023) · Education Sciences · doi:10.3390/educsci13090856 · Cited by 877 · see citing works
“In line with previou studi s, th resul s revealed t t ChatGPT h s the potential to revolutionize education in different ways [3,12,14,34].”
BackgroundResultInfluentialChiu, T. K. F. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2023.2253861 · Cited by 852 · see citing works
“Many educators and policymakers have questions about what GenAI is capable of, and what its limitations are (Rahman & Watanobe, 2023).”
Bahroun, Z., Anane, C., Ahmed, V., & Zacca, A. (2023) · Sustainability · doi:10.3390/su151712983 · Cited by 784 · see citing works
Fan, Y., Tang, L., Le, H., Shen, K., Tan, S., Zhao, Y., Shen, Y., Li, X., & Gašević, D. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13544 · Cited by 805 · see citing works
Impact of the Implementation of ChatGPT in Education: A Systematic Review
Montenegro-Rueda, M., Fernández-Cerero, J., Fernández-Batanero, J. M., & López-Meneses, E. (2023) · Computers · doi:10.3390/computers12080153 · Cited by 509 · see citing works
“Rahman and Watanobe (2023) [19] Quantitative Bangladesh, Asia • Very useful as a support for educational work.”
BackgroundInfluentialFuture research recommendations for transforming higher education with generative AI
Chiu, T. K. F. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100197 · Cited by 511 · see citing works
Artificial intelligence in teaching and teacher professional development: A systematic review
Tan, X., Cheng, G., & Ling, M. H. (2025) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100355 · Cited by 420 · see citing works
Zhang, S., Zhao, X., Zhou, T., & Kim, J. H. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00467-0 · Cited by 409 · see citing works
“Using ChatGPT allows students to obtain quick and direct answers by simply asking questions, which may enhance their academic performance in the short term (Alshater, 2022; Rahman & Watanobe, 2023).”
MethodAI literacy and its implications for prompt engineering strategies
Knoth, N., Tolzin, A., Janson, A., & Leimeister, J. M. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100225 · Cited by 368 · see citing works
“…ChatGPT ’ s, capacity to enrich the educational experience by supporting a wide range of learning methodologies, including adaptive learning, personalized learning, and self-directed learning (Rahman & Watanobe, 2023; Rasul et al., 2023; Ruwe & Mayweg-Paus, 2023; Zhu, Sun, Luo, Li, & Wang, 2023).”
BackgroundYilmaz, R., & Karaoglan Yilmaz, F. G. (2023) · Computers in Human Behavior: Artificial Humans · doi:10.1016/j.chbah.2023.100005 · Cited by 344 · see citing works
Wang, J., & Fan, W. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04787-y · Cited by 322 · see citing works
“Conversely, other scholars have argued that ChatGPT hinders students ’ learning performance, learning perception, and higher-order thinking (Hays et al. 2024; Rahman and Watanobe, 2023).”
RetractedStöhr, C., Ou, A. W., & Malmström, H. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100259 · Cited by 319 · see citing works
“…students in acquiring information, answering questions (Chen et al., 2023), facilitating group discussions, and resolving problems instantaneously (Rahman & Watanobe, 2023), thereby enriching students ’ learning experiences, offering personalized support, and potentially boosting academic…”
BackgroundLiu, M., Ren, Y., Nyagoga, L. M., Stonier, F., Wu, Z., & Yu, L. (2023) · Future in Educational Research · doi:10.1002/fer3.10 · Cited by 267 · see citing works
Bulathwela, S., Pérez-Ortiz, M., Holloway, C., Cukurova, M., & Shawe-Taylor, J. (2024) · Sustainability · doi:10.3390/su16020781 · Cited by 269 · see citing works
“Recent experiments show the viability of using LLMs as the base for building intelligent educational systems capable of question generation, question answering, learning content generation, and summarisation [41,45,106,107].”
Higher education students’ perceptions of ChatGPT: A global study of early reactions
Ravšelj, D., Keržič, D., Tomaževič, N., Umek, L., Brezovar, N., A. Iahad, N., Abdulla, A. A., Akopyan, A., Aldana Segura, M. W., AlHumaid, J., et al. (2025) · PLOS ONE · doi:10.1371/journal.pone.0315011 · Cited by 376 · see citing works
Drivers and Consequences of ChatGPT Use in Higher Education: Key Stakeholder Perspectives
Hasanein, A. M., & Sobaih, A. E. E. (2023) · European Journal of Investigation in Health Psychology and Education · doi:10.3390/ejihpe13110181 · Cited by 329 · see citing works
“…reliance on biased data [6], the limitation of knowledge to information available up to 2021 [7], relying on inaccurate or fabricated content, including fictitious citations [8], and over-reliance on ChatGPT might have negative impact students’ critical thinking and problem-solving abilities [9].”
BackgroundInfluentialPaoli, S. D. (2023) · Social Science Computer Review · doi:10.1177/08944393231220483 · Cited by 301 · see citing works
“Others focused on the application of ChatGPT such as in marketing research (Peres et al., 2023), or education (Rahman & Watanobe, 2023).”
MethodLiu, M., Okuhara, T., Chang, X., Shirabe, R., Nishiie, Y., Okada, H., & Kiuchi, T. (2024) · Journal of Medical Internet Research · doi:10.2196/60807 · Cited by 231 · see citing works
Tossell, C. C., Tenhundfeld, N. L., Momen, A., Cooley, K., & de Visser, E. J. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3355015 · Cited by 218 · see citing works
“Beyond support in the traditional educational settings, there have been several applications of AI-based tools to enhance personalized learning, better support students with disabilities and inclusivity, and help to make teaching and grading more efficient [10].”
BackgroundYaseen, H., Mohammad, A. S., Ashal, N. M., Abusaimeh, H., Ali, A. A. A., & Sharabati, A. A. (2025) · Sustainability · doi:10.3390/su17031133 · Cited by 336 · see citing works
Empowering education development through AIGC: A systematic literature review
Chen, X., Hu, Z., & Wang, C. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12549-7 · Cited by 222 · see citing works
Educational Applications of the ChatGPT AI System: A Systematic Review Research
İPEK, Z. H., Gözüm, A. İ. C., Papadakis, S., & Kalogiannakis, M. (2023) · Educational Process International Journal · doi:10.22521/edupij.2023.123.2 · Cited by 209 · see citing works
“…can provide detailed answers to students preparing for tests and exams, providing them with guidance on exam scope and the necessary course content to be learned (Choi et al., 2023; Duong & Solomon, 2023; Huh, 2023; Nisar & Aslam, 2023; Qadir, 2022; Rahman & Watanobe, 2023; Sok & Heng, 2023).”
BackgroundPerkins, M., Roe, J., Postma, D., McGaughran, J., & Hickerson, D. (2024) · Journal of Academic Ethics · doi:10.1007/s10805-023-09492-6 · Cited by 234 · see citing works
“This advancement in AI has sparked serious ethical concerns related to the potential misrepresentation of authorship and authenticity of academic work (Cotton et al., 2023 ; Perkins, 2023; Rahman & Watanobe, 2023 ; Rodgers et al., 2023 ; Rudolph et al., 2023).”
Ansari, A. N., Ahmad, S., & Bhutta, S. M. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12223-4 · Cited by 198 · see citing works
“The studies reported that teachers utilize ChatGPT for various purposes, including personal and professional learning, improving teaching practices and student outcomes, assessing students’ learning, and academic research (Annuš, 2023; Mhlanga, 2023; Rahman & Watanobe, 2023; Seetharaman, 2023).”
Sohail, S. S., Farhat, F., Himeur, Y., Nadeem, M., Madsen, D. Ø., Singh, Y., Atalla, S., & Mansoor, W. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101675 · Cited by 192 · see citing works
“Rahman and Watanobe (2023) has experimentally shown that ChatGPT can be used to solve both technical problems, such as engineering and computer programming, and non-technical problems, such as language and literature.”
Tian, W., Ge, J., Zhao, Y., & Zheng, X. (2024) · Frontiers in Psychology · doi:10.3389/fpsyg.2024.1268549 · Cited by 232 · see citing works
Sun, D., Boudouaia, A., Zhu, C., & Li, Y. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00446-5 · Cited by 187 · see citing works
“In their study, Rahman and Watanobe (2023) highlight the following drawbacks of ChatGPT in programming education: a lack of capability in reasoning through simple logic, potential biases, difficulties in complex logical reasoning, and an inability to process visual information.”
BackgroundA systematic literature review of empirical research on ChatGPT in education
Albadarin, Y., Saqr, M., Pope, N., & Tukiainen, M. (2024) · Discover Education · doi:10.1007/s44217-024-00138-2 · Cited by 191 · see citing works
Samala, A. D., Rawas, S., Wang, T., Reed, J. M., Kim, J., Howard, N.-J., & Ertz, M. (2025) · Education and Information Technologies · doi:10.1007/s10639-024-12936-0 · Cited by 229 · see citing works
“If the instructions are adequate, ChatGPT can produce abstracts for fictional scientific papers (Fergus et al., 2023; Rahman & Watanobe, 2023; Sedaghat, 2023).”
Kofinas, A. K., Tsay, C. H., & Pike, D. (2025) · British Journal of Educational Technology · doi:10.1111/bjet.13585 · Cited by 207 · see citing works
Hashem, R., Ali, N., El Zein, F., Fidalgo, P., & Abu Khurma, O. (2023) · Research and Practice in Technology Enhanced Learning · doi:10.58459/rptel.2024.19023 · Cited by 173 · see citing works
“…ChatGPT can swiftly generate lesson plans and educational materials of high quality that are customised to address the particularities of each class and student which is also confirmed by previous studies (Farrokhnia et al., 2023; Kasneci et al., 2023; Qadir, 2022; Rahman & Watanobe, 2023).”
Applying large language models and chain-of-thought for automatic scoring
Lee, G.-G., Latif, E., Wu, X., Liu, N., & Zhai, X. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100213 · Cited by 186 · see citing works
Al-khresheh, M. H. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100218 · Cited by 163 · see citing works
ChatGPT—A Challenging Tool for the University Professors in Their Teaching Practice
Kiryakova, G., & Angelova, N. (2023) · Education Sciences · doi:10.3390/educsci13101056 · Cited by 171 · see citing works
Lu, J., Zheng, R., Gong, Z., & Xu, H. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3369690 · Cited by 144 · see citing works
“ThereisnodoubtthatAIGChasalso evokedpolarizedreactionsineducation,havingasignificant andfar-reachingimpactonschooleducationinashortperiod [7].”
Assessing AI adoption in developing country academia: A trust and privacy-augmented UTAUT framework
Rana, M. M., Siddiqee, M. S., Sakib, M. N., & Ahamed, M. R. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e37569 · Cited by 143 · see citing works
ChatGPT in higher education - a synthesis of the literature and a future research agenda
Bhullar, P. S., Joshi, M., & Chugh, R. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12723-x · Cited by 139 · see citing works
“Achieving a bal - ance between using AI and the advantages of human guidance and mentorship is a persistent challenge that requires attention (Neumann et al., 2023 ; Rahman et al., 2023 ).”
ChatGPT in Teaching and Learning: A Systematic Review
Ali, D., Fatemi, Y., Boskabadi, E., Nikfar, M., Ugwuoke, J., & Ali, H. (2024) · Education Sciences · doi:10.3390/educsci14060643 · Cited by 146 · see citing works
Kavadella, A., da Silva, M. A. D., Kaklamanos, E. G., Stamatopoulos, V., & Giannakopoulos, K. (2023) · JMIR Medical Education · doi:10.2196/51344 · Cited by 169 · see citing works
“…perspective, students using LLMs have the potential to develop new competencies including 21st-century soft skills, such as self-reflection abilities, problem-solving skills, creative and critical thinking, and collaboration, thus becoming motivated and autonomous learners [3,4,16,33,49].”
InfluentialRole of activity-based learning and ChatGPT on students' performance in education
Al Shloul, T., Mazhar, T., Abbas, Q., Iqbal, M., Ghadi, Y. Y., Shahzad, T., Mallek, F., & Hamam, H. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100219 · Cited by 173 · see citing works
“Students can collaborate and engage in communication through the utilization of ChatGPT, hence customizing their learning experience (Rahman et al., 2023).”
BackgroundBoudouaia, A., Mouas, S., & Kouider, B. (2024) · Journal of Educational Computing Research · doi:10.1177/07356331241247465 · Cited by 169 · see citing works
“According to Rahman and Watanobe (2023), ChatGPT-4 can generate human-like text based on the input it receives.”
The Educational Affordances and Challenges of ChatGPT: State of the Field
Crompton, H., & Burke, D. (2024) · TechTrends · doi:10.1007/s11528-024-00939-0 · Cited by 141 · see citing works
“One of the advantages of ChatGPT is that it can be used across all learner levels, from primary grades through higher education (Rahman & Watanobe, 2023).”
Jukiewicz, M. (2024) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2024.101522 · Cited by 132 · see citing works
Gemini versus ChatGPT: applications, performance, architecture, capabilities, and implementation
Rane, N., Choudhary, S., & Rane, J. (2024) · Journal of Applied Artificial Intelligence · doi:10.48185/jaai.v5i1.1052 · Cited by 127 · see citing works
Al-Abdullatif, A. M., & Alsubaie, M. A. (2024) · Behavioral Sciences · doi:10.3390/bs14090845 · Cited by 126 · see citing works
“Launched as a successor to earlier models, ChatGPT is designed to understand and generate human-like text based on the vast amount of information it has been trained on [3,4].”
Opportunities, Challenges, and Strategies for Using ChatGPT in Higher Education: A Literature Review
Sok, S., & Heng, K. (2024) · Journal of Digital Educational Technology · doi:10.30935/jdet/14027 · Cited by 120 · see citing works
“Given that ChatGPT could be leveraged to produce written assignments that are indistinguishable from humans (Rahman & Watanobe, 2023; Thurzo et al., 2023), it is hard, if not impossible, for lecturers to identify whether the work is written by students or ChatGPT.”
BackgroundGenerative Artificial Intelligence in Education, Part One: the Dynamic Frontier
Hsu, Y.-C., & Ching, Y.-H. (2023) · TechTrends · doi:10.1007/s11528-023-00863-9 · Cited by 118 · see citing works
“…and smooth performance while interacting with users, people of all pro-fessions have leveraged this software tool for various tasks and purposes involving text generation, such as composing simple to relative complex textual codes of computer programming (professional) (Rahman & Watanobe, 2023).”
Method(2023) · doi:10.1057/s41307-023-00323-2 · Cited by 115 · see citing works
“5 and GPT-4 models, which has been the subject of extensive discussion regarding its potential influence on academic integrity (Cotton et al. 2023; Perkins 2023; Rahman and Watanobe 2023; Rudolph et al. 2023; Uzun 2023).”
Ahmed, Z., Shanto, S. S., Rime, M. H. K., Morol, M. K., Fahad, N., Hossen, M. J., & Abdullah-Al-Jubair, M. (2024) · IEEE Access · doi:10.1109/access.2024.3461874 · Cited by 123 · see citing works
“Several works have been depicted, providing an overall synopsis of the impact of ChatGPT in education [10], [11], [12].”
BackgroundInfluentialScaffolding Computational Thinking With ChatGPT
Liao, J., Zhong, L., Zhe, L., Xu, H., Liu, M., & Xie, T. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3392896 · Cited by 113 · see citing works
How understanding large language models can inform the use of ChatGPT in physics education
Polverini, G., & Gregorcic, B. (2023) · European Journal of Physics · doi:10.1088/1361-6404/ad1420 · Cited by 131 · see citing works
Critical Thinking in the Age of AI: A Systematic Review of AI’s Effects on Higher Education
Melisa, R., Ashadi, A., Triastuti, A., Hidayati, S., Salido, A., Ero, P. E. L., Marlini, C., Zefrin, Z., & Al Fuad, Z. (2025) · Educational Process International Journal · doi:10.22521/edupij.2025.14.31 · Cited by 117 · see citing works
“Rahman and Watanobe (2023) and Nikolic et al. (2023) addressed the challenges ChatGPT poses in terms of plagiarism and assessment authenticity, suggesting that the tool's capacity to produce near-perfect answers raises questions about its influence on fostering or hindering critical analysis.”
Alshammarı, S. H., & Babu, E. (2025) · Scientific Reports · doi:10.1038/s41598-025-91634-4 · Cited by 109 · see citing works
“…educational needs thereby making them feel satisfied and encouraged to continue utilising in future 19 , better personalization of feedback and conversation, enhanced accessibility, better lesson planning, better integration of assessment and infusing new strategies in teaching and learning 46 .”
The impact of chatbots based on large language models on second language vocabulary acquisition
Zhang, Z., & Huang, X. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e25370 · Cited by 107 · see citing works
“Researchers have expressed optimism about LLMs in education [24,37], highlighting their potential to enhance learning and teaching experiences, reduce anxiety, increase interest, and provide psychological satisfaction through interactions with AI chatbots.”
BackgroundGuo, K., & Li, D. (2024) · System · doi:10.1016/j.system.2024.103362 · Cited by 129 · see citing works
A Systematic Review of the Limitations and Associated Opportunities of ChatGPT
Cong‐Lem, N., Soyoof, A., & Tsering, D. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2344142 · Cited by 125 · see citing works
“Scholars have discussed various ways ChatGPT can support this domain, including creating course materials, designing lesson plans and assessments, providing feedback, explaining complex knowledge, and personalizing the learning experience (Clark, 2023; Rahman & Watanobe, 2023).”
AI hype as a cyber security risk: the moral responsibility of implementing generative AI in business
Humphreys, D., Koay, A., Desmond, D., & Mealy, E. (2024) · AI and Ethics · doi:10.1007/s43681-024-00443-4 · Cited by 115 · see citing works
“Some of this literature focuses on the threat which generative AI will have for jobs [46, 47 ], bias in training data affecting its output [ 48, 49], or a diminishing of critical thinking and problem-solving skills amongst users [50].”
Dalgıç, A., Yaşar, E., & Demir, M. (2024) · Journal of Hospitality Leisure Sport & Tourism Education · doi:10.1016/j.jhlste.2024.100481 · Cited by 99 · see citing works
Students’ Experiences of Using ChatGPT in an Undergraduate Programming Course
Haindl, P., & Weinberger, G. (2024) · IEEE Access · doi:10.1109/access.2024.3380909 · Cited by 108 · see citing works
“Itcanbe summarized that only two studies [20], [21] incorporated students’ experiences about using ChatGPT in programming education.”
BackgroundKosar, T., Ostojić, D., Liu, Y. D., & Mernik, M. (2024) · Mathematics · doi:10.3390/math12050629 · Cited by 98 · see citing works
“On the positive side, the benefits of using ChatGPT in education are well argued for [7].”
BackgroundFošner, A. (2024) · Sustainability · doi:10.3390/su16198668 · Cited by 98 · see citing works
Prather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3689187.3709614 · Cited by 97 · see citing works
Ulla, M. B., Perales, W. F., & Busbus, S. O. (2023) · Learning Research and Practice · doi:10.1080/23735082.2023.2257252 · Cited by 95 · see citing works
“Such drawbacks were also evident in previous studies by Jeon and Lee (2023), Kohnke et al. (2023), and Rahman and Watanobe (2023).”
BackgroundResultInfluentialExamining Students’ Acceptance and Use of ChatGPT in Saudi Arabian Higher Education
Sobaih, A. E. E., Elshaer, I. A., & Hasanein, A. M. (2024) · European Journal of Investigation in Health Psychology and Education · doi:10.3390/ejihpe14030047 · Cited by 123 · see citing works
“[2,6], to understand the adoption of AI, e.g., ChatGPT, in education.”
BackgroundMustapha, K. B. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.103066 · Cited by 96 · see citing works
Groothuijsen, S., van den Beemt, A., Remmers, J. C., & van Meeuwen, L. W. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100290 · Cited by 92 · see citing works
Chien, C.-C., Chan, H.-Y., & Hou, H.-T. (2025) · Journal of Research on Technology in Education · doi:10.1080/15391523.2024.2338085 · Cited by 90 · see citing works
“Correct or partially correct feedback and suggestions can also be obtained based on the problem description, contextualization, and question and answer style (Rahman & Watanobe, 2023).”
InfluentialGustilo, L., Ong, E., & Lapinid, M. R. (2024) · International Journal for Educational Integrity · doi:10.1007/s40979-024-00153-8 · Cited by 89 · see citing works
“While they seem to be performing better and faster than their human counterparts, the output from AI systems is not perfect, as attested by previous studies (Lim et al. 2023; Rahman & Watanobe 2023) current findings.”
BackgroundMethodResultInfluentialΠέλλας, Ν. (2023) · Smart Learning Environments · doi:10.1186/s40561-023-00276-4 · Cited by 87 · see citing works
“Therefore, it is crucial to strike a balance between harnessing the creative potential of AI-generated videos and ensuring that these technologies are developed and deployed with a keen awareness of their impact on sociodemographic inclusivity and equity (Rahman & Watanobe, 2023).”
BackgroundWhy do students use ChatGPT? Answering through a triangulation approach
Niloy, A. C., Bari, M. A., Sultana, J., Chowdhury, R., Raisa, F. M., Islam, A., Mahmud, S., Jahan, I., Sarkar, M., Akter, S., et al. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100208 · Cited by 104 · see citing works
Artificial Intelligence Integration: Pedagogical Strategies and Policies at Leading Universities
Alqahtani, N., & Wafula, Z. (2025) · Innovative Higher Education · doi:10.1007/s10755-024-09749-x · Cited by 84 · see citing works
“Many educators and students have a limited understanding of how it can be used in educational contexts, questioning its capabilities and limitations (Rahman & Watanobe, 2023 ).”
Zhang, X., Zhang, P., Shen, Y., Liu, M., Wang, Q., Gašević, D., & Fan, Y. (2024) · Frontiers of Digital Education · doi:10.1007/s44366-024-0028-5 · Cited by 83 · see citing works
Beyond the hype: exploring faculty perceptions and acceptability of AI in teaching practices
Ofosu-Ampong, K. (2024) · Discover Education · doi:10.1007/s44217-024-00128-4 · Cited by 79 · see citing works
ChatGPT as an important tool in organizational management: A review of the literature
Ayinde, L., Wibowo, M. P., Ravuri, B., & Emdad, F. B. (2023) · Business Information Review · doi:10.1177/02663821231187991 · Cited by 74 · see citing works
Kayalı, B., Yavuz, M., Balat, Ş., & Çalışan, M. (2023) · Australasian Journal of Educational Technology · doi:10.14742/ajet.8915 · Cited by 103 · see citing works
“It can also make extra contributions in terms of providing instant feedback for students and creating scenario-based learning environments (Rahman & Watanobe, 2023).”
BackgroundEnhancing academic integrity among students in GenAI Era:A holistic framework
Rasul, T., Nair, S., Kalendra, D., Balaji, M. S., Santini, F. D. O., Ladeira, W. J., Rather, R. A., Yasin, N., Rodriguez, R. V., Kokkalis, P., et al. (2024) · The International Journal of Management Education · doi:10.1016/j.ijme.2024.101041 · Cited by 82 · see citing works
Meo, S. A., Al-Masri, A. A., Alotaibi, M., Meo, M. Z. S., & Meo, M. O. S. (2023) · Healthcare · doi:10.3390/healthcare11142046 · Cited by 74 · see citing works
“ChatGPT may develop the option of cheating on online exams and minimize critical thinking skills [3].”
BackgroundJing, Y., Wang, H., Chen, X., & Wang, C. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-02751-w · Cited by 82 · see citing works
“…of learners learning programming with the assistance of ChatGPT, such as the opportunities for scientists and engineers to learn programming using AI tools like ChatGPT (Guo, 2023), and the methods and limitations of ChatGPT in enhancing learners ’ programming skills (Rahman and Watanobe, 2023).”
Liu, M., Okuhara, T., Dai, Z., Huang, W., Gu, L., Okada, H., Furukawa, E., & Kiuchi, T. (2025) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105673 · Cited by 73 · see citing works
AI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review
Gu, X., & Ericson, B. J. (2025) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3702652.3744217 · Cited by 113 · see citing works
“Amid reflections on how generative AI tools could reshape computing education objectives [51, 64, 96], teaching AI literacy emerged as a potential path forward.”
AI Literacy in K-12 and Higher Education in the Wake of Generative AI: An Integrative Review
Gu, X., & Ericson, B. J. (2025) · arXiv · doi:10.48550/arxiv.2503.00079 · Cited by 73
“Amid reflections on how generative AI tools could reshape computing education objectives [51, 64, 96], teaching AI literacy emerged as a potential path forward.”
Li, Y., Sadiq, G., Qambar, G., & Zheng, P. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12981-9 · Cited by 86 · see citing works
“…of information, summa - rization of research articles, hypothesis development, writing outlines for articles and theses, writing initial drafts, providing feedback, and evaluating written content (Atlas, 2023; Halaweh, 2023 ; Karakose et al., 2023 ; Kasneci et al., 2023 ; Rahman & Watanobe, 2023 ).”
InfluentialStudents’ appraisals post-ChatGPT use: Students’ narrative after using ChatGPT for writing
Nugroho, A., Andriyanti, E., Widodo, P., & Mutiaraningrum, I. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2024.2319184 · Cited by 95 · see citing works
“This technology improves automated speech recognition to perform informative automatic tasks (Rahman & Watanobe, 2023).”
Perspectives of Generative AI in Chemistry Education Within the TPACK Framework
Feldman-Maggor, Y., Blonder, R., & Alexandron, G. (2024) · Journal of Science Education and Technology · doi:10.1007/s10956-024-10147-3 · Cited by 100 · see citing works
“…assessment methods (Chiu, 2023), point to the role of AI-related professional development (Nazaretsky et al., 2022) and transparent system design (Feldman-Mag-gor et al., 2024a) in the acceptance of educational AI, and express concerns over their possible misuse (e.g., Rahman & Watanobe, 2023).”
Xu, T., Liu, Y., Jin, Y., Qu, Y.-Y., Bai, J., Zhang, W., & Zhou, Y. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13530 · Cited by 95 · see citing works
Caratıquıt, K., & CARATIQUIT, L. J. (2023) · Journal of Social Humanity and Education · doi:10.35912/jshe.v4i1.1558 · Cited by 90 · see citing works
Perceptions of Higher Education Students towards ChatGPT Usage
Das, S. R., & Madhusudan, J. V. (2024) · International Journal of Technology in Education · doi:10.46328/ijte.583 · Cited by 103 · see citing works
“Other limitations include a lack of contextual understanding and limited depth of comprehension (Farrokhnia et al., 2023; Rahman and Watanobe, 2023).”
BackgroundDo large language models have a legal duty to tell the truth?
Wachter, S., Mittelstadt, B., & Russell, C. (2024) · Royal Society Open Science · doi:10.1098/rsos.240197 · Cited by 85 · see citing works
Nasr, N. R., Tu, C., Werner, J., Bauer, T., Yen, C., & Sujo-Montes, L. E. (2025) · Education Sciences · doi:10.3390/educsci15091198 · Cited by 104 · see citing works
“On the other hand, GenAI’s tendency to generate inaccurate information suggests the possibility that its uncritical adoption could lead to shallow learning outcomes (Rahman & Watanobe, 2023).”
Application of ChatGPT in Higher Education and Research – A Futuristic Analysis
Aithal, P. S., & Aithal, S. (2023) · International Journal of Applied Engineering and Management Letters · doi:10.47992/ijaeml.2581.7000.0193 · Cited by 71 · see citing works
Exploring the Role of ChatGPT in Education: Applications and Challenges
Mosaiyebzadeh, F., Pouriyeh, S., Parizi, R. M., Dehbozorgi, N., Dorodchi, M., & Batista, D. M. (2023) · Conference on Information Technology Education · doi:10.1145/3585059.3611445 · Cited by 80 · see citing works
“It has been considered a cuttingedge technology that has the potential to revolutionize various fields such as education [28], healthcare [5], programming [27], and agriculture [4], in the next few decades.”
BackgroundAnh, V. T. K., & Nguyen, H. T. T. (2024) · Journal of University Teaching and Learning Practice · doi:10.53761/fr1rkj58 · Cited by 121 · see citing works
“Rahman et al. (2023) carried out a survey with students and teachers to explore the impact of ChatGPT in a programming course.”
Background(2023) · International Research Journal of Modernization in Engineering Technology and Science · doi:10.56726/irjmets45212 · Cited by 74 · see citing works
Youssef, E., Medhat, M., Abdellatif, S., & Al Malek, M. (2024) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2024.100316 · Cited by 65 · see citing works
Large Language Models in Education: A Systematic Review
Dong, B., Bai, J., Xu, T., & Zhou, Y. (2024) · International Conference on Computer Science and Technologies in Education (CSTE) · doi:10.1109/cste62025.2024.00031 · Cited by 63 · see citing works
“However, the existing research also highlights the limitations of generative AI, including deficiencies in training, a lack of common sense, and challenges in reasoning [7].”
BackgroundMamo, Y., Crompton, H., Burke, D., & Nickel, C. (2024) · TechTrends · doi:10.1007/s11528-024-00954-1 · Cited by 65 · see citing works
“ChatGPT and other generative models involve the collection and processing of personal data, which raises concerns about privacy and data security (Rahman & Watanobe, 2023).”
Cambaz, D., & Zhang, X. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630958 · Cited by 62 · see citing works
“LLM-based code generation models can create additional learning resources in programming education, such as practice exercises [8, 10, 13, 19, 20] and sample answers [1, 2, 13, 19, 20].”
BackgroundMethodInfluential
- Authors
- Rahman, M. M., & Watanobe, Y.
- Affiliations
- University of Aizu (JP)
- Corresponding author
- Rahman, Md. Mostafizer
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2023-05-08
- Volume
- 13
- Issue
- 9
- Pages
- 5783
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W4381982883
- Semantic Scholar
- 3b69ec0fc17c5134c9b002bebe283e0189bfdae5
- Cited by
- 1,080
- References
- 58
- Citations by year
- 2026: 193 · 2025: 338 · 2024: 425 · 2023: 124
- FWCI
- 253.56
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- Preprints.org (acceptedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Japan Society for the Promotion of Science (JSPS) KAKENHI (23H03508)
- Topics
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- ChatGPT, large language models, artificial intelligence in education, academic dishonesty, critical thinking, programming education, code generation, code correction, student and teacher perceptions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
