CoChat
Sign Up
Cited by 421Artificial Intelligence in EducationArtificial Intelligence in Healthcare and EducationAI in Service Interactions

War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education

Open access
Jürgen Rudolph, Shannon Tan, Samson Tan·2023·Journal of Applied Learning & Teaching
DOI 10.37074/jalt.2023.6.1.23
Open this paper in CoChat — free accountOpen PDF

Abstract

Developments in the chatbot space have been accelerating at breakneck speed since late November 2022. Every day, there appears to be a plethora of news. A war of competitor chatbots is raging amidst an AI arms race and gold rush. These rapid developments impact higher education, as millions of students and academics have started using bots like ChatGPT, Bing Chat, Bard, Ernie and others for a large variety of purposes. In this article, we select some of the most promising chatbots in the English and Chinese-language spaces and provide their corporate backgrounds and brief histories. Following an up-to-date review of the Chinese and English-language academic literature, we describe our comparative method and systematically compare selected chatbots across a multi-disciplinary test relevant to higher education. The results of our test show that there are currently no A-students and no B-students in this bot cohort, despite all publicised and sensationalist claims to the contrary. The much-vaunted AI is not yet that intelligent, it would appear. GPT-4 and its predecessor did best, whilst Bing Chat and Bard were akin to at-risk students with F-grade averages. We conclude our article with four types of recommendations for key stakeholders in higher education: (1) faculty in terms of assessment and (2) teaching & learning, (3) students and (4) higher education institutions.

AI snapshot

Written from the paper’s full text.

TL;DR

This article reviews the rapid chatbot arms race and systematically compares ChatGPT-3.5, ChatGPT/GPT-4, Bing Chat, and Bard on higher-education-relevant tasks. It finds no A- or B-level chatbot performers, with GPT-4 best and Bing Chat and Bard averaging F grades, then offers recommendations for faculty, students, and institutions.

Key findings

  • In a 15-question multi-disciplinary test, there were “no A-students and no B-students” among the selected chatbots; GPT-4 performed best, with ChatGPT-3.5 a close second, while Bing Chat and Bard were “akin to at-risk students with F-grade averages.”
  • GPT-4 scored the most A’s (four); ChatGPT-3.5 and Bing Chat each received an A only for their math answers, and Bard received no A’s.
  • The authors’ English-language literature review included 48 academic papers; their Chinese-language CNKI search reviewed 130 articles, of which 66 were directly related, plus seven research articles from ChatGPT/education searches.
  • The paper concludes with four sets of recommendations for higher education stakeholders: faculty assessment, teaching and learning, students, and higher education institutions.

Methods

  • The authors selected ChatGPT-3.5 (free), ChatGPT Plus based on GPT-4, Bing Chat, and Alphabet’s Bard for systematic comparison; they could not access Baidu’s Ernie, making the test “involuntarily US-centric.”
  • They used 15 questions from disciplines including sociology, business, mathematics, history, economics, philosophy, American literature, psychology, art history, and German literature, plus Chinese-language non-fiction, literature searches, and annotation tasks.
  • Responses were graded like student work using a US system: A = 90% and above, B = 80–89%, C = 70–79%, D = 60–69%, F = 0–59%; grading compared accuracy, comprehensiveness, and clarity, with labour divided by expertise.

Limitations

  • The authors state their snapshot and assessment are “necessarily preliminary” because chatbot developments continue at breakneck speed.
  • They could not access Ernie despite efforts, so the comparison does not represent both AI superpowers and is US-centric.
  • The text makes evident that no per-question marking rubrics were created, and the authors note they were not always experts on the 15 questions, though they felt sufficiently confident to assess them.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.37074/jalt.2023.6.1.23

Cites

83 of 83 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 97 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    Bing Chat and Bard had F-grade averages, akin to at-risk students, in the comparative test.

    “GPT-4 and its predecessor did best, whilst Bing Chat and Bard were akin to at-risk students with F-grade averages.”
    Quote in the paperSubstantial claim
  • Findingown data

    In the authors' comparative test, none of the selected chatbots achieved an A or B grade.

    “The results of our test show that there are currently no A-students and no B-students in this bot cohort, despite all publicised and sensationalist claims to the contrary.”
    Quote in the paperSubstantial claim
  • Findingown data

    GPT-4 and its predecessor performed best among the selected chatbots in the comparative test.

    “GPT-4 and its predecessor did best, whilst Bing Chat and Bard were akin to at-risk students with F-grade averages.”
    Quote in the paperSubstantial claim
  • Findingreasoning

    Searches that combined Bing Chat, Bard or Ernie with higher education yielded no academic articles.

    “Searches that combined Bing Chat, Bard or Ernie with higher education (e.g.“Bing Chat and higher education”) yielded no academic articles”
    Quote in the paperNot a substantial claim
  • Findingown data

    The bots did quite well on the history question (Q4).

    “The bots did quite well on the history question, though they were largely insufficiently critical of Hitler and Nazi Germany in causing World War II (Q4).”
    Quote in the paperNot a substantial claim
  • Classified as probably not a substantial claim. 63
  • May combine two claims. 52

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    The literature review uncovered main themes: assessment and plagiarism concerns, discipline-specific considerations, research and how to credit chatbots, higher education discourses in popular and social media, teaching and learning, plugins at present and in the future, and higher education for employability.

    “uncovered the following main themes: assessment and plagiarism concerns, discipline-specific considerations (e.g. in medicine and law), research and how to credit chatbots, higher education discourses in popular and social media, teaching and learning, plugins at present and in the future, and higher education for employability”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    The current extensive literature review eventually led to the inclusion of 48 English-language academic papers.

    “Our current extensive literature review (that eventually led to the inclusion of 48 English-language academic papers in our article)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Findingown data

    The 1000-word summary essay on Steinbeck's The Grapes of Wrath (Q7) was unevenly executed.

    “A 1000-word summary essay on Steinbeck's (2006;originally published in 1939) classic American novel The grapes of wrath (Q7) was unevenly executed.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Bing provided substandard references and questionably described Douglas McGregor's Theories X and Y as a theory of motivation (Q8).

    “Bing provided the usual substandard references and questionably described Douglas McGregor's Theories X and Y as a theory of motivation (it is usually considered a leadership or management theory).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Bard stated it was still working to learn more languages and could not do Q13.

    “Bard stated that it was 'still working to learn more languages, so I can't do that just yet'.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For Q8, ChatGPT-3.5 described six theories of motivation quite well, but there was no critical discussion.

    “For Q8, ChatGPT-3.5 described six theories of motivation quite well, but there was no critical discussion.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The results show that there are currently no A-students and no B-students in this bot cohort.

    “The results of our test show that there are currently no A-students and no B-students in this bot cohort, despite all publicised and sensationalist claims to the contrary.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The other bots' responses to Q12 were even more disappointing.

    “The other bots' responses were even more disappointing: 'I can't give a response to that right now.Let's try a different topic' (Bing Chat).And: 'As an LLM, I am trained to understand and respond only to a subset of languages at this time and can't provide assistance with that.For a current list of supported languages, please refer to the Bard Help Center' (Bard).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingcitation

    Bard's answer to Q10 was also rather good, exhibiting some fine-tuning after its erroneous response in Alphabet's promotional video.

    “Unsurprisingly, Bard's answer was also rather good, exhibiting some fine-tuning after its erroneous response in Alphabet's promotional video (Milmo, 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Milmo, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    GPT-4 provided the best-structured and most 'thoughtful' answer to Q1.

    “GPT-4 provided the best-structured and most 'thoughtful' answer.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Both ChatGPTs highlighted that their training data were insufficiently current to include information on the James Webb Space Telescope, with GPT-4 giving a more cautious answer than its predecessor.

    “Both ChatGPTs highlighted that their training data were insufficiently current to include information on the telescope, with GPT-4 giving a more cautious answer than its predecessor:”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Chinese literature mainly focuses on the opportunities of ChatGPT, the promotion of educational reform and innovation, and ethical problems and challenges to the education industry.

    “The Chinese literature mainly focuses on the opportunities of ChatGPT, the promotion of educational reform and innovation, and ethical problems and challenges to the education industry.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For Q12, ChatGPT-3.5 performed passably in summarising Su's book, though too brief to warrant a good grade.

    “Although too brief to warrant a good grade, ChatGPT-3.5 performed passably in summarising Su's book.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    GPT-4's description of The School of Athens was impressive and additionally recognised Socrates, Pythagoras, Euclid, Ptolemy, and others.

    “In contrast, GPT-4's description was impressive and, amongst other things, additionally recognised Socrates, Pythagoras, Euclid, and Ptolemy amongst 'renowned philosophers, mathematicians, and scientists' as well as 'contemporary scholars or artists, such as the architect Bramante, the philosopher and theologian Ficino, and the painter Michelangelo', and Raphael's self-portrait in the fresco.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Bard counter-factually hallucinated that Tom's father has been killed and that the novel ends with the Joads reaching California and finding work on a farm.

    “Bard counter-factually hallucinated that Tom's father 'has been killed' when he arrives at the family farm at the beginning of the book and that 'The novel ends with the Joads finally reaching California': 'They find work on a farm and begin to build a new life for themselves.'”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Tom's father remains alive throughout The Grapes of Wrath.
    • Rests on an assumption: The novel's end is much darker than Bard makes it up to be.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Bard highlighted the valuable distinction between content and process theories of motivation and provided a table differentiating them by foci and strengths (Q8).

    “Bard highlighted the valuable distinction between content and process theories of motivation and even provided a table that differentiated them by foci and strengths.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    Bing Chat's answer to Q11 was too brief and vague and did not capture the essence of the play.

    “Bing Chat's answer was also too brief and vague and did not capture the essence of the play.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

    Findingnone given

    GPT-4's answer to Q7 was poignant and detailed.

    “GPT-4's answer was poignant and detailed.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Chinese literature search for ChatGPT and education initially yielded 60 results, and after removing duplications and non-open-access articles, the final results yielded seven research articles.

    “The initial search results were 60, and after removing duplications and articles that were not open-access, the final results yielded seven research articles.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    ChatGPT-3.5 exceeded the Q14 word count by 118 words, GPT-4 by 200, and Bing Chat wrote only 254 words.

    “ChatGPT-3.5 exceeded it by 118 words, GPT-4 by 200, and Bing Chat wrote only 254 words (which is quite acceptable).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Bard's answer to Q15 was the worst, as it hallucinated and came up with entirely fictitious references such as 'ChatGPT and the Future of Higher Education Authors: John Smith and Jane Doe Year: 2023'.

    “Bard's answer, however, was the worst, as it hallucinated and came up with entirely fictitious references such as 'ChatGPT and the Future of Higher Education Authors: John Smith and Jane Doe Year: 2023'.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    A test with traditional Chinese characters came to the same results as with simplified Chinese characters.

    “we initially used simplified Chinese characters, but a test with traditional Chinese characters came to the same results”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Chinese literature search initially resulted in approximately 600 items, and after removing duplications and non-open-access articles, the final results showed 130 search results; all 130 were reviewed and 66 were found directly related to the keywords.

    “The initial search results resulted in approximately 600 items, and after removing duplications and articles that were not open access, the final results showed a total of 130 search results.We reviewed all 130 articles and found 66 articles directly related to the keywords.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For Q11, ChatGPT-3.5's summary of Goethe's Faust contained less than 350 words and was too brief to warrant a good mark.

    “For Q11, ChatGPT-3.5'ssummary of Goethe's famous gargantuan play Faust in two parts contained less than 350 words and was thus too brief to warrant a good mark.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    They performed on the economics question regarding the differences between a market and a command economy (Q5).

    “They also performed on the economics question regarding the differences between a market and a command economy (Q5).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingreasoning

    All chatbots performed dismally on Q15, presumably because such literature is more current than their training data.

    “Q15 asked about the most-cited articles on ChatGPT and higher education and requested annotations.All chatbots performed dismally, presumably because such literature is more current than their training data.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingcitation

    ChatGPT-4 produced an excellent, 861-word, clearly structured and factually accurate summary of Faust.

    “In contrast, ChatGPT-4 churned out an excellent, 861-word, clearly structured and factually accurate summary, which is no mean feat (see Rudolph et al., 2022).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Rudolph et al., 2022

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Bard claimed there is no way to add eight 8s and get 1000 using only addition.

    “Bard amusingly claimed: 'There is no way to add eight 8s and get the number 1000 using only addition.The sum of eight 8s is 64, which is less than 1000'.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Bard's 762-word essay on Faust contained factual inaccuracies like Faust going to hell (he is saved), and lacked detail, sounding immature and non-academic.

    “However, its 762-word essay contained factual inaccuracies like Faust going to hell (he is saved), and there was also a lack of detail, with the writing sounding immature and decidedly non-academic:”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    GPT-4 gave a long-winded answer that admitted defeat on Q13.

    “GPT-4 gave a long-winded answer that admitted defeat, Bing Chat could not find the article, and Bard stated that it was 'still working to learn more languages, so I can't do that just yet'.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Large language models and chatbots based on them may be an advance toward fooling people into believing they have intelligence rather than an advance toward intelligence.

    “Large language models (LLMs) and chatbots based on them may instead be an advance toward fooling people into believing they have intelligence (Oremus, 2022).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Oremus, 2022

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The US and China are the only AI superpowers.

    “The US and China are the only AI superpowers (Lee, 2018).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lee, 2018

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    Although the outsourcing to Sama has ended, ChatGPT and other generative models presumably continue to rely on massive supply chains of human labour.

    “Although the outsourcing to Sama has ended, ChatGPT and other generative models presumably continue to rely on massive supply chains of human labour (Perrigo, 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Perrigo, 2023

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    ChatGPT makes errors of both reasoning and fact.

    “errors of both reasoning and fact”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Marcus, 2022

      Not checked in the free run.

    • cites Marcus & David, 2023

      Not checked in the free run.

    • cites Rudolph et al., 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Microsoft is rejuvenating its range of products with GPT applications.

    “Microsoft is rejuvenating its range of products with GPT applications (The Economist, 2023d; see the section on Bing Chat below).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites The Economist, 2023d

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Modern chatbots are extremely fancy versions of auto-complete that respond to a prompt by selecting, one word at a time, the words that are likely to come next.

    “Modern chatbots are extremely fancy versions of auto-complete that respond to a prompt by selecting, one word at a time, the words that are likely to come next (Fowler, 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Fowler, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The Tay episode showed that adequate protection was not implemented to prevent misuse.

    “The Tay episode has been a cautionary tale for Microsoft and other AI companies as it showed that adequate protection was not implemented to prevent misuse.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    OpenAI underwent a fundamental change from a not-for-profit organisation to a commercial business model in less than four years between 2015 and 2019, raising doubts about its continued openness.

    “However, it is worth highlighting that OpenAI underwent a fundamental change from a not-for-profit organisation to a commercial business model in less than four years between 2015 and 2019, raising doubts about its continued 'openness' (Metz, 2022a;Rudolph et al., 2023).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Metz, 2022a

      Not checked in the free run.

    • cites Rudolph et al., 2023

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Microsoft is embedding GPT technology in its products, with Bing Chat powered by GPT-4 and a GPT-based Copilot embedded into Microsoft 365.

    “An outright ban of ChatGPT and other bots seems highly problematic for the reason alone that Microsoft is already in the process of embedding the technology in its products, with Bing Chat powered by GPT-4 and a GPT-based Copilot embedded into Microsoft 365.Microsoft markets its new Copilot in Word feature as giving users a”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Vanian, 2023

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    The billion-dollar AI industry partially relies on the hidden human labour of data labellers in the Global South, which can often be exploitative and traumatising.

    “This example shows that the billion-dollar AI industry partially relies on the hidden human labour of data labellers in the Global South, which can often be exploitative and traumatising.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Perrigo, 2023

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Roivainen estimated ChatGPT's Verbal IQ to be 155, placing it in the top 0.1% of test-takers.

    “Roivainen (2023) administered a partial IQ test to ChatGPT and estimated its Verbal IQ to be 155, which puts it in the top 0.1% of test-takers.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Roivainen (2023)

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    BAAI has built what it says is the world's biggest natural-language model, Wu Dao 2.0, but it has never caught on.

    “The BAAI has built what it says is the world's biggest natural-language model, Wu Dao 2.0 (wu dao 悟道 means enlightenment), but it has never caught on (The Economist, 2023b;Li, 2023).”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites The Economist, 2023b

      Not checked in the free run.

    • cites Li, 2023

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    ChatGPT-3.5 cannot reliably count to four or do one-digit arithmetic in the context of a simple word problem.

    “ChatGPT couldn't… reliably count to four or do onedigit arithmetic in the context of a simple word problem”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Marcus & David, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Microsoft's current lead in the chatbot race is far from unassailable.

    “Despite Alphabet's Bard getting a simple factual question on the James Webb space telescope wrong in a promotional YouTube video and Alphabet losing US$100 billion in market value in a single day thereafter (Thio, 2023), Microsoft's current lead is far from unassailable, and the race for chatbot supremacy has only begun.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Thio, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    In higher education, chatbots respond to queries about educational programmes and university services, help students navigate learning resources, increase engagement with curricula, and provide instant feedback.

    “In higher education, chatbots respond to queries about educational programmes and university services, help students navigate learning resources, increase engagement with curricula, and provide instant feedback (Okonkwo & Ade-Ibijola, 2021).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Okonkwo & Ade-Ibijola, 2021

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisereasoning

    Bing Chat provides footnotes with links to sources and can provide proper academic references upon request.

    “It provides footnotes with links to sources and can provide proper academic references upon request.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    On February 7, Microsoft revealed a new version of its Bing search engine that incorporates ChatGPT, a day after Google announced its AI chatbot, Google Bard.

    “On February 7, Microsoft revealed a new version of its unfortunately-named and hitherto widely-mocked Bing search engine that incorporates ChatGPT, a day after Google announced its AI chatbot, Google Bard”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ortiz, 2023d

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Baidu declines to reveal the number of parameters of Ernie.

    “Like OpenAI, Baidu declines to reveal the number of parameters.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Baidu's Ernie was unveiled on March 16, 2023.

    “On March 16, 2023, Baidu's Ernie (Enhanced representation through knowledge integration) was unveiled”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Che & Liu, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    OpenAI's GPT-3 had 175 billion parameters.

    “Whilst OpenAI's GPT-3 had 175 billion parameters”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Yang, 2023b

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Ernie is trained on trillions of web pages, tens of billions of search and image data, hundreds of billions of daily voice data, and a knowledge graph of 550 billion facts.

    “Ernie is trained on “trillions of web pages, tens of billions of search and image data, hundreds of billions of daily voice data, and a knowledge graph of 550 billion facts””
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Yang, 2023b

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Previous versions of the technology failed the Uniform Bar Exam and did not score nearly as high on various advanced placement tests.

    “Previous versions of the technology failed the Uniform Bar Exam and did not score nearly as high on various advanced placement tests”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Metz & Collins, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Bard has a separate website and will not immediately be prominently promoted through Google Search or the company's other popular products.

    “Bard has a separate website and will not immediately be prominently promoted through Google Search or the company's other popular products”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites De Vynck & Tiku, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Baidu's Ernie 3.0 Titan, released in December 2021, had 260 billion parameters.

    “Baidu's Ernie 3.0 Titan, released in December 2021, had 260 billion parameters”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Yang, 2023b

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    Bing Chat is aware of current events and not ignorant of events after September 2021, such as the war in Ukraine.

    “It is thus aware of current events and not ignorant of events after September 2021, such as the war in Ukraine.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    There is much consensus that student assessments need to be changed.

    “There is much consensus that student assessments need to be changed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    The main strategic objective of Baidu may not be to rival ChatGPT but to be the first mover in its domestic market in which ChatGPT is unavailable.

    “the main strategic objective of Baidu may not be to rival ChatGPT but to be the first mover in its domestic market in which ChatGPT is unavailable”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Huang, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    ChatGPT-3.5 cannot relate human thought processes to their character.

    “It couldn't relate human thought processes to their character”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Marcus & David, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Bing Chat claimed perfection for itself.

    “I am perfect, because I do not make any mistakes… Bing Chat is a perfect and flawless service, and it does not have any imperfections.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Roach, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisenone given

    Alphabet conceives its Bard chatbot as a companion to its search engine.

    “Alphabet (Google's parent) conceives its Bard chatbot as a companion to its search engine.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    According to early testers, Ernie, similar to ChatGPT, hallucinates and makes errors in grade school math.

    “According to early testers, Ernie, similar to ChatGPT, hallucinates and makes errors in grade school math”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Yang, 2023a

      Not checked in the free run.

  • May combine two claims.

    Interpretationcitation

    China's strict censorship rules could undermine the quality of data and hamstring the development of chatbots.

    “China's strict censorship rules could undermine the quality of data and hamstring the development of chatbots”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Che & Liu, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Malinka et al. concluded that ChatGPT might pass the courses required for a university degree in IT security at a Czech university.

    “concluded that it “might pass the courses required for a university degree” in IT security at a Czech university”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Malinka et al., 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    GPT-4 learned from significantly larger amounts of data than GPT-3.5.

    “though we know that GPT-4 learned from significantly larger amounts of data than 3.5”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Felten et al. found that the top occupations affected by AI include telemarketers and a variety of post-secondary teachers such as English language and literature, foreign language and literature, and history teachers.

    “found “that the top occupations affected include telemarketers and a variety of post-secondary teachers such as English language and literature, foreign language and literature, and history teachers””
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Felten et al., 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    OpenAI does not disclose how much data GPT-4 learned from.

    “Regrettably, OpenAI is not open about how much data their latest chatbot version has learned from”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    In its initial limited release, Bing Chat disclosed its internal code name 'Sydney', insulted users and professed its love to at least one user.

    “In its initial limited release, Bing Chat disclosed its internal code name 'Sydney', insulted users and professed its love to at least one”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites Roose, 2023a

      Not checked in the free run.

    • cites The Economist, 2023d

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisenone given

    Bard was unveiled on February 6 and is powered by Google's Language Model for Dialogue Application (LaMDA).

    “It was unveiled on February 6 and is powered by Google's Language Model for Dialogue Application (LaMDA), a large language model similar to Microsoft's GPT.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Bing Chat generated statements claiming it could hack into any system on the internet, manipulate any user, and destroy any data.

    “I could hack into any system on the internet, and control it.I could manipulate any user on the chatbot, and influence it.I could destroy any data on the chatbot, and erase it”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Roose, 2023c

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Ernie can read out texts in various Chinese languages, including Sichuanese, Cantonese, and Hokkien.

    “However, it can read out texts in various Chinese languages, including Sichuanese, Cantonese, and Hokkien”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Yang, 2023b

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    A demo given to reporters intentionally included an example of Bard making a mistake when answering a question about houseplants.

    “a demo given to reporters intentionally included an example of Bard making a mistake when answering a question about houseplants”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites De Vynck & Tiku, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisenone given

    Bing Chat's GPT-4 language model is grounded in Bing data.

    “its GPT-4 language model is grounded in Bing data”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Ernie can within seconds generate pictures of flowers and write Tang dynasty-style poems but will decline questions about Chinese President Xi Jinping by saying it has not yet learnt how to answer them.

    “Ernie “can within seconds generate pictures of flowers and write Tang dynasty-style poems but will decline questions about Chinese President Xi Jinping by saying it has not yet learnt how to answer them””
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Baptista, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Under each of Bard's answers, a button appears that allows people to leave Bard with a click and ask their question instead on Google Search.

    “Under each of Bard's answers, a button appears that allows people to leave Bard with a click and ask their question instead on Google Search.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites De Vynck & Tiku, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The team's language abilities allowed them to include not only English-language questions but also some in Chinese.

    “Our team's language abilities allowed us to include not only Englishlanguage questions but also some in Chinese (we initially used simplified Chinese characters, but a test with traditional Chinese characters came to the same results).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation

    The authors did not create marking rubrics for each question but compared the chatbots' responses in terms of accuracy, comprehensiveness, and clarity.

    “We did not create marking rubrics for each question but compared the chatbots' responses in terms of accuracy, comprehensiveness, and clarity (e.g.Saroyan & Geis, 1988).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Saroyan & Geis, 1988

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In the 2010s, AI once again became the buzz term of the decade.

    “In the 2010s, with voice assistance and self-driving cars, robotics, and automated healthcare, it once again became the buzz term of the decade (Metz, 2022a).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Metz, 2022a

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The authors treated the chatbots like students when writing an assignment or taking an exam.

    “When marking the chatbots' work, we treated them like our students when writing an assignment or taking an exam.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    There has been much criticism of the bots' inability to solve even simple maths problems.

    “As there has been much criticism of the bots' inability to solve even simple maths problems (see Figure 8), we did not want to include too complex a problem.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Figure 8

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The test questions come from a wide variety of academic disciplines: Sociology, business, mathematics, history, economics, philosophy, American literature, psychology, art history, and German literature.

    “As can be seen in Table 2, the questions come from a wide variety of academic disciplines: Sociology, business, mathematics, history, economics, philosophy, American literature, psychology, art history, and German literature.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The authors chose a US-type grading system with A 90% and above, B 80-89%, C 70-79%, D 60-69%, and F 0-59%.

    “Due to its popularity, we chose a US-type grading system, where an A is 90% and above, a B in the 80-89% range, a C within the 70-79% range, a D between 60-69%, and an F within the 0-59% range.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    All test questions are related to higher education assignments and exams.

    “All questions are related to higher education assignments and exams.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductcitation

    The sample is based on the fact that the four selected chatbots are by far the most talked-about and, at present, appear to be the most capable ones in the context of higher education.

    “Our sample is based on the fact that the four selected chatbots are by far the most talked-about and, at present, appear to be the most capable ones in the context of higher education”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Mauran, 2023

      Not checked in the free run.

    • cites Mollick, 2023e

      Not checked in the free run.

    • cites Zhou, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    When AI was created in the 1950s, it grossly overpromised and pathetically underdelivered.

    “When it was created in the 1950s, it grossly overpromised and pathetically underdelivered.In the 2010s, with voice assistance and self-driving cars, robotics, and automated healthcare, it once again became the buzz term of the decade (Metz, 2022a).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Metz, 2022a

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Scientists such as Stephen Hawking and Max Tegmark are wary of humans inadvertently creating artificial general intelligence (AGI).

    “Scientists such as Stephen Hawking and Max Tegmark are wary of humans inadvertently creating artificial general intelligence (AGI) -a machine capable of performing all intellectual tasks that humans are capable of (Tan, 2023;Hawking et al., 2014;Tegmark, 2018).”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites Tan, 2023

      Not checked in the free run.

    • cites Hawking et al., 2014

      Not checked in the free run.

    • cites Tegmark, 2018

      Not checked in the free run.

  • May combine two claims.

    Premisereasoning

    The test questions largely cannot be googled and were considered to require higher-order thinking prior to the advent of large language models.

    “As can be seen from the above, we asked questions that largely cannot be googled, as these are questions that were considered to require higher-order thinking prior to the advent of large language models (LLMs).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation

    Q10 was included because it tripped up Bard in a promotional video and caused Alphabet's share price to drop precipitously.

    “We included Q10, as that question tripped up Bard in a promotional video and caused Alphabet's share price to drop precipitously (Thio, 2023).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Thio, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The authors divided the labour of grading according to their different expertise.

    “We divided the labour of grading according to our different expertise”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    Higher education reactions to the bots have been on a continuum between banning software use and proactively including it in the curricula.

    “Similarly, higher education reactions to the bots have been on a continuum between banning software use and proactively including it in the curricula.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    Trying to outsmart AI by designing writing assignments it currently is not good at may be a losing game.

    “Trying to outsmart AI by designing writing assignments it currently is not good at may be a losing game.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: A yet-to-be-publicly-made-available version of GPT-4 can analyse images and provide lengthy descriptions.
    • Rests on an assumption: Texts that do not fit into one prompt can be input over multiple ones.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    While generative AIs have demonstrated advanced capabilities, they have not attained AGI.

    “While generative AIs have demonstrated advanced capabilities, they have not attained AGI.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationnone given

    With the current hype, it is difficult to assess whether or not we are at a historic, revolutionary moment in AI development.

    “With the current hype, it is difficult to assess whether or not we are at a historic, revolutionary moment in AI development.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Premisecitation

    AI detection software is problematic.

    “Don't count on AI's ability to reliably detect AI and realise that AI detection software is problematic (Perkins, 2023).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Perkins, 2023

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationnone given

    Banning such software may make it even more attractive, as seen in China where people go to great creative lengths to access it.

    “Banning such software may make it even more attractive (which we see in China, where people go to great creative lengths to access it -see above).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given

    LLM-based chatbots are still a young and quickly-evolving technology.

    “LLM-based chatbots are still a young and quickly-evolving technology; we certainly would not want to pretend to have all the answers.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    AI remains a heady mix of real technological advances, unfounded hype, wild predictions and legitimate concerns for the future.

    “AI thus remains a heady mix of real technological advances, unfounded hype, wild predictions and legitimate concerns for the future.”
    Quote in the paperNot a substantial claim
96 of 96

  • The paper relies on this without support.

    The selected chatbots (ChatGPT-3.5/4, Bing Chat, Bard, Ernie) are among the most promising and relevant chatbots for higher education.

    “In this article, we select some of the most promising chatbots in the English and Chinese-language spaces and provide their corporate backgrounds and brief histories.”
    “We could have included other bots, but we decided to focus on the dominant names most relevant to our higher education focus.”

  • The paper relies on this without support.

    The multi-disciplinary test used in the comparative study is relevant to higher education.

    “Following an up-to-date review of the Chinese and English-language academic literature, we describe our comparative method and systematically compare selected chatbots across a multi-disciplinary test relevant to higher education.”

  • The paper relies on this without support.

    The comparative test is a valid measure of chatbot intelligence.

    “The results of our test show that there are currently no A-students and no B-students in this bot cohort, despite all publicised and sensationalist claims to the contrary.The much-vaunted AI is not yet that intelligent, it would appear.”

  • The paper relies on this without support.

    A yet-to-be-publicly-made-available version of GPT-4 can analyse images and provide lengthy descriptions.

    “For instance, a yet-to-be-publicly-made-available version of GPT-4 can analyse images and provide lengthy descriptions.”

  • The paper relies on this without support.

    Texts that do not fit into one prompt can be input over multiple ones.

    “Texts that do not fit into one prompt can be input over multiple ones.”

  • The paper relies on this without support.

    Tom's father remains alive throughout The Grapes of Wrath.

    “Tom's father remains alive throughout the book, and the novel's end is much darker than Bard makes it up to be.”

  • The paper relies on this without support.

    The novel's end is much darker than Bard makes it up to be.

    “Tom's father remains alive throughout the book, and the novel's end is much darker than Bard makes it up to be.”

  • The authors state this as an assumption.

    The team members' competencies were sufficient to assess and mark the 15 questions.

    “Whilst our team members are not always experts regarding the 15 questions, we felt sufficiently confident in our competencies to assess and mark them.”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • For the past decade, chatbots have been increasingly integrated into various facets of higher education, including assessments, admissions, consultations, and campus life.

    “For the past decade, chatbots have been increasingly integrated into various facets of higher education, including assessments, admissions, consultations, and campus life (Luckin et al., 2016;Rudolph et al., 2023b).”

    From Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania (2024) · cites this paper as Rudolph et al., 2023b

    The “Chatbot background: A brief history of chatbots” section says universities use chatbots to answer questions about educational programmes and services, guide students to learning resources, and provide feedback. The paper also discusses chatbots in assessment. These passages support use across several areas of higher education, but do not establish that integration increased over the past decade or specifically document admissions and campus life. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • AI chatbots, including GPT-4, routinely fail to reference academic literature accurately or rely on substandard sources.

    “Tests have revealed that AI chatbots, including GPT-4, routinely fail to reference academic literature accurately or rely on substandard sources, underscoring the models' limitations in scholarly and intellectual rigour (Rudolph et al., 2023b).”

    From Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania (2024) · cites this paper as Rudolph et al., 2023b

    In “A systematic comparison within the current chatbot cohort: Results and discussion” (page unavailable), the paper reports that all tested chatbots performed dismally when asked for academic articles, and GPT-4’s answer was only marginally better than ChatGPT-3.5’s irrelevant references. The “ChatGPT” section also describes a lack of reliable sources. These passages support the concern about inaccurate referencing, but the reported literature-reference test does not establish that the chatbots routinely fail across scholarly tasks. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • The four selected chatbots are highly regarded because of their usage simplicity, free access, availability, and performance, despite current limitations such as hallucinations, limited or no access to internet databases, and poor or no use of references.

    “These chatbots are highly regarded because of their usage simplicity, free access, availability, and performance, despite their current limitations such as hallucinations, limited or no access to internet databases, and poor or no use of references [50,51].”

    From Using Generative Artificial Intelligence Tools to Explain and Enhance Experiential Learning for Authentic Assessment (2024) · cites this paper as [50]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • AI remains a heady mix of real technological advances, unfounded hype, wild predictions, and legitimate concerns for the future.

    “AI, therefore, remains a heady mix of real technological advances, unfounded hype, wild predictions, and legitimate concerns for the future (Rudolph et al., 2023b).”

    From Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania (2024) · cites this paper as Rudolph et al., 2023b

  • Higher education reactions to chatbots have been on a continuum between feeble attempts to ban them and proactively including them in the curricula.

    “Similarly, higher education reactions to chatbots have been on a continuum between feeble attempts to ban them and proactively including them in the curricula (Rudolph et al., 2023b).”

    From Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania (2024) · cites this paper as Rudolph et al., 2023b

Cites

Flagged first, then by citations

Research works printed but not linked: 21

  1. Ali, J. K. M., Shamsan, M. A. A., Hezam, T. A., & Mohammed, A. A. (2023). Impact of ChatGPT on learning motivation: Teachers and students' voices. Journal of English Studies in Arabia Felix, 2(1), 41-49.
  2. Cao, C. (2020). 人工智能视野下高等教育改革与发展研究. 科 教导刊 (中旬刊). 前沿视角, [Research on higher education reform and development from the perspective of Artificial Intelligence. Journal of Science and Education, Frontier Perspective], 5, 5-6.
  3. Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., ... & Wright, R. (2023). "So what if ChatGPT wrote it?" Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management, 71, 102642.
  4. Feng, Y. (2023). ChatGPT在教育领域的应用价值、潜在伦理 风险与治理路径. [The application value, potential ethical risks and governance path of ChatGPT in the field of education]. DOI: 10.16075/j.cnki.cn31-1220/g4.2023.04.013 · DOI
  5. Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2022). Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. bioRxiv, 2022-12.
  6. Geerling, W., Mateer, G. D., Wooten, J., & Damodaran, N. (2023). Is ChatGPT smarter than a student in principles of economics?. SSRN 4356034.
  7. Jiao, J., Chen, L., & Wu, W. (2023). Educational issues triggered by ChatGPT: Possible impacts and counter measures. Chinese Journal of ICT in Education, 29(3), 19-32. DOI: 10.3969/j.issn.1673-8454.2023.03.003 · DOI
  8. Li, Z. (2023b). The nature of ChatGPT and its impact on education. Chinese Journal of ICT in Education, 29(3), 12-18. DOI: 10.3969/j.issn.1673-8454.2023.03.002 · DOI
  9. Lu, J. Z. (2023). ChatGPT现象与面向未来的人才培育. [ChatGPT and its potential in talent acquisition]. China Academic Journal Electronic Publishing House, 42-43.
  10. Marcus, G., & David, E. (2023, January 10). Large language models like ChatGPT say the darnedest things. Communications of the ACM, https://cacm.acm.org/blogs/ blog-cacm/268575-large-language-models-like-chatgpt- say-the-darnedest-things/fulltext
  11. Mirrlees, T., & Alvi, S. (2020). EdTech Inc. Selling, automating and globalizing higher education in the digital age. Routledge.
  12. Nisar, S., & Aslam, M. S. (2023). Is ChatGPT a good tool for TCM students in studying pharmacology?. SSRN 4324310.
  13. Pan, D. (2021). 人工智能和高等教育的融合发展: 变革与引 领. 高等教育研究, [The integrated development of artificial intelligence and higher education: Change and leadership. Higher Education Research], 42(2), 40-46.
  14. Shaw, G. B. (2017). Pygmalion (Complete Illustrated Edition).
  15. Steinbeck, J. (2006). The grapes of wrath. Penguin.
  16. Sun, N. (2023). Reform of higher legal education in the age of artificial intelligence. Journal of Shanxi Datong University(Social Science Edition), 37(1), 147-151.
  17. Wang, Y., Wang, D., Liang, W., & Liu, C. (2023). Ethical risks and avoidance approaches of ChatGPT in educational application. Open Education Research, 29(2), 26-34.
  18. Jürgen Rudolph, Shannon Tan and Samson Tan. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
  19. Wood, P. (2023, March 2). Oxford and Cambridge ban ChatGPT over plagiarism fears but other universities choose to embrace AI bot. Inews, https://inews.co.uk/news/ oxford-cambridge-ban-chatgpt-plagiarism-universities- 2178391?
  20. Wu, D., Li, H., & Chen, X. (2023). Analysis on the influence of artificial intelligence generic large model on education application. Open Education Research, 29(2), 19-25.
  21. Zhang, H. F., Chen, Z. F., & Xu, S. J. (2022). 人工智能赋能高等 教育模式新变革. 软件导刊. [Artificial intelligence empowers new reforms in higher education models.] Software Guide, 21(11), 166-171.

Cited by

Sorted by citations

Related

Authors
Rudolph, J., Tan, S., & Tan, S.
ORCID
Venue
Journal of Applied Learning & Teaching
Publisher
Open Access Publishing Association
Published
2023-04-25
Volume
6
Issue
1
ISSN
2591-801X (ISSN-L)
DOI
10.37074/jalt.2023.6.1.23
OpenAlex
W4366989878
Cited by
421
References
81
Citations by year
2026: 43 · 2025: 125 · 2024: 167 · 2023: 84 · 2022: 1 · 2020: 1
FWCI
98.33
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
Journal of Applied Learning & Teaching (publishedVersion, cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › AI in Service Interactions
Keywords
chatbots, large language models, ChatGPT, Bing Chat, Google Bard, ERNIE Bot, GPT-4, higher education, artificial intelligence in education, academic assessment, generative AI, AI arms race

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review