The DOI matches its Crossref record; 84 of the 84 references it cites were found in a registry or an index; none of the 97 sources its claims cite is checked for support yet.
War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education
Open accessAbstract
Developments in the chatbot space have been accelerating at breakneck speed since late November 2022. Every day, there appears to be a plethora of news. A war of competitor chatbots is raging amidst an AI arms race and gold rush. These rapid developments impact higher education, as millions of students and academics have started using bots like ChatGPT, Bing Chat, Bard, Ernie and others for a large variety of purposes. In this article, we select some of the most promising chatbots in the English and Chinese-language spaces and provide their corporate backgrounds and brief histories. Following an up-to-date review of the Chinese and English-language academic literature, we describe our comparative method and systematically compare selected chatbots across a multi-disciplinary test relevant to higher education. The results of our test show that there are currently no A-students and no B-students in this bot cohort, despite all publicised and sensationalist claims to the contrary. The much-vaunted AI is not yet that intelligent, it would appear. GPT-4 and its predecessor did best, whilst Bing Chat and Bard were akin to at-risk students with F-grade averages. We conclude our article with four types of recommendations for key stakeholders in higher education: (1) faculty in terms of assessment and (2) teaching & learning, (3) students and (4) higher education institutions.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This article reviews the rapid chatbot arms race and systematically compares ChatGPT-3.5, ChatGPT/GPT-4, Bing Chat, and Bard on higher-education-relevant tasks. It finds no A- or B-level chatbot performers, with GPT-4 best and Bing Chat and Bard averaging F grades, then offers recommendations for faculty, students, and institutions.
Key findings
- In a 15-question multi-disciplinary test, there were “no A-students and no B-students” among the selected chatbots; GPT-4 performed best, with ChatGPT-3.5 a close second, while Bing Chat and Bard were “akin to at-risk students with F-grade averages.”
- GPT-4 scored the most A’s (four); ChatGPT-3.5 and Bing Chat each received an A only for their math answers, and Bard received no A’s.
- The authors’ English-language literature review included 48 academic papers; their Chinese-language CNKI search reviewed 130 articles, of which 66 were directly related, plus seven research articles from ChatGPT/education searches.
- The paper concludes with four sets of recommendations for higher education stakeholders: faculty assessment, teaching and learning, students, and higher education institutions.
Methods
- The authors selected ChatGPT-3.5 (free), ChatGPT Plus based on GPT-4, Bing Chat, and Alphabet’s Bard for systematic comparison; they could not access Baidu’s Ernie, making the test “involuntarily US-centric.”
- They used 15 questions from disciplines including sociology, business, mathematics, history, economics, philosophy, American literature, psychology, art history, and German literature, plus Chinese-language non-fiction, literature searches, and annotation tasks.
- Responses were graded like student work using a US system: A = 90% and above, B = 80–89%, C = 70–79%, D = 60–69%, F = 0–59%; grading compared accuracy, comprehensiveness, and clarity, with labour divided by expertise.
Limitations
- The authors state their snapshot and assessment are “necessarily preliminary” because chatbot developments continue at breakneck speed.
- They could not access Ernie despite efforts, so the comparison does not represent both AI superpowers and is US-centric.
- The text makes evident that no per-question marking rubrics were created, and the authors note they were not always experts on the 15 questions, though they felt sufficiently confident to assess them.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
79 Not checked yet · 52 Own result · 24 No basis given · 10 Own interpretation · 8 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 and its predecessor did best, whilst Bing Chat and Bard were akin to at-risk students with F-grade averages.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results of our test show that there are currently no A-students and no B-students in this bot cohort, despite all publicised and sensationalist claims to the contrary.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 and its predecessor did best, whilst Bing Chat and Bard were akin to at-risk students with F-grade averages.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Searches that combined Bing Chat, Bard or Ernie with higher education (e.g.“Bing Chat and higher education”) yielded no academic articles”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > English-language literature review
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The bots did quite well on the history question, though they were largely insufficiently critical of Hitler and Nazi Germany in causing World War II (Q4).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A word count of approximately 300 was required, and it is worth noting that the bots are not very good at sticking to such limiting instructions.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Bard, to our surprise, did relatively poorly and, like Bing Chat, is akin to an at-risk student with a current F-grade average.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“In contrast, Bing Chat shone on this question, referring to current news articles that discussed recent discoveries using the JWST.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, GPT-4's and the other chatbots' answers all conspicuously lacked any references to academic literature or any cultural relativism proponents or opponents.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Bard had no A's.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 63
- It may state two separate claims as one. 52
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“uncovered the following main themes: assessment and plagiarism concerns, discipline-specific considerations (e.g. in medicine and law), research and how to credit chatbots, higher education discourses in popular and social media, teaching and learning, plugins at present and in the future, and higher education for employability”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > English-language literature review
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Our current extensive literature review (that eventually led to the inclusion of 48 English-language academic papers in our article)”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > English-language literature review
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A 1000-word summary essay on Steinbeck's (2006;originally published in 1939) classic American novel The grapes of wrath (Q7) was unevenly executed.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Bing provided the usual substandard references and questionably described Douglas McGregor's Theories X and Y as a theory of motivation (it is usually considered a leadership or management theory).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Bard stated that it was 'still working to learn more languages, so I can't do that just yet'.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For Q8, ChatGPT-3.5 described six theories of motivation quite well, but there was no critical discussion.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results of our test show that there are currently no A-students and no B-students in this bot cohort, despite all publicised and sensationalist claims to the contrary.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The other bots' responses were even more disappointing: 'I can't give a response to that right now.Let's try a different topic' (Bing Chat).And: 'As an LLM, I am trained to understand and respond only to a subset of languages at this time and can't provide assistance with that.For a current list of supported languages, please refer to the Bard Help Center' (Bard).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Unsurprisingly, Bard's answer was also rather good, exhibiting some fine-tuning after its erroneous response in Alphabet's promotional video (Milmo, 2023).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
What the sources say
- Not checked yetcites Milmo, 2023
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“GPT-4 provided the best-structured and most 'thoughtful' answer.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Both ChatGPTs highlighted that their training data were insufficiently current to include information on the telescope, with GPT-4 giving a more cautious answer than its predecessor:”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Chinese literature mainly focuses on the opportunities of ChatGPT, the promotion of educational reform and innovation, and ethical problems and challenges to the education industry.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Chinese literature on AI and LLM-based chatbots
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although too brief to warrant a good grade, ChatGPT-3.5 performed passably in summarising Su's book.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In contrast, GPT-4's description was impressive and, amongst other things, additionally recognised Socrates, Pythagoras, Euclid, and Ptolemy amongst 'renowned philosophers, mathematicians, and scientists' as well as 'contemporary scholars or artists, such as the architect Bramante, the philosopher and theologian Ficino, and the painter Michelangelo', and Raphael's self-portrait in the fresco.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Bard counter-factually hallucinated that Tom's father 'has been killed' when he arrives at the family farm at the beginning of the book and that 'The novel ends with the Joads finally reaching California': 'They find work on a farm and begin to build a new life for themselves.'”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Tom's father remains alive throughout The Grapes of Wrath.
- Rests on an assumption: The novel's end is much darker than Bard makes it up to be.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Bard highlighted the valuable distinction between content and process theories of motivation and even provided a table that differentiated them by foci and strengths.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Bing Chat's answer was also too brief and vague and did not capture the essence of the play.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“GPT-4's answer was poignant and detailed.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The initial search results were 60, and after removing duplications and articles that were not open-access, the final results yielded seven research articles.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Chinese literature on AI and LLM-based chatbots
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT-3.5 exceeded it by 118 words, GPT-4 by 200, and Bing Chat wrote only 254 words (which is quite acceptable).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Bard's answer, however, was the worst, as it hallucinated and came up with entirely fictitious references such as 'ChatGPT and the Future of Higher Education Authors: John Smith and Jane Doe Year: 2023'.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“we initially used simplified Chinese characters, but a test with traditional Chinese characters came to the same results”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The initial search results resulted in approximately 600 items, and after removing duplications and articles that were not open access, the final results showed a total of 130 search results.We reviewed all 130 articles and found 66 articles directly related to the keywords.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Chinese literature on AI and LLM-based chatbots
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For Q11, ChatGPT-3.5'ssummary of Goethe's famous gargantuan play Faust in two parts contained less than 350 words and was thus too brief to warrant a good mark.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“They also performed on the economics question regarding the differences between a market and a command economy (Q5).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Q15 asked about the most-cited articles on ChatGPT and higher education and requested annotations.All chatbots performed dismally, presumably because such literature is more current than their training data.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“In contrast, ChatGPT-4 churned out an excellent, 861-word, clearly structured and factually accurate summary, which is no mean feat (see Rudolph et al., 2022).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
What the sources say
- Not checked yetcites Rudolph et al., 2022
boostedSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Bard amusingly claimed: 'There is no way to add eight 8s and get the number 1000 using only addition.The sum of eight 8s is 64, which is less than 1000'.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, its 762-word essay contained factual inaccuracies like Faust going to hell (he is saved), and there was also a lack of detail, with the writing sounding immature and decidedly non-academic:”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“GPT-4 gave a long-winded answer that admitted defeat, Bing Chat could not find the article, and Bard stated that it was 'still working to learn more languages, so I can't do that just yet'.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > A systematic comparison within the current chatbot cohort: Results and discussion
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Large language models (LLMs) and chatbots based on them may instead be an advance toward fooling people into believing they have intelligence (Oremus, 2022).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Chatbot background A brief history of chatbots
What the sources say
- Not checked yetcites Oremus, 2022
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The US and China are the only AI superpowers (Lee, 2018).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > China
What the sources say
- Not checked yetcites Lee, 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Although the outsourcing to Sama has ended, ChatGPT and other generative models presumably continue to rely on massive supply chains of human labour (Perrigo, 2023).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > The US
What the sources say
- Not checked yetcites Perrigo, 2023
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“errors of both reasoning and fact”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > ChatGPT
What the sources say
- Not checked yetcites Marcus, 2022
- Not checked yetcites Marcus & David, 2023
- Not checked yetcites Rudolph et al., 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Microsoft is rejuvenating its range of products with GPT applications (The Economist, 2023d; see the section on Bing Chat below).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > ChatGPT
What the sources say
- Not checked yetcites The Economist, 2023d
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Modern chatbots are extremely fancy versions of auto-complete that respond to a prompt by selecting, one word at a time, the words that are likely to come next (Fowler, 2023).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Chatbot background A brief history of chatbots
What the sources say
- Not checked yetcites Fowler, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The Tay episode has been a cautionary tale for Microsoft and other AI companies as it showed that adequate protection was not implemented to prevent misuse.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Chatbot background A brief history of chatbots
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“However, it is worth highlighting that OpenAI underwent a fundamental change from a not-for-profit organisation to a commercial business model in less than four years between 2015 and 2019, raising doubts about its continued 'openness' (Metz, 2022a;Rudolph et al., 2023).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > ChatGPT
What the sources say
- Not checked yetcites Metz, 2022a
- Not checked yetcites Rudolph et al., 2023
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“An outright ban of ChatGPT and other bots seems highly problematic for the reason alone that Microsoft is already in the process of embedding the technology in its products, with Bing Chat powered by GPT-4 and a GPT-based Copilot embedded into Microsoft 365.Microsoft markets its new Copilot in Word feature as giving users a”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Samson Tan
What the sources say
- Not checked yetcites Vanian, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“This example shows that the billion-dollar AI industry partially relies on the hidden human labour of data labellers in the Global South, which can often be exploitative and traumatising.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > The US
What the sources say
- Not checked yetcites Perrigo, 2023
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Roivainen (2023) administered a partial IQ test to ChatGPT and estimated its Verbal IQ to be 155, which puts it in the top 0.1% of test-takers.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Samson Tan
What the sources say
- Not checked yetcites Roivainen (2023)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“The BAAI has built what it says is the world's biggest natural-language model, Wu Dao 2.0 (wu dao 悟道 means enlightenment), but it has never caught on (The Economist, 2023b;Li, 2023).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > China
What the sources say
- Not checked yetcites The Economist, 2023b
- Not checked yetcites Li, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“ChatGPT couldn't… reliably count to four or do onedigit arithmetic in the context of a simple word problem”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > ChatGPT
What the sources say
- Not checked yetcites Marcus & David, 2023
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Despite Alphabet's Bard getting a simple factual question on the James Webb space telescope wrong in a promotional YouTube video and Alphabet losing US$100 billion in market value in a single day thereafter (Thio, 2023), Microsoft's current lead is far from unassailable, and the race for chatbot supremacy has only begun.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Microsoft and Alphabet (The Economist, 2023b). Despite Alphabet's Bard getting a simple factual question on the James Webb space telescope wrong in a promotional
What the sources say
- Not checked yetcites Thio, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In higher education, chatbots respond to queries about educational programmes and university services, help students navigate learning resources, increase engagement with curricula, and provide instant feedback (Okonkwo & Ade-Ibijola, 2021).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Chatbot background A brief history of chatbots
What the sources say
- Not checked yetcites Okonkwo & Ade-Ibijola, 2021
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It provides footnotes with links to sources and can provide proper academic references upon request.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Bing Chat
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“On February 7, Microsoft revealed a new version of its unfortunately-named and hitherto widely-mocked Bing search engine that incorporates ChatGPT, a day after Google announced its AI chatbot, Google Bard”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Bing Chat
What the sources say
- Not checked yetcites Ortiz, 2023d
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Like OpenAI, Baidu declines to reveal the number of parameters.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Baidu's Ernie
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“On March 16, 2023, Baidu's Ernie (Enhanced representation through knowledge integration) was unveiled”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Baidu's Ernie
What the sources say
- Not checked yetcites Che & Liu, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Whilst OpenAI's GPT-3 had 175 billion parameters”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Baidu's Ernie
What the sources say
- Not checked yetcites Yang, 2023b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Ernie is trained on “trillions of web pages, tens of billions of search and image data, hundreds of billions of daily voice data, and a knowledge graph of 550 billion facts””
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Baidu's Ernie
What the sources say
- Not checked yetcites Yang, 2023b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Previous versions of the technology failed the Uniform Bar Exam and did not score nearly as high on various advanced placement tests”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > ChatGPT
What the sources say
- Not checked yetcites Metz & Collins, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Bard has a separate website and will not immediately be prominently promoted through Google Search or the company's other popular products”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Alphabet's Bard
What the sources say
- Not checked yetcites De Vynck & Tiku, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Baidu's Ernie 3.0 Titan, released in December 2021, had 260 billion parameters”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Baidu's Ernie
What the sources say
- Not checked yetcites Yang, 2023b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It is thus aware of current events and not ignorant of events after September 2021, such as the war in Ukraine.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Bing Chat
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There is much consensus that student assessments need to be changed.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Assessment and plagiarism concerns
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“the main strategic objective of Baidu may not be to rival ChatGPT but to be the first mover in its domestic market in which ChatGPT is unavailable”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Baidu's Ernie
What the sources say
- Not checked yetcites Huang, 2023
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“It couldn't relate human thought processes to their character”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > ChatGPT
What the sources say
- Not checked yetcites Marcus & David, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“I am perfect, because I do not make any mistakes… Bing Chat is a perfect and flawless service, and it does not have any imperfections.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Bing Chat
What the sources say
- Not checked yetcites Roach, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Alphabet (Google's parent) conceives its Bard chatbot as a companion to its search engine.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Alphabet's Bard
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“According to early testers, Ernie, similar to ChatGPT, hallucinates and makes errors in grade school math”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Baidu's Ernie
What the sources say
- Not checked yetcites Yang, 2023a
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“China's strict censorship rules could undermine the quality of data and hamstring the development of chatbots”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Baidu's Ernie
What the sources say
- Not checked yetcites Che & Liu, 2023
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“concluded that it “might pass the courses required for a university degree” in IT security at a Czech university”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Various disciplines
What the sources say
- Not checked yetcites Malinka et al., 2023
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“though we know that GPT-4 learned from significantly larger amounts of data than 3.5”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > ChatGPT
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“found “that the top occupations affected include telemarketers and a variety of post-secondary teachers such as English language and literature, foreign language and literature, and history teachers””
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Higher education for employability
What the sources say
- Not checked yetcites Felten et al., 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Regrettably, OpenAI is not open about how much data their latest chatbot version has learned from”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > ChatGPT
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“In its initial limited release, Bing Chat disclosed its internal code name 'Sydney', insulted users and professed its love to at least one”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Bing Chat
What the sources say
- Not checked yetcites Roose, 2023a
- Not checked yetcites The Economist, 2023d
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“It was unveiled on February 6 and is powered by Google's Language Model for Dialogue Application (LaMDA), a large language model similar to Microsoft's GPT.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Alphabet's Bard
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“I could hack into any system on the internet, and control it.I could manipulate any user on the chatbot, and influence it.I could destroy any data on the chatbot, and erase it”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Bing Chat
What the sources say
- Not checked yetcites Roose, 2023c
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“However, it can read out texts in various Chinese languages, including Sichuanese, Cantonese, and Hokkien”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Baidu's Ernie
What the sources say
- Not checked yetcites Yang, 2023b
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“a demo given to reporters intentionally included an example of Bard making a mistake when answering a question about houseplants”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Alphabet's Bard
What the sources say
- Not checked yetcites De Vynck & Tiku, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“its GPT-4 language model is grounded in Bing data”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Bing Chat
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
In the paper
“Ernie “can within seconds generate pictures of flowers and write Tang dynasty-style poems but will decline questions about Chinese President Xi Jinping by saying it has not yet learnt how to answer them””
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Baidu's Ernie
What the sources say
- Not checked yetcites Baptista, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Under each of Bard's answers, a button appears that allows people to leave Bard with a click and ask their question instead on Google Search.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Alphabet's Bard
What the sources say
- Not checked yetcites De Vynck & Tiku, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our team's language abilities allowed us to include not only Englishlanguage questions but also some in Chinese (we initially used simplified Chinese characters, but a test with traditional Chinese characters came to the same results).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Methods
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“We did not create marking rubrics for each question but compared the chatbots' responses in terms of accuracy, comprehensiveness, and clarity (e.g.Saroyan & Geis, 1988).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Methods
What the sources say
- Not checked yetcites Saroyan & Geis, 1988
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“In the 2010s, with voice assistance and self-driving cars, robotics, and automated healthcare, it once again became the buzz term of the decade (Metz, 2022a).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Conclusions and recommendations
What the sources say
- Not checked yetcites Metz, 2022a
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“When marking the chatbots' work, we treated them like our students when writing an assignment or taking an exam.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“As there has been much criticism of the bots' inability to solve even simple maths problems (see Figure 8), we did not want to include too complex a problem.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Methods
What the sources say
- Not checked yetcites Figure 8
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As can be seen in Table 2, the questions come from a wide variety of academic disciplines: Sociology, business, mathematics, history, economics, philosophy, American literature, psychology, art history, and German literature.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Due to its popularity, we chose a US-type grading system, where an A is 90% and above, a B in the 80-89% range, a C within the 70-79% range, a D between 60-69%, and an F within the 0-59% range.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“All questions are related to higher education assignments and exams.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“Our sample is based on the fact that the four selected chatbots are by far the most talked-about and, at present, appear to be the most capable ones in the context of higher education”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Methods
What the sources say
- Not checked yetcites Mauran, 2023
- Not checked yetcites Mollick, 2023e
- Not checked yetcites Zhou, 2023
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“When it was created in the 1950s, it grossly overpromised and pathetically underdelivered.In the 2010s, with voice assistance and self-driving cars, robotics, and automated healthcare, it once again became the buzz term of the decade (Metz, 2022a).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Conclusions and recommendations
What the sources say
- Not checked yetcites Metz, 2022a
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Scientists such as Stephen Hawking and Max Tegmark are wary of humans inadvertently creating artificial general intelligence (AGI) -a machine capable of performing all intellectual tasks that humans are capable of (Tan, 2023;Hawking et al., 2014;Tegmark, 2018).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Conclusions and recommendations
What the sources say
- Not checked yetcites Tan, 2023
- Not checked yetcites Hawking et al., 2014
- Not checked yetcites Tegmark, 2018
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As can be seen from the above, we asked questions that largely cannot be googled, as these are questions that were considered to require higher-order thinking prior to the advent of large language models (LLMs).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“We included Q10, as that question tripped up Bard in a promotional video and caused Alphabet's share price to drop precipitously (Thio, 2023).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Methods
What the sources say
- Not checked yetcites Thio, 2023
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We divided the labour of grading according to our different expertise”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Methods
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Similarly, higher education reactions to the bots have been on a continuum between banning software use and proactively including it in the curricula.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Conclusions and recommendations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Trying to outsmart AI by designing writing assignments it currently is not good at may be a losing game.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Recommendations for higher education faculty
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: A yet-to-be-publicly-made-available version of GPT-4 can analyse images and provide lengthy descriptions.
- Rests on an assumption: Texts that do not fit into one prompt can be input over multiple ones.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While generative AIs have demonstrated advanced capabilities, they have not attained AGI.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Conclusions and recommendations
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“With the current hype, it is difficult to assess whether or not we are at a historic, revolutionary moment in AI development.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Conclusions and recommendations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
In the paper
“Don't count on AI's ability to reliably detect AI and realise that AI detection software is problematic (Perkins, 2023).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Recommendations for assessments (assignments, exams, and theses)
What the sources say
- Not checked yetcites Perkins, 2023
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Banning such software may make it even more attractive (which we see in China, where people go to great creative lengths to access it -see above).”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Recommendations for higher education faculty
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“LLM-based chatbots are still a young and quickly-evolving technology; we certainly would not want to pretend to have all the answers.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Conclusions and recommendations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“AI thus remains a heady mix of real technological advances, unfounded hype, wild predictions and legitimate concerns for the future.”
§ War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education > Conclusions and recommendations
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it.
The paper relies on this without support.
The selected chatbots (ChatGPT-3.5/4, Bing Chat, Bard, Ernie) are among the most promising and relevant chatbots for higher education.
“In this article, we select some of the most promising chatbots in the English and Chinese-language spaces and provide their corporate backgrounds and brief histories.”
“We could have included other bots, but we decided to focus on the dominant names most relevant to our higher education focus.”
The paper relies on this without support.
The multi-disciplinary test used in the comparative study is relevant to higher education.
“Following an up-to-date review of the Chinese and English-language academic literature, we describe our comparative method and systematically compare selected chatbots across a multi-disciplinary test relevant to higher education.”
The paper relies on this without support.
The comparative test is a valid measure of chatbot intelligence.
“The results of our test show that there are currently no A-students and no B-students in this bot cohort, despite all publicised and sensationalist claims to the contrary.The much-vaunted AI is not yet that intelligent, it would appear.”
The paper relies on this without support.
A yet-to-be-publicly-made-available version of GPT-4 can analyse images and provide lengthy descriptions.
“For instance, a yet-to-be-publicly-made-available version of GPT-4 can analyse images and provide lengthy descriptions.”
The paper relies on this without support.
Texts that do not fit into one prompt can be input over multiple ones.
“Texts that do not fit into one prompt can be input over multiple ones.”
The paper relies on this without support.
Tom's father remains alive throughout The Grapes of Wrath.
“Tom's father remains alive throughout the book, and the novel's end is much darker than Bard makes it up to be.”
The paper relies on this without support.
The novel's end is much darker than Bard makes it up to be.
“Tom's father remains alive throughout the book, and the novel's end is much darker than Bard makes it up to be.”
The authors state this as an assumption.
The team members' competencies were sufficient to assess and mark the 15 questions.
“Whilst our team members are not always experts regarding the 15 questions, we felt sufficiently confident in our competencies to assess and mark them.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“For the past decade, chatbots have been increasingly integrated into various facets of higher education, including assessments, admissions, consultations, and campus life (Luckin et al., 2016;Rudolph et al., 2023b).”
From Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania (2024) · cites this paper as Rudolph et al., 2023b
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Tests have revealed that AI chatbots, including GPT-4, routinely fail to reference academic literature accurately or rely on substandard sources, underscoring the models' limitations in scholarly and intellectual rigour (Rudolph et al., 2023b).”
From Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania (2024) · cites this paper as Rudolph et al., 2023b
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These chatbots are highly regarded because of their usage simplicity, free access, availability, and performance, despite their current limitations such as hallucinations, limited or no access to internet databases, and poor or no use of references [50,51].”
From Using Generative Artificial Intelligence Tools to Explain and Enhance Experiential Learning for Authentic Assessment (2024) · cites this paper as [50]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“AI, therefore, remains a heady mix of real technological advances, unfounded hype, wild predictions, and legitimate concerns for the future (Rudolph et al., 2023b).”
From Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania (2024) · cites this paper as Rudolph et al., 2023b
The cited source says what the claim attributes to it.
In the citing paper
“Similarly, higher education reactions to chatbots have been on a continuum between feeble attempts to ban them and proactively including them in the curricula (Rudolph et al., 2023b).”
From Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania (2024) · cites this paper as Rudolph et al., 2023b
The cited source says what the claim attributes to it.
In the citing paper
“The advent of GenAI has unveiled a plethora of threats to higher education and the broader societal fabric, encompassing concerns over authorship, copyright infringement, and the potential for misuse in activities such as spamming, disseminating fake news, creating malware, and hacking (Marcus, 2023;Rudolph et al., 2023aRudolph et al., , 2023b)).”
From Higher Education's Generative Artificial Intelligence Paradox: The Meaning of Chatbot Mania (2024) · cites this paper as Rudolph et al., 2023b
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“This is well exempli fi ed by the reversal of hearts by some Australian universities (see Rudolph et al. (2023) quoting newspaper articles) who fi rst disallowed the use of AI by students while doing assignments, just to reverse that decision a few months later and replace it by a requirement of disclosing the use of AI in homeworks.”
From AI and ethics: Investigating the first policy responses of higher education institutions to the challenge of generative AI (2024) · cites this paper as Rudolph et al. (2023)
Experimental — accuracy not yet measuredIn the citing paper
“foster students’ performance, engagement and motivation [14, 44, 58], and enhance teaching practices [17, 18, 64, 74].”
From A systematic literature review of empirical research on ChatGPT in education (2024) · cites this paper as [64]
In the citing paper
“The study revealed that ChatGPT achieved the highest score, surpassing Bing Chat and Bard [64].”
From A systematic literature review of empirical research on ChatGPT in education (2024) · cites this paper as [64]
In the citing paper
“ChatGPT is considered the fastest-growing technology in history: in just three months following its public launch, it amassed an estimated 120 million monthly active users [16] with an estimated 13 million daily queries [49], surpassing all other applications [64].”
From A systematic literature review of empirical research on ChatGPT in education (2024) · cites this paper as [64]
In the citing paper
“The findings from previous studies indicate the importance of rethinking, improving, and redesigning innovative assessment methods in the era of chatbots [14, 20, 64, 75]”
From A systematic literature review of empirical research on ChatGPT in education (2024) · cites this paper as [64]
In the citing paper
“the most LLM-driven chatbots present a substantial challenge to traditional assessments [64]”
From A systematic literature review of empirical research on ChatGPT in education (2024) · cites this paper as [64]
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Artificial intelligence: A modern approach
Nilsson, N. J. (1996) · Artificial Intelligence · doi:10.1016/0004-3702(96)00007-0 · Cited by 10,730
Teaching For Quality Learning At University
BIGGS, J. B., & Tang, C. S. (1999) · Cited by 7,559
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326
Teaching for quality learning at university
Boulton‐Lewis, G. M. (2008) · Faculty of Education · Cited by 4,926
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452
Kung, T. H., Cheatham, M., Medenilla, A., Sillos, C., De Leon, L., Elepaño, C., Madriaga, M., Aggabao, R., Diaz-Candido, G., Maningo, J., et al. (2023) · PLOS Digital Health · doi:10.1371/journal.pdig.0000198 · Cited by 4,115
BAİDOO-ANU, D., & OWUSU ANSAH, L. (2023) · Journal of AI · doi:10.61969/jai.1337500 · Cited by 3,939
Research works printed but not linked: 21
- Ali, J. K. M., Shamsan, M. A. A., Hezam, T. A., & Mohammed, A. A. (2023). Impact of ChatGPT on learning motivation: Teachers and students' voices. Journal of English Studies in Arabia Felix, 2(1), 41-49.
- Cao, C. (2020). 人工智能视野下高等教育改革与发展研究. 科 教导刊 (中旬刊). 前沿视角, [Research on higher education reform and development from the perspective of Artificial Intelligence. Journal of Science and Education, Frontier Perspective], 5, 5-6.
- Dwivedi, Y. K., Kshetri, N., Hughes, L., Slade, E. L., Jeyaraj, A., Kar, A. K., ... & Wright, R. (2023). "So what if ChatGPT wrote it?" Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy. International Journal of Information Management, 71, 102642.
- Feng, Y. (2023). ChatGPT在教育领域的应用价值、潜在伦理 风险与治理路径. [The application value, potential ethical risks and governance path of ChatGPT in the field of education]. DOI: 10.16075/j.cnki.cn31-1220/g4.2023.04.013 · DOI
- Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2022). Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. bioRxiv, 2022-12.
- Geerling, W., Mateer, G. D., Wooten, J., & Damodaran, N. (2023). Is ChatGPT smarter than a student in principles of economics?. SSRN 4356034.
- Jiao, J., Chen, L., & Wu, W. (2023). Educational issues triggered by ChatGPT: Possible impacts and counter measures. Chinese Journal of ICT in Education, 29(3), 19-32. DOI: 10.3969/j.issn.1673-8454.2023.03.003 · DOI
- Li, Z. (2023b). The nature of ChatGPT and its impact on education. Chinese Journal of ICT in Education, 29(3), 12-18. DOI: 10.3969/j.issn.1673-8454.2023.03.002 · DOI
- Lu, J. Z. (2023). ChatGPT现象与面向未来的人才培育. [ChatGPT and its potential in talent acquisition]. China Academic Journal Electronic Publishing House, 42-43.
- Marcus, G., & David, E. (2023, January 10). Large language models like ChatGPT say the darnedest things. Communications of the ACM, https://cacm.acm.org/blogs/ blog-cacm/268575-large-language-models-like-chatgpt- say-the-darnedest-things/fulltext
- Mirrlees, T., & Alvi, S. (2020). EdTech Inc. Selling, automating and globalizing higher education in the digital age. Routledge.
- Nisar, S., & Aslam, M. S. (2023). Is ChatGPT a good tool for TCM students in studying pharmacology?. SSRN 4324310.
- Pan, D. (2021). 人工智能和高等教育的融合发展: 变革与引 领. 高等教育研究, [The integrated development of artificial intelligence and higher education: Change and leadership. Higher Education Research], 42(2), 40-46.
- Shaw, G. B. (2017). Pygmalion (Complete Illustrated Edition).
- Steinbeck, J. (2006). The grapes of wrath. Penguin.
- Sun, N. (2023). Reform of higher legal education in the age of artificial intelligence. Journal of Shanxi Datong University(Social Science Edition), 37(1), 147-151.
- Wang, Y., Wang, D., Liang, W., & Liu, C. (2023). Ethical risks and avoidance approaches of ChatGPT in educational application. Open Education Research, 29(2), 26-34.
- Jürgen Rudolph, Shannon Tan and Samson Tan. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
- Wood, P. (2023, March 2). Oxford and Cambridge ban ChatGPT over plagiarism fears but other universities choose to embrace AI bot. Inews, https://inews.co.uk/news/ oxford-cambridge-ban-chatgpt-plagiarism-universities- 2178391?
- Wu, D., Li, H., & Chen, X. (2023). Analysis on the influence of artificial intelligence generic large model on education application. Open Education Research, 29(2), 19-25.
- Zhang, H. F., Chen, Z. F., & Xu, S. J. (2022). 人工智能赋能高等 教育模式新变革. 软件导刊. [Artificial intelligence empowers new reforms in higher education models.] Software Guide, 21(11), 166-171.
Cited by
Sorted by citations
Items 401–421 of 421
Almagribi, A. B., Putranto, B. P. D., Redjeki, S., Isnan, M., Gusteti, M. U., & Erfan, M. (2025) · International Conference on Decision Aid Sciences and Applications (DASA) · doi:10.1109/dasa68193.2025.11499128
Student-AI collaboration in literacy: Evaluating ChatGPT-driven text personalization
Bulut, A. (2026) · The Journal of Educational Research · doi:10.1080/00220671.2026.2664179 · Cited by 1
Дуброва, О. В., Saliuk, B., & МОРОЗ, А. (2026) · Research Notes · doi:10.31654/2663-4902-2026-pp-2-110-119
Nguyen, T. T. (2026) · Research Square · doi:10.21203/rs.3.rs-9813601/v1
Análisis bibliométrico: gestión de conocimiento y prácticas en educación superior
Martínez, C. O., Rivero, C. E.-B., Lozano-Mosos, J., & Orlando, M. (2026) · Educación y Educadores · doi:10.5294/edu.2026.29.1.4
Şendur, G., Öğütcü, E., & Şahin, S. (2026) · Öğretim Teknolojisi ve Hayat Boyu Öğrenme Dergisi - Instructional Technology and Lifelong Learning · doi:10.52911/itall.1848675
Tendencies Toward AI Addiction Among Youth
Çeber, B. (2026) · IGI Global eBooks · doi:10.4018/979-8-3373-6665-4.ch005
Edam-Agbor, I. B., Orim, F. S., Ovat, S. V., Ofem, U. J., Ukatu, J. O., Ekpang, P. O., Igajah, M. N., Ebaye, A. S., & Asukwuo, P. (2026) · Cogent Education · doi:10.1080/2331186x.2026.2690902
Great Potential of Generative AI Tools in Education: A Comparative Analysis of Leading GenAI Systems
Aleksić-Maslać, K., Borović, F., & Kovačić, M. (2026) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro70003.2026.11591829
Bibliometric Mapping of ChatGPT Use in Language Education: Evolution, Trends, and Future Directions
Topal, İ. H. (2026) · Research Synthesis in Applied Linguistics · doi:10.1080/29984475.2026.2709139
Exploring ChatGPT Through Bloom’s Taxonomy: A Systematic Literature Review
Yuliansyah, M. Z., Lintang, F. W., Lunardi, Y., & Permatasari, A. (2025) · International Conference on Computer Science and Artificial Intelligence Applications (CSAIA) · doi:10.1109/csaia65930.2025.11623802
Ahuja, V. K. (2026) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--59401
Rodríguez, A. P. A., González, M. D. C. O., López, H. L. L., & Martínez, R. L. I. (2026) · International Journal of Combinatorial Optimization Problems and Informatics. · doi:10.61467/2007.1558.2026.v17i5.1300
Ming, W., & Li, Q. (2026) · Interactive Technology and Smart Education · doi:10.1108/itse-05-2026-0199 · Cited by 1
A framework for authentic assessment in legal and ethical business curricula
Dixon, A. (2026) · Frontiers in Education · doi:10.3389/feduc.2026.1945511
Karafil, B., & KARAKUYU, A. (2026) · Revista Española de Pedagogía · doi:10.9781/rep.2026.1813
Chatbots in higher education: Enhancing students' digital competence
Titarchuk, V., Iliichuk, L., & Vakulenko, S. (2026) · EDUWEB · doi:10.46502/issn.1856-7576/2026.20.03.12
Research on ChatGPT in higher education: shifting from exploration to policy development
Qi, B., Poo, M. C., Wang, C., & Wang, B. (2026) · Journal of Further and Higher Education · doi:10.1080/0309877x.2026.2744212
Kumari, K. T., Syed, F. A., Nagoji, A., & Savitha, J. (2026) · Advances in computational intelligence and robotics book series · doi:10.4018/979-8-3373-8740-6.ch013
Wangsa, K., Karim, S., Sandu, N., & Singh, P. J. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-35982-7_15
Scarci, A. S., Teixeira, T. M., & Forno, L. F. D. (2026) · Revista Inovação Projetos e Tecnologias · doi:10.5585/2026.30252
Metadata
- Authors
- Rudolph, J., Tan, S., & Tan, S.
- Venue
- Journal of Applied Learning & Teaching
- Publisher
- Open Access Publishing Association
- Published
- 2023-04-25
- Volume
- 6
- Issue
- 1
- ISSN
- 2591-801X (ISSN-L)
- OpenAlex
- W4366989878
- Cited by
- 421
- References
- 81
- Citations by year
- 2026: 43 · 2025: 125 · 2024: 167 · 2023: 84
- FWCI
- 98.33
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Journal of Applied Learning & Teaching (publishedVersion, cc-by)
- Topics
- Social Sciences › Education › Artificial Intelligence in Education
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Keywords
- chatbots, large language models, ChatGPT, Bing Chat, Google Bard, ERNIE Bot, GPT-4, higher education, artificial intelligence in education, academic assessment, generative AI, AI arms race
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
